ExpressionDFilter result not as expected

[playgithub]

env

python 3.12.9
pylib 0.9.7

data

https://github.com/chenditc/investment_data/releases/tag/2026-01-11
随机验证了一下，$close / $factor = 未复权价格，数据应该没问题

problem

问题：按$close>2000，选出来的票，实际后复权价格不符合$close>2000

from qlib.data import D
from qlib.data.filter import ExpressionDFilter

expressionDFilter = ExpressionDFilter(rule_expression='$close > 2000')
instruments_cfg = D.instruments(market='csi300', filter_pipe=[expressionDFilter])
instruments = D.list_instruments(instruments=instruments_cfg,
                                 start_time='2025-01-01',
                                 end_time='2026-01-09',
                                 as_list=True)
print(instruments)

output

['SH600837', 'SH601989', 'SZ002049']

instruments = ['SZ002049']
fields = ['$close', '$volume', 'Ref($close, 1)', 'Mean($close, 3)', '$high-$low', '$factor']
df = D.features(instruments,
                fields,
                start_time='2025-01-01',
                end_time='2026-01-09',
                freq='day')
print(df)

output

                          $close        $volume  Ref($close, 1)  Mean($close, 3)  $high-$low   $factor
instrument datetime                                                                                   
SZ002049   2025-01-02  54.424286  272530.312500       57.611427        57.426189    3.606804  0.894989
           2025-01-03  53.189999  173090.734375       54.424286        55.075237    1.682606  0.895003
           2025-01-06  52.715714  117793.007812       53.189999        53.443333    1.217205  0.895004
           2025-01-07  54.022858  137582.578125       52.715714        53.309525    1.440968  0.895011
           2025-01-08  53.162857  181623.218750       54.022858        53.300476    2.452293  0.894998
...                          ...            ...             ...              ...         ...       ...
           2025-12-23  68.699997  163727.125000       69.284286        68.697617    1.185257  0.897922
           2025-12-24  69.967140  225979.796875       68.699997        69.317139    1.508530  0.897936
           2025-12-25  71.088570  274110.406250       69.967140        69.918571    1.894615  0.897923
           2025-12-26  72.031425  330256.906250       71.088570        71.029045    2.496223  0.897924
           2025-12-29  70.765717  262366.625000       72.031425        71.295235    1.948502  0.897928

[SunsetWolf]

Hi, @playgithub , Thanks for the detailed report. This behavior is not caused by ExpressionDFilter itself, but by inconsistent instrument metadata in the dataset.

Specifically, list_instruments relies on instruments/csi300.txt to determine each instrument’s valid date ranges. In your data, SZ002049 is declared as active until 2026-01-13, but the actual feature data (features/sz002049/close.day.bin) only exists up to 2025-12-29.

You can verify this by checking instruments/csi300.txt.

SZ002049 2025-06-30 2025-12-30
SZ002049 2025-12-31 2026-01-13

Workarounds:

Remove or fix problematic date intervals in instruments/csi300.txt, or wait for an updated version of the dataset in the investment_data repository to see if the new version of the data will fix this issue.

NOTE: Also, note that using $close / $factor alone to validate data correctness is insufficient; instrument date alignment is equally critical.

Hope this helps clarify the issue.

[playgithub]

SZ002049 is in a trading halt from 2025.12.30 to 2025.1.14, so the data is ok.

[SunsetWolf]

Hi, @playgithub , Thanks for the clarification.

I agree that the trading halt itself is valid, and the raw data is not “wrong” in the sense of being fabricated or corrupted. However, the issue here is how Qlib interprets the instrument’s active date ranges during a halt.

In Qlib, list_instruments determines whether an instrument is eligible solely based on the date intervals declared in instruments/csi300.txt, regardless of whether feature data actually exists for those dates. During the halt period (2025-12-30 → 2026-01-14), there is no feature data, but the instrument is still considered “active”.

As a result, ExpressionDFilter evaluates $close > 2000 on dates where $close is missing, which leads to unexpected instrument selection, even though later inspection of available prices looks perfectly normal.

So the problem is not that the halt is incorrect, but that:

the instrument is marked as tradable over dates with no feature data, and

Qlib does not currently distinguish “halted / no data” from “valid trading days” when filtering instruments.

This is essentially a metadata–feature alignment issue, not a price adjustment issue. Cleaning or splitting the date ranges in csi300.txt, or improving how halted periods are handled in filtering logic, would avoid this confusion.

Hope this clarifies why the behavior still occurs even if the halt itself is legitimate.

[playgithub]

ExpressionDFilter evaluates $close > 2000 on dates where $close is missing

While filtering, is it better to ignore the missing data?
So that filtering is filtering, it's simple, no ambiguity, perfect :)