Data Definition Object Useless or Not Working

[yiannimercer]

Overview of Issue

The DataSummaryPreset is calculating metrics on columns not defined in the data definition, but just happen to exist on the dataset.

Sample

    data_definition = DataDefinition(
        numerical_columns=numeric_features,  # list
        categorical_columns=categorical_features, # list
        regression=regression,  # [Regression(name='default', target='acquisition_price', prediction='predicted')]
    )


# Convert datasets to evidently objects
    reference_evidently = Dataset.from_pandas(
        reference,  # pd.DataFrame
        data_definition=data_definition,  # DataDefinition object
    )
    current_evidently = Dataset.from_pandas(
        current,   # pd.DataFrame
        data_definition=data_definition,  # DataDefinition object
    )

# List of reports
report_list = [
   DataDriftPreset(columns=model_features_list),
   DataDriftPreset(columns=target_predictions_cols),
   DataSummaryPreset(columns=model_features + target_prediction_cols),
   RegressionPreset()
]

# Report 
report = Report(report_list, include_tests=True)
executed_report = report.run(current_data=current_evidently, reference_data=reference_evidently) 

Issue

The DatasetSummaryPreset() is calculating summary metrics and tests are failing on columns that are not defined in my data definition. If this is because the dataset has these extra columns that are not defined in model_features or target_prediction_cols, I ask what is the point of the DataDefintion object? I am hoping I am just doing something wrong and it's not using it. Otherwise, for the DataDriftPreset, I already need to define the columns in the Report object, so I don't see the necessity of the DataDefinition object there either,.

[PavanCodes05]

Hello! I saw this issue has been open for some time. I’d be happy to contribute and work on it—please let me know if it’s still available.

[Liraim]

@yiannimercer,
Thanks you waiting!
I am investigating why DatasetSummaryPreset can use columns that aren’t explicitly mentioned. What I’ve found so far:

• The preset reads the entire dataset to compute overall dataset metrics (e.g., counts of columns by type, etc.)
• It uses provided column list for ValueStats calculations over those columns (e.g., min/max values, etc.)

Because of this, tests in this preset should compute only for given metrics. Failures related to unmentioned columns are expected only in first set of metrics. Does that behavior match what you’re seeing?

It would also really help if you could share which tests are failing along with actual vs. expected outputs. That will let me confirm I’ve caught all the issues in this preset. Thanks!

I already need to define the columns in the Report object, so I don't see the necessity of the DataDefinition object there either
Some presets and metrics allow omitting the columns list and we can infer it from DataDefinition object for further use. For example:

• DataDriftPreset - if you omit columns argument - it would automatically use all numerical, categorical and text columns from DataDefinition to calculate metrics.

[yiannimercer]

Apologies, just seeing this. I can't recall how to replicate the tests, but I think we found a work around. Ultimately though, this boiled up into the issue I just left here: #1765

What is the point of the DataDefinition if at the end of the day, the preprocessing, the tests (example on this issue) are using all the columns you provide in the DataFrame. The DataDefintion effectively does nothing.

If I tell evidently, "these are my categorical columns, these are my numeric columns, this is my target column, this column has my predictions, this is my Date column, .., etc." (what the DataDefinition seems to be for), why would things like data preprocessing (see linked issue), drift tests, or really anything at all consider columns that are defined in this object?

[jasperbruin]

Hey! Just wanted to check in on this one. Has there been any progress on the auto-inference behaviour? Still running into situations where defining just a few columns triggers inference on the entire dataframe, which makes it hard to control what gets processed. Any updates or workarounds in the meantime?