Issue
The presets do not work with the pandas native missing value class pd.NA (alias for <NA>), and will throw a pydantic ValidationError
Example:
import pandas as pd
from evidently import DataDefinition, Dataset
from evidently.presets import DataDriftPreset, DataSummaryPreset
df = pd.DataFrame(
{"a": pd.Series(["x", "y", "z", pd.NA], dtype="str")}
)
definition = DataDefinition(categorical_columns=['a'])
report = Report([DataSummaryPreset(),])
data = Dataset.from_pandas(df, definition)
evaluation = report.run(current_data=data)This will fail with:
│ /Users/xxxxx/lib/python3.12/site-packages/evidently/core/metric_types.py:1004 │
│ in result │
│ │
│ 1001 │ │ raise NotImplementedError │
│ 1002 │ │
│ 1003 │ def result(self, count: Dict[Label, Value], shares: Dict[Label, Value]) -> ByLabelCo │
│ ❱ 1004 │ │ return ByLabelCountValue( │
│ 1005 │ │ │ counts={ │
│ 1006 │ │ │ │ k: SingleValue( │
│ 1007 │ │ │ │ │ value=v,
| /Users/xxxxx/lib/python3.12/site-packages/pydantic/v1/main.py:347 in __init__ │
│ │
│ 344 │ │ # Uses something other than `self` the first arg to allow "self" as a settable a │
│ 345 │ │ values, fields_set, validation_error = validate_model(__pydantic_self__.__class_ │
│ 346 │ │ if validation_error: │
│ ❱ 347 │ │ │ raise validation_error │
│ 348 │ │ try: │
│ 349 │ │ │ object_setattr(__pydantic_self__, '__dict__', values) │
│ 350 │ │ except TypeError as e:
ValidationError: 4 validation errors for ByLabelCountValue
counts -> __key__
value is not a valid integer (type=type_error.integer)
counts -> __key__
str type expected (type=type_error.str)
shares -> __key__
value is not a valid integer (type=type_error.integer)
shares -> __key__
str type expected (type=type_error.str)
Motivation
The workaround would be to convert the pd.NAto None.
However, ideally I would like to run the exploration before performing any data processing at all, on the raw data. This issue prevents the initial data monitoring from running.
Additional Info
Tested with DataSummaryPreset and DataDriftPreset
Evidently version: 0.7.4
Pandas version: 2.1.4
Python Version: 3.12.9
Issue
The presets do not work with the pandas native missing value class
pd.NA(alias for<NA>), and will throw apydantic ValidationErrorExample:
This will fail with:
Motivation
The workaround would be to convert the
pd.NAtoNone.However, ideally I would like to run the exploration before performing any data processing at all, on the raw data. This issue prevents the initial data monitoring from running.
Additional Info
Tested with
DataSummaryPresetandDataDriftPresetEvidently version:
0.7.4Pandas version:
2.1.4Python Version:
3.12.9