fix(pandas-engine): respect optional pydantic fields in PydanticModel dtype - #2415
Open
Karthick-dev-cart wants to merge 2 commits into
Open
fix(pandas-engine): respect optional pydantic fields in PydanticModel dtype#2415Karthick-dev-cart wants to merge 2 commits into
Karthick-dev-cart wants to merge 2 commits into
Conversation
… dtype DataFrameSchema auto-generates a Column for each field of a PydanticModel dtype when no columns are explicitly declared. These auto-generated columns always defaulted to nullable=False, so optional pydantic fields (e.g. `Optional[int] = None`) were rejected whenever the underlying data contained nulls or the column was missing entirely, even though the pydantic model itself allows this. Add PydanticModel.column_nullability, which maps each column name (alias or field name) to whether the corresponding pydantic field is required, and use it when building the auto-generated columns so optional fields are marked nullable. Fixes unionai-oss#2406 Signed-off-by: Karthick <karthickd.tech@gmail.com>
… assertion
The end-to-end null-rejection test failed under pydantic 1.10.11 in CI
(python 3.11-3.13, all OSes): pydantic v1's `str` validator coerces a
non-string input like float('nan') into the string 'nan' instead of
rejecting it, so a null title never reaches PydanticModel.coerce()'s
ValidationError handling on v1. Confirmed this is pydantic v1's own
coercion behavior, not a bug in column_nullability - the nullable flag
on the auto-generated Column is correct on both versions.
Added a version-agnostic test asserting the nullable flag directly, and
skipped the end-to-end rejection test on pydantic v1 with the reason
above.
Signed-off-by: Karthick <karthickd.tech@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Fixes #2406: optional pydantic fields (e.g.
Optional[int] = None) were rejected as non-nullable when validating apandera.DataFrameModelwhoseConfig.dtype = PydanticModel(...), even though the pydantic model itself allows the field to be missing orNone.Root cause
DataFrameSchema.__init__auto-generates a panderaColumnfor each field of aPydanticModeldtype when no columns are explicitly declared (added in #2381). That auto-generation always builtColumn(None, name=name), andColumn's default isnullable=False— so every auto-generated column was non-nullable regardless of whether the corresponding pydantic field was actually optional. This was a regression between 0.32.0 (working) and 0.32.1 (broken), introduced by #2381.Fix
PydanticModel.column_nullability, which maps each column name (alias or field name, matching the existingcolumn_namesproperty) to whether the corresponding pydantic field is not required (checksFieldInfo.is_required()on Pydantic v2,ModelField.requiredon v1).DataFrameSchema.__init__now passes the matchingnullablevalue when building each auto-generated column, instead of hardcodingFalse.Testing
ratingcolumn missing entirely, and present with allNonevalues) againstmain— both raisedSchemaError.test_pydantic_model_optional_field_missing_column,test_pydantic_model_optional_field_null_values).test_pydantic_model_required_field_still_rejects_nullsto confirm required fields are unaffected — nulls in a required field still correctly raise.PydanticModel(tests/pandas/test_pydantic_dtype.py,tests/pandas/test_pydantic.py,tests/pandas/test_model.py): all 123 tests pass.ruff checkclean on the changed files.Notes
Scope is intentionally limited to the pandas backend /
PydanticModeldtype path (the two files touched by #2381); the polarsPydanticModelimplementation is separate and unaffected.