feat(narwhals): support the pandas schema API in the narwhals backend - #2417
Merged
Conversation
Route pd.DataFrame validation through the shared narwhals backend when PANDERA_USE_NARWHALS_BACKEND=True, following the registration pattern established for polars, ibis, and pyspark: - register_pandas_backends() gains a use_narwhals_backend flag that force-overrides only the pd.DataFrame entries for DataFrameSchema (including the deprecated root-level class), Column, and narwhals-frame Check dispatch. SeriesSchema, Index, MultiIndex, Parser, Hypothesis, and other pandas-like frame types (dask, modin, geopandas, pyspark.pandas) stay on the native backends. - re_register_pandas_backends() replays fqn-parameterized registrations when the flag is toggled at runtime; set_config re-registration and clear_narwhals_compatible_backend_registry now cover pandas. - pandas Column gains the selector property the narwhals backend depends on (mirroring polars/ibis/pyspark). - check_dtype compares native pandas dtypes through the pandas engine (schema-driven dispatch like the existing pyspark case) so Column(str) accepts object columns and nullable/categorical/tz dtypes keep native semantics. - The check backend passes pd.Series / pd.DataFrame to native=True check functions and normalizes Series/DataFrame/numpy outputs, so existing pandas-style checks work unchanged. - ignore_na now ORs in the input column's nullness on pandas-like frames (pandas NaN comparisons yield False instead of propagating null); polars/ibis behavior is unchanged. - Failure cases for pandas validations are built and concatenated as pandas DataFrames (including all-scalar error sets), with no pandas index leakage; scalar failure cases no longer require polars on polars-free installs. - Index components emit a SchemaWarning and are skipped (narwhals has no index concept); column-level coerce=True keeps the shared no-op warning behavior. Tests: the tests/narwhals/backends parity suite now runs against pandas as a third backend; new registration and pandas-specific e2e suites cover dtype semantics, pandas-style checks, NaN ignore_na handling, failure-case types, and runtime flag toggling. Docs: narwhals_backend.md gains a "pandas: differences from the native backend" section; configuration.md, supported_libraries.md, the API reference, and config docstrings now mention pandas. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: cosmicBboy <niels.bantilan@gmail.com>
… CI matrix - pandas >= 3.0 reports __module__ as "pandas" (not "pandas.core.*") for its public classes; pandas-like detection in the narwhals check backend now compares the root module segment (fix landed in the feature commit). - test_lazy_failure_cases_scalar_only: string columns are dtype "object" on pandas < 3 and "str" on pandas >= 3 — derive the expected dtype name from the frame instead of hardcoding "object". - test_schemas_on_pyspark_pandas.py::test_failure_cases: accept both failure-cases shapes — the native pandas backend reshapes into a "failure_case" column while the narwhals backend (flag-on CI session) returns failing rows with original column names. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: cosmicBboy <niels.bantilan@gmail.com>
cosmicBboy
force-pushed
the
narwhals-pandas-backend
branch
from
July 16, 2026 13:49
c46461b to
1306a92
Compare
- Add tests for pandas-style checks returning boolean DataFrames (with and without CHECK_OUTPUT_KEY), iterable dtype-check results, scalar failure cases on polars-free installs, non-frame err.data fallback in failure_cases_metadata, drop_invalid_rows on nullable violations, and the _is_pandas_like helper. - Mark ImportError guards that no CI environment can reach (pandas / narwhals are always co-installed there) as no-cover, following the existing pyspark register precedent. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: cosmicBboy <niels.bantilan@gmail.com>
Because pandas frames are eager, the Narwhals pandas path can reach feature parity with the native pandas backend. Almost every step is now handled Narwhals-native; only Index/MultiIndex validation+coercion (Narwhals has no index concept) and Hypothesis checks (scipy) are delegated to the native pandas backend. - Index/MultiIndex components are validated (and coerced) by delegating to the native Index/MultiIndex backends; Narwhals preserves the pandas index through its operations, so the parsed frame still carries it. Non-pandas frames with an index component still warn and skip. - Column-/schema-level coerce=True uses a hybrid strategy: plain numpy dtypes cast via nw.cast, pandas extension dtypes (nullable Int64, Categorical, tz-aware datetime, string) fall back to the pandas dtype engine so native semantics are preserved. Cast failures report DATATYPE_COERCION with the offending values. Column-level coerce is no longer a no-op warning on pandas. - add_missing_columns, set_default, and custom parsers are applied Narwhals-native (parsers run on the native frame pre-wrap); added columns follow the same hybrid dtype rule as coercion. - groupby column-check-groups run Narwhals-native via NarwhalsCheckBackend.apply_groupby (builds the pandas group dict); Hypothesis checks are delegated to the native pandas hypothesis backend. - unique_column_names is detected on the native frame pre-wrap (Narwhals rejects duplicate labels at wrap time). Docs updated to describe the pandas parity and remaining delegated gaps. Adds extensive e2e coverage in tests/narwhals/test_pandas_e2e.py (index/multiindex validation+coercion, hybrid coercion fidelity, parsers, add_missing_columns, set_default, groupby, hypothesis, unique column names). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- test_schema_level_coerce: derive the expected str-coercion dtype from the pandas engine instead of hardcoding `object`. On pandas >= 3.0 `Column(str)` coerces to `pd.StringDtype(na_value=nan)`, not `object`, so the hardcoded assertion failed on the pandas-3.0 narwhals CI. - test_set_defaults: drop the now-stale strict xfail for the narwhals backend. set_default is implemented for the narwhals backend, so the test XPASSed and failed the strict-xfail gate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Routes
pd.DataFramevalidation through the shared Narwhals backend when it is enabled (PANDERA_USE_NARWHALS_BACKEND=Trueorpandera.set_config(use_narwhals_backend=True)), following the registration pattern established for polars, ibis, and pyspark. The flag remains off by default — native behavior is unchanged.Because pandas frames are eager, the Narwhals pandas path now reaches near feature-parity with the native pandas backend. Almost every step is handled Narwhals-native; only two things are delegated to the native pandas backend:
Hypothesischecks — these rely on scipy statistical tests with no Narwhals-expression equivalent.Scope: only
pd.DataFramevalidation viaDataFrameSchema/DataFrameModelis rerouted.SeriesSchemaand the other pandas-like frame types (dask, modin, geopandas,pyspark.pandas) always stay on the native pandas backends.Changes
Registration
register_pandas_backends()gains ause_narwhals_backendflag that force-overrides only thepd.DataFrameregistry entries forDataFrameSchema(including the deprecated root-level class),Column, and narwhals-frameCheckdispatch. The(Check, pd.DataFrame)entry intentionally stays native so directCheckcalls on native frames are unchanged.re_register_pandas_backends()replays every registered fqn with the new flag value;set_config()re-registration andclear_narwhals_compatible_backend_registry()now cover pandas.pandera.pandasColumngains theselectorproperty the narwhals backend depends on.Narwhals backend adaptations (all gated to pandas-like frames; polars/ibis/pyspark paths unchanged)
check_dtypecompares native pandas dtypes through the pandas engine, soColumn(str)acceptsobjectcolumns, nullableInt64, categoricals, and tz-aware datetimes behave as under the native backend.native=Truecolumn checks receive thepd.Series, dataframe-level checks thepd.DataFrame; boolean Series/DataFrame/ndarray and numpy-scalar outputs are normalized.native=Falsepasses a narwhals column expression as with the other backends.coerce=True(column- and schema-level): hybrid strategy — plain numpy dtypes cast vianw.cast(Narwhals-native); pandas extension dtypes (nullableInt64,Categorical, tz-aware datetime,string) and any cast that raises fall back to the pandas dtype engine so native semantics are preserved. Failures reportDATATYPE_COERCIONwith the offending values. Column-levelcoerceis no longer a no-op warning on pandas.add_missing_columns,set_default, customparsers: applied Narwhals-native. Parsers run on the native frame pre-wrap (arbitrary user code); missing/default columns are built from Narwhals expressions (nw.lit/fill_null/cast) and follow the same hybrid dtype rule as coercion (a missing nullableInt64is added asInt64/<NA>, notobject); missing columns are inserted at their schema position.groupbycolumn-check-groups: run Narwhals-native viaNarwhalsCheckBackend.apply_groupby(builds the pandas group dict, reduces per-group output to a single pass/fail).groups=filtering supported.unique_column_names: detected on the native frame pre-wrap, since Narwhals rejects duplicate column labels at wrap time.ignore_nacorrectness fix for NaN: pandas comparisons on NaN yieldFalserather than propagating null, soignore_nanow also ORs in the input column's nullness — in check evaluation, deferred failure-case reconstruction, anddrop_invalid_rows.__index_level_0__);_concat_failure_casesgains a pandas-like branch, and the all-scalar case (e.g. dtype-only errors) resolves the frame type from the schema class. Scalar failure cases no longer hard-require polars.Docs
narwhals_backend.md: rewritten "pandas: differences from the native backend" section describing the parity and the two delegated gaps (Index/MultiIndex, Hypothesis); known-gaps list scoped to non-pandas backends where appropriate.configuration.md,supported_libraries.md,reference/narwhals.rst, andconfig.pydocstrings updated.Tests
tests/narwhals/backends/parity suite runs against pandas as a third backend (newPandasBackendfixture + registeredpandasmarker).tests/narwhals/test_pandas_narwhals_register.py: activation/deactivation, native fallback, idempotence, fqn replay,set_configruntime toggling.tests/narwhals/test_pandas_e2e.py: native dtype semantics, pandas-style/aggregate/element-wise/expression checks, NaNignore_naincl.drop_invalid_rows, pandas failure-case types with no index leakage, regex columns,DataFrameModel, ordered/strict — plus the new parity coverage: Index/MultiIndex validation + coercion, hybrid coercion fidelity (numpy vianw.cast, extension dtypes via pandas engine, failure reporting), custom parsers (chained/element-wise), add_missing_columns (ordering, extension-dtype fidelity, required-default errors), set_default, groupby checks (dict output,groups=filter), Hypothesis checks, and unique_column_names.tests_narwhals_backendnox matrix — the fulltests/pandas/suite exercises features the narwhals backend does not support (rationale documented in the noxfile). The new suites run in the existingunit-tests-narwhalsCI job.Verification
tests/pandas+tests/base+tests/polars+tests/ibisunchanged.tests/narwhals/test_pandas_e2e.py→ 72 passed; fulltests/narwhalssuite shows no new failures relative tomain(the same pre-existing ibis/pyspark local-env failures reproduce identically on the unmodified branch point).🤖 Generated with Claude Code