diff --git a/docs/source/configuration.md b/docs/source/configuration.md index 4ae3f5752..d99d5f603 100644 --- a/docs/source/configuration.md +++ b/docs/source/configuration.md @@ -38,7 +38,7 @@ This can be achieved by setting the environment variable Pandera ships an optional [Narwhals](https://narwhals-dev.github.io/narwhals/)-powered backend that -unifies the Polars, Ibis, and PySpark SQL validation paths. It is **opt-in**; by +unifies the Polars, Ibis, PySpark SQL, and pandas validation paths. It is **opt-in**; by default the native backends are used. Install the `narwhals` extra and enable the backend with either: diff --git a/docs/source/narwhals_backend.md b/docs/source/narwhals_backend.md index abc4fc06d..89c30dacf 100644 --- a/docs/source/narwhals_backend.md +++ b/docs/source/narwhals_backend.md @@ -4,13 +4,14 @@ As of *0.32.0*, Pandera ships an optional [Narwhals](https://narwhals-dev.github.io/narwhals/)-based validation -backend that powers the {ref}`Polars `, {ref}`Ibis `, and -{ref}`PySpark SQL ` integrations behind a single unified code -path. The Narwhals backend is **opt-in**: by default Pandera continues to use -the native Polars, Ibis, and PySpark backends. The public API +backend that powers the {ref}`Polars `, {ref}`Ibis `, +{ref}`PySpark SQL `, and {ref}`pandas ` +integrations behind a single unified code path. The Narwhals backend is +**opt-in**: by default Pandera continues to use the native Polars, Ibis, +PySpark, and pandas backends. The public API (`import pandera.polars as pa`, `import pandera.ibis as pa`, -`import pandera.pyspark as pa`) is unchanged regardless of which backend is -active. +`import pandera.pyspark as pa`, `import pandera.pandas as pa`) is unchanged +regardless of which backend is active. ## Enabling the Narwhals backend @@ -21,6 +22,7 @@ backend(s) you use: pip install 'pandera[narwhals,polars]' # Polars pip install 'pandera[narwhals,ibis]' # Ibis pip install 'pandera[narwhals,pyspark]' # PySpark SQL +pip install 'pandera[narwhals,pandas]' # pandas ``` Then enable it using **either** of the following options. @@ -39,7 +41,7 @@ This value is read when `pandera.config` is first imported. ### Programmatic configuration Call {func}`~pandera.set_config` at any point — before or after importing -`pandera.polars`, `pandera.ibis`, or `pandera.pyspark`: +`pandera.polars`, `pandera.ibis`, `pandera.pyspark`, or `pandera.pandas`: ```python import pandera @@ -66,7 +68,18 @@ register_polars_backends(use_narwhals_backend=True) ``` The same pattern applies to `register_ibis_backends` and -`register_pyspark_backends`. +`register_pyspark_backends`. The pandas register function is additionally +parameterized by the fully qualified name of the frame class being +validated: + +```python +from pandera.backends.pandas.register import register_pandas_backends + +register_pandas_backends.cache_clear() +register_pandas_backends( + "pandas.core.frame.DataFrame", use_narwhals_backend=True +) +``` If `PANDERA_USE_NARWHALS_BACKEND=True` but `narwhals` is not installed, schema construction raises an `ImportError` directing you to install @@ -85,7 +98,7 @@ Two behaviours govern how that mapping is established and updated at runtime. ### Lazy registration -Validation backends for Polars, Ibis, and PySpark SQL are registered +Validation backends for Polars, Ibis, PySpark SQL, and pandas are registered **lazily** — not when you import a pandera backend module, but the first time a schema needs a backend. Concretely, registration runs when you: @@ -93,10 +106,12 @@ a schema needs a backend. Concretely, registration runs when you: {py:class}`~pandera.api.ibis.container.DataFrameSchema`, or {py:class}`~pandera.api.pyspark.container.DataFrameSchema`, or - call `validate()` on a column or schema component that triggers backend - lookup. + lookup (for pandas schemas, registration always happens at validation + time, when the type of the validated object is known). -Until one of those happens, importing ``pandera.polars``, ``pandera.ibis``, or -``pandera.pyspark`` has no effect on which validation backend is active: +Until one of those happens, importing ``pandera.polars``, ``pandera.ibis``, +``pandera.pyspark``, or ``pandera.pandas`` has no effect on which validation +backend is active: ```python import pandera.polars as pa @@ -128,8 +143,8 @@ backends have already been registered, pandera **re-registers** them automatically: 1. The global ``CONFIG.use_narwhals_backend`` value is updated. -2. Pandera detects which of the Polars / Ibis / PySpark register functions had - already run. +2. Pandera detects which of the Polars / Ibis / PySpark / pandas register + functions had already run. 3. Registration caches are cleared and existing registry entries for those backends are removed. 4. Only the backends that were previously registered are registered again, now @@ -154,8 +169,8 @@ registered backend from the global registry on each ``validate()`` call. Re-registration applies only to backends that had already been registered in the current process. If you call ``set_config(use_narwhals_backend=True)`` -before constructing any Polars/Ibis/PySpark schema, no re-registration occurs -— the first lazy registration picks up the updated config silently. +before constructing any Polars/Ibis/PySpark/pandas schema, no re-registration +occurs — the first lazy registration picks up the updated config silently. :::{note} Runtime re-registration is triggered by {func}`~pandera.set_config`, which @@ -177,12 +192,13 @@ it executed natively by each supported engine. ## What it changes for you -* **Unified checks across Polars, Ibis, and PySpark SQL.** Built-in checks - (`isin`, `in_range`, `str_matches`, etc.) are implemented as Narwhals - expressions and run unchanged on Polars LazyFrames, Ibis tables, and - PySpark SQL DataFrames when the Narwhals backend is enabled. PySpark SQL - is a SQL-lazy backend: element-wise checks are not supported, and row - sampling (`sample=` / `tail=` parameters) is not supported. +* **Unified checks across Polars, Ibis, PySpark SQL, and pandas.** Built-in + checks (`isin`, `in_range`, `str_matches`, etc.) are implemented as + Narwhals expressions and run unchanged on Polars LazyFrames, Ibis tables, + PySpark SQL DataFrames, and pandas DataFrames when the Narwhals backend is + enabled. PySpark SQL is a SQL-lazy backend: element-wise checks are not + supported, and row sampling (`sample=` / `tail=` parameters) is not + supported. * **Lazy validation stays lazy.** For Polars LazyFrames, Ibis tables, and PySpark SQL DataFrames, Pandera threads validation through the native lazy API: no full-frame `.collect()` / `.execute()` is triggered during @@ -235,12 +251,82 @@ backend: `dataframe.pandera.errors` accessor, use the native PySpark backend (see {ref}`Opting out `). +(narwhals-pandas-differences)= + +## pandas: differences from the native backend + +When the Narwhals backend is enabled, `pandas.DataFrame` validation through +{py:class}`~pandera.api.pandas.container.DataFrameSchema` (and +`DataFrameModel`) is routed through the shared Narwhals code path. Because +pandas frames are **eager**, the Narwhals pandas path reaches feature parity +with the native pandas backend. Almost everything is handled **Narwhals-native** +(parsers, `add_missing_columns`, `set_default`, column/schema coercion, +`groupby` checks, and all cross-backend checks). The **only** steps delegated to +the native pandas backend are **Index/MultiIndex validation** (and its +coercion) — because Narwhals has no index concept — and **`Hypothesis`** +checks, which rely on scipy statistical tests. A few behavioural notes: + +- **Only `pd.DataFrame` validation is rerouted.** `SeriesSchema` validation — + and the other pandas-like frame types (dask, modin, geopandas, + `pyspark.pandas`) — always use the native pandas backends, regardless of + the flag. `DataFrameSchema` `Index`/`MultiIndex` **components** are + validated (see below). +- **Index and MultiIndex components are validated (delegated).** Narwhals + preserves the pandas index through its operations, so a schema with an + `index=` component is validated by delegating to the native Index/MultiIndex + backends; index-level coercion is applied and the index is preserved in the + output. (For non-pandas frames — polars/ibis/pyspark — Narwhals has no index + concept, so an `index=` component there still emits a ``SchemaWarning`` and + is skipped.) +- **Column dtype checks compare native pandas dtypes.** Dtype *checks* still + use the pandas dtype engine, so `Column(str)` accepts `object` columns, and + nullable extension dtypes (`Int64`), categoricals, and tz-aware datetimes are + recognised as under the native backend. +- **pandas-style check functions keep working.** Column-level checks receive + the `pd.Series` for the column (e.g. `pa.Check(lambda s: s > 0)`), and + dataframe-level checks receive the `pd.DataFrame` — the same convention as + the native backend. Checks may return boolean Series/DataFrames, numpy + boolean arrays, or scalar booleans. Setting `native=False` passes a + Narwhals column expression instead, making the check portable across all + Narwhals-backed integrations. +- **`coerce=True` uses a hybrid strategy.** Column- and schema-level coercion + cast plain numpy dtypes (`int`, `float`, `str`, `bool`) Narwhals-native via + ``nw.cast``, and **fall back to the pandas dtype engine for pandas extension + dtypes** — nullable ``Int64``, ``Categorical``, tz-aware datetimes, and + ``string`` — so their native semantics are preserved (e.g. a nullable + ``Int64`` column keeps its ```` values). A Narwhals cast that raises also + falls back to the pandas engine, which reports the offending values in the + ``DATATYPE_COERCION`` error. (Index-level coercion is delegated to the native + Index backend.) +- **`parsers`, `add_missing_columns`, and `set_default` are applied + Narwhals-native.** Custom `parsers=` run on the native frame inside the + Narwhals backend; `add_missing_columns=True` and per-`Column` `default=` + values are built from Narwhals expressions (`nw.lit`, `fill_null`, `cast`). + Added columns follow the same hybrid dtype rule as coercion: plain numpy + dtypes are cast Narwhals-native, while extension-dtype or null-valued added + columns get their dtype from the pandas dtype engine (so a missing nullable + `Int64` column is added as `Int64` with ``, not `object`). +- **`groupby` column-check-groups are Narwhals-native; `Hypothesis` checks are + delegated.** `groupby=` checks are handled by the Narwhals check backend + (building the pandas group dict for pandas-like frames). `Hypothesis` checks + are delegated to the native pandas hypothesis backend (scipy). Custom checks + that expect the `pd.Series`/`pd.DataFrame` keep working. +- **Data synthesis strategies** (`schema.strategy()` / `schema.example()`) + are unaffected by the backend flag — they operate on the pandas schema API + and generate pandas data directly. +- **`failure_cases` are pandas DataFrames** (including when every failure + case is a scalar, e.g. dtype errors). Unlike the native backend, the + ``index`` column of the aggregated failure-cases frame is null for + column/dataframe-level checks — the Narwhals code path does not report + failing row positions (index-component failures do report the failing + index value). + (narwhals-opting-out)= ## Opting out The Narwhals backend is **off by default**, so no action is needed to -continue using the native Polars, Ibis, and PySpark backends. If you +continue using the native Polars, Ibis, PySpark, and pandas backends. If you previously opted in and want to switch back, unset the environment variable (or set it to `False`): @@ -273,19 +359,36 @@ backend. Follow-up milestones track each of the gaps below: frame. This is because scalar Polars frames cannot be converted to PySpark without a live ``SparkSession`` at the error-collection site; this gap is tracked for a future release. -* Column-level `coerce=True` is currently a no-op for **all** Narwhals backends - (Polars, Ibis, PySpark SQL). Pandera emits a one-time ``SchemaWarning`` per - column so the subsequent ``WRONG_DATATYPE`` error is understandable rather than - silent. Full column-level coercion support is tracked as a follow-up. +* Column-level `coerce=True` is currently a no-op for the **non-pandas** + Narwhals backends (Polars, Ibis, PySpark SQL). Pandera emits a one-time + ``SchemaWarning`` per column so the subsequent ``WRONG_DATATYPE`` error is + understandable rather than silent. Full column-level coercion support for + those backends is tracked as a follow-up. (The **pandas** Narwhals backend + applies column-level coercion via a hybrid of ``nw.cast`` and the pandas + dtype engine — preserving native semantics for extension dtypes; see + {ref}`pandas differences `.) +* The ``index`` column of aggregated ``failure_cases`` frames is null for + column/dataframe-level checks — failing row positions are not reported + (SQL-lazy backends have no row order; the pandas path follows the same + convention). Index-**component** failures do report the failing index value. * `coerce` for the Ibis backend (deferred; `Ibis` coerces eagerly today) -* `add_missing_columns` parser and `set_default` for `Column` fields -* `group_by`-based checks beyond element-wise and column-wise expressions +* `add_missing_columns` parser and `set_default` for `Column` fields on the + **non-pandas** Narwhals backends (both are Narwhals-native on the pandas + backend) +* `group_by`-based checks beyond element-wise and column-wise expressions on + the **non-pandas** Narwhals backends (`groupby` column-check-groups are + Narwhals-native on the pandas backend — see + ``NarwhalsCheckBackend.apply_groupby``) * Element-wise checks for SQL-lazy backends (Ibis and PySpark SQL). As a consequence, the shared built-in check suite in ``tests/common/`` does not run for the PySpark Narwhals backend (all shared checks are element-wise; running them would produce only skips with no useful coverage signal). -* Schema IO (YAML/JSON) for Narwhals-backed schemas -* Hypothesis data-synthesis strategies +* Schema IO (YAML/JSON) for Narwhals-backed schemas. (Unaffected for the + pandas backend: `to_yaml`/`from_yaml` operate on the pandas schema API.) +* Hypothesis checks and hypothesis-based data-synthesis strategies for the + **non-pandas** Narwhals backends. (On the pandas backend, `Hypothesis` + checks are delegated to the native pandas backend and `schema.strategy()` / + `schema.example()` work through the pandas schema API.) * `sample=` / `tail=` row sampling for SQL-lazy backends (Ibis and PySpark SQL) * `check_unique` (column-level uniqueness) does not produce a per-row boolean `check_output`, so `drop_invalid_rows=True` cannot filter rows that fail a diff --git a/docs/source/reference/narwhals.rst b/docs/source/reference/narwhals.rst index c14586aa7..9a3b009d1 100644 --- a/docs/source/reference/narwhals.rst +++ b/docs/source/reference/narwhals.rst @@ -6,7 +6,7 @@ Narwhals Backend *new in 0.32.0* Opt-in `Narwhals `__-powered -validation backend that powers the Polars, Ibis, and PySpark SQL +validation backend that powers the Polars, Ibis, PySpark SQL, and pandas integrations behind a single unified code path. Requires the ``narwhals`` extra. Enable with ``PANDERA_USE_NARWHALS_BACKEND=True`` or :func:`~pandera.set_config`. Backends register lazily on first schema use; diff --git a/docs/source/supported_libraries.md b/docs/source/supported_libraries.md index 2eaf96259..2ffc9b72d 100644 --- a/docs/source/supported_libraries.md +++ b/docs/source/supported_libraries.md @@ -99,7 +99,7 @@ are expressed in the native library's API. :widths: 25 75 * - {ref}`Narwhals ` - - Unified opt-in backend for Polars, Ibis, and PySpark SQL. Enable with ``PANDERA_USE_NARWHALS_BACKEND=True`` or {func}`~pandera.set_config`. + - Unified opt-in backend for Polars, Ibis, PySpark SQL, and pandas. Enable with ``PANDERA_USE_NARWHALS_BACKEND=True`` or {func}`~pandera.set_config`. ::: ```{toctree} diff --git a/noxfile.py b/noxfile.py index 4667907ce..8a4ee5a0d 100644 --- a/noxfile.py +++ b/noxfile.py @@ -340,6 +340,13 @@ def tests_narwhals_backend(session: Session, extra: str) -> None: or tests/ibis/ suite then exercises the narwhals backend rather than the native one. Tests that expose narwhals backend gaps should be marked xfail in the test files. + + pandas is intentionally NOT in this matrix: the full tests/pandas/ suite + exercises features the narwhals backend does not support (column-level + coerce, index/multiindex validation, parsers, hypothesis strategies). + Narwhals-backed pandas coverage lives in tests/narwhals/ (test_pandas_e2e, + test_pandas_narwhals_register, and the backends/ parity suite), which runs + in the unit-tests-narwhals CI job. """ deps = PYPROJECT["project"]["optional-dependencies"] requirements = [ diff --git a/pandera/api/narwhals/utils.py b/pandera/api/narwhals/utils.py index e3a7b7b43..273486377 100644 --- a/pandera/api/narwhals/utils.py +++ b/pandera/api/narwhals/utils.py @@ -24,6 +24,25 @@ } ) +# Eager pandas-like implementations. These are wrapped as ``nw.DataFrame`` +# (then ``.lazy()``-ed by the backend) and unwrap back to a pandas-API frame. +# They share pandas dtype semantics (e.g. numpy object columns, NaN-based +# missing integers) that the backend special-cases in a few places. +_EAGER_PANDAS_LIKE_IMPLEMENTATIONS: frozenset = frozenset( + { + nw.Implementation.PANDAS, + nw.Implementation.MODIN, + nw.Implementation.CUDF, + } +) + + +def _is_pandas_like(frame) -> bool: + """True if frame is backed by an eager pandas-like implementation.""" + if not isinstance(frame, (nw.DataFrame, nw.LazyFrame)): + return False + return frame.implementation in _EAGER_PANDAS_LIKE_IMPLEMENTATIONS + def _to_native(frame): """Convert a Narwhals frame to its native backend frame. diff --git a/pandera/api/pandas/array.py b/pandera/api/pandas/array.py index 10356b975..8fe18ca80 100644 --- a/pandera/api/pandas/array.py +++ b/pandera/api/pandas/array.py @@ -45,15 +45,24 @@ def dtype(self, value: PandasDtypeInputTypes | None) -> None: @staticmethod def register_default_backends(check_obj_cls: type): from pandera.backends.pandas.register import register_pandas_backends + from pandera.config import CONFIG + + use_narwhals_backend = CONFIG.use_narwhals_backend _cls = check_obj_cls try: - register_pandas_backends(f"{_cls.__module__}.{_cls.__name__}") + register_pandas_backends( + f"{_cls.__module__}.{_cls.__name__}", + use_narwhals_backend=use_narwhals_backend, + ) except BackendNotFoundError: for base_cls in _cls.__bases__: base_cls_name = f"{base_cls.__module__}.{base_cls.__name__}" try: - register_pandas_backends(base_cls_name) + register_pandas_backends( + base_cls_name, + use_narwhals_backend=use_narwhals_backend, + ) except BackendNotFoundError: pass diff --git a/pandera/api/pandas/components.py b/pandera/api/pandas/components.py index 1e09d6f6a..9acd2f023 100644 --- a/pandera/api/pandas/components.py +++ b/pandera/api/pandas/components.py @@ -13,6 +13,7 @@ from pandera.api.pandas.types import PandasDtypeInputTypes from pandera.dtypes import UniqueSettings from pandera.import_utils import strategy_import_error +from pandera.utils import is_regex class Column(ArraySchema[pd.DataFrame]): @@ -115,6 +116,23 @@ def _allow_groupby(self) -> bool: """Whether the schema or schema component allows groupby operations.""" return True + @property + def selector(self): + """Column selector used by the Narwhals validation backend. + + Mirrors the ``selector`` property on the polars/ibis/pyspark + ``Column`` components: returns a regex pattern when ``regex=True`` + and the plain column name otherwise. Non-string names (e.g. tuples + for MultiIndex columns) are returned as-is. + """ + if ( + isinstance(self.name, str) + and not is_regex(self.name) + and self.regex + ): + return f"^{self.name}$" + return self.name + @property def properties(self) -> dict[str, Any]: """Get column properties.""" diff --git a/pandera/api/pandas/container.py b/pandera/api/pandas/container.py index 21406e590..55aa1dcc0 100644 --- a/pandera/api/pandas/container.py +++ b/pandera/api/pandas/container.py @@ -209,15 +209,24 @@ def _validate( @staticmethod def register_default_backends(check_obj_cls: type): from pandera.backends.pandas.register import register_pandas_backends + from pandera.config import CONFIG + + use_narwhals_backend = CONFIG.use_narwhals_backend _cls = check_obj_cls try: - register_pandas_backends(f"{_cls.__module__}.{_cls.__name__}") + register_pandas_backends( + f"{_cls.__module__}.{_cls.__name__}", + use_narwhals_backend=use_narwhals_backend, + ) except BackendNotFoundError: for base_cls in _cls.__bases__: base_cls_name = f"{base_cls.__module__}.{base_cls.__name__}" try: - register_pandas_backends(base_cls_name) + register_pandas_backends( + base_cls_name, + use_narwhals_backend=use_narwhals_backend, + ) except BackendNotFoundError: pass diff --git a/pandera/backends/narwhals/base.py b/pandera/backends/narwhals/base.py index a1bb11e12..0b8bce082 100644 --- a/pandera/backends/narwhals/base.py +++ b/pandera/backends/narwhals/base.py @@ -8,9 +8,18 @@ import narwhals.stable.v1 as nw from pandera.api.narwhals.error_handler import ErrorHandler -from pandera.api.narwhals.utils import _is_lazy, _is_sql_lazy, _materialize +from pandera.api.narwhals.utils import ( + _EAGER_PANDAS_LIKE_IMPLEMENTATIONS, + _is_lazy, + _is_pandas_like, + _is_sql_lazy, + _materialize, +) from pandera.backends.base import BaseSchemaBackend, CoreCheckResult -from pandera.backends.narwhals.checks import NarwhalsCheckBackend +from pandera.backends.narwhals.checks import ( + NarwhalsCheckBackend, + _use_input_nullness, +) from pandera.constants import CHECK_OUTPUT_KEY from pandera.errors import ( FailureCaseMetadata, @@ -25,6 +34,19 @@ pl = None # type: ignore[assignment] +def _lit_nullable_int32(value, implementation) -> Any: + """Int32 literal that tolerates ``None`` on pandas-like backends. + + ``nw.lit(None).cast(nw.Int32)`` raises on pandas-like implementations + (numpy cannot astype ``None`` to int32). Fall back to a Float64 cast so + the column materializes as float64 with NaN — the idiomatic pandas + representation of a missing integer. + """ + if value is None and implementation in _EAGER_PANDAS_LIKE_IMPLEMENTATIONS: + return nw.lit(None).cast(nw.Float64) + return nw.lit(value).cast(nw.Int32) + + def _check_identifier(err: SchemaError) -> Any: """Derive a short, human-readable identifier for the Check on an error.""" if err.check is None: @@ -38,15 +60,22 @@ def _check_identifier(err: SchemaError) -> Any: return str(err.check) -def _concat_failure_cases(items: list) -> Any: +def _concat_failure_cases(items: list, implementation=None) -> Any: """Concatenate per-error failure-case frames into a single frame. Items are one of: - ``nw.DataFrame`` / ``nw.LazyFrame`` — from ``_build_lazy_failure_case`` - (Polars LazyFrame, Ibis, PySpark). Dispatch on ``item.implementation``. + (Polars LazyFrame, Ibis, PySpark, pandas). Dispatch on + ``item.implementation``. - ``pl.DataFrame`` — from ``_build_eager_failure_case`` and ``_build_scalar_failure_case`` (eager Polars path). + ``implementation`` is the ``nw.Implementation`` of the frame that was + validated (when known). It disambiguates the all-scalar case — when every + failure case is a Python scalar there is no narwhals-wrapped item to + dispatch on, and pandas-like validations must still return a pandas + failure-cases frame instead of the historical polars default. + For PySpark-backed Narwhals frames: unwrap to native PySpark DataFrames and union via ``pyspark.sql.DataFrame.union()``. Scalar ``pl.DataFrame`` items from ``_build_scalar_failure_case`` cannot be converted to PySpark @@ -58,10 +87,20 @@ def _concat_failure_cases(items: list) -> Any: are present; collects and merges eager ``pl.DataFrame`` items (from ``_build_eager_failure_case`` / ``_build_scalar_failure_case``) when both are present — both sources can coexist in a single polars validation run. + For pandas-like Narwhals frames (pandas, modin, cuDF): unwrap to native + frames and concatenate via ``pandas.concat``; scalar items are converted + to pandas frames. For native ``pl.DataFrame`` items: ``pl.concat``. - Returns an empty ``pl.DataFrame`` if the collection is empty. + Returns an empty ``pl.DataFrame`` (or ``pandas.DataFrame`` for + pandas-like validations) if the collection is empty. """ + is_pandas_like = implementation in _EAGER_PANDAS_LIKE_IMPLEMENTATIONS + if not items: + if is_pandas_like: # pragma: no cover — defensive default + import pandas as pd + + return pd.DataFrame() return pl.DataFrame() if pl is not None else None # pragma: no cover # Separate Narwhals-wrapped items from native Polars items @@ -134,11 +173,43 @@ def _concat_failure_cases(items: list) -> Any: ) return pl.concat([eager_result] + pl_items) return lazy_result + elif first_nw.implementation in _EAGER_PANDAS_LIKE_IMPLEMENTATIONS: + # pandas-like path (pandas, modin, cuDF): materialize the + # Narwhals items to native frames and concatenate with pandas. + # Scalar failure-case items (pl_items, built by + # _build_scalar_failure_case) are converted via to_dict — no + # Arrow roundtrip, so pyarrow is not required. + import pandas as pd # local import: module must stay pandas-free + + native_items = [ + nw.to_native(_materialize(item)) for item in nw_items + ] + for item in pl_items: + if pl is not None and isinstance(item, pl.DataFrame): + item = pd.DataFrame(item.to_dict(as_series=False)) + native_items.append(item) + return pd.concat(native_items, ignore_index=True) else: # SQL-lazy path (Ibis, DuckDB, etc.): unwrap to native and union. native_items = [nw.to_native(item) for item in nw_items] return functools.reduce(lambda a, b: a.union(b), native_items) + if is_pandas_like: + # All-scalar pandas-like path: every failure case was a Python + # scalar (e.g. wrong-dtype or missing-column errors), so there is no + # narwhals-wrapped item to dispatch on — build a pandas frame. + import pandas as pd # local import: module must stay pandas-free + + converted = [ + ( + pd.DataFrame(item.to_dict(as_series=False)) + if pl is not None and isinstance(item, pl.DataFrame) + else item + ) + for item in pl_items + ] + return pd.concat(converted, ignore_index=True) + # All-Polars path: pl.DataFrame items from eager/scalar builders return pl.concat(pl_items) if pl is not None else None # pragma: no cover @@ -150,6 +221,43 @@ class NarwhalsSchemaBackend(BaseSchemaBackend): DataFrameSchemaBackend (container.py). """ + @staticmethod + def is_native_delegated_check(check) -> bool: + """True if a check must run through the native pandas check backend. + + ``Hypothesis`` checks rely on scipy statistical tests over grouped + samples with no Narwhals-expression equivalent, so for eager pandas + frames they are delegated to the native pandas hypothesis backend. + (``groupby`` column-check-groups are handled natively by the Narwhals + check backend — see ``NarwhalsCheckBackend.apply_groupby``.) + """ + from pandera.api.hypotheses import Hypothesis + + return isinstance(check, Hypothesis) + + def run_native_check( + self, check_obj, schema, check, check_index, column_name=None + ) -> CoreCheckResult: + """Run a single check via the native pandas check backend. + + Used for ``Hypothesis`` checks on eager pandas frames. ``check_obj`` is + a Narwhals frame; it is unwrapped to the native pandas frame so the + check dispatches to the native pandas hypothesis backend (which stays + registered for ``pd.DataFrame`` under the override). Returns the native + ``CoreCheckResult`` unchanged — the Narwhals error-collection loop + already understands it. + """ + from pandera.api.narwhals.utils import _to_native + from pandera.backends.pandas.base import PandasSchemaBackend + + native_obj = _to_native(check_obj) + args = () if column_name is None else (column_name,) + # run_check does not rely on backend instance state, so a bare native + # backend instance is enough to reuse its implementation. + return PandasSchemaBackend().run_check( + native_obj, schema, check, check_index, *args + ) + def subsample( self, check_obj, @@ -229,10 +337,24 @@ def run_check(self, check_obj, schema, check, check_index, *args): expr.alias(CHECK_OUTPUT_KEY) ) if check.ignore_na: - check_col = check_col.with_columns( + na_pass = ( nw.col(CHECK_OUTPUT_KEY) | nw.col(CHECK_OUTPUT_KEY).is_null() ) + checked_key = ( + check_result.checked_object.key + if check_result.checked_object is not None + else None + ) + if _use_input_nullness( + check.ignore_na, check_col, checked_key + ): + # pandas-like: NaN comparisons yield False, not + # null — OR in the input column's nullness. + na_pass = na_pass | nw.col(checked_key).is_null() + check_col = check_col.with_columns( + na_pass.alias(CHECK_OUTPUT_KEY) + ) fc = check_col.filter(~nw.col(CHECK_OUTPUT_KEY)) if check_result.checked_object is not None: key = check_result.checked_object.key @@ -297,6 +419,36 @@ def failure_cases_metadata( """ failure_case_collection: list = [] + # Implementation of the validated frame — used to disambiguate the + # all-scalar failure-cases path in _concat_failure_cases (e.g. pandas + # validations must return pandas frames even when every failure case + # is a Python scalar). ``err.data`` is nulled by the lazy + # ErrorHandler to avoid holding frame copies, so detection falls + # back to the schema class: pandas-API schemas are only routed to + # this backend for pandas frames. + data_implementation = None + for err in schema_errors: + data = getattr(err, "data", None) + if data is not None: + if not isinstance(data, (nw.DataFrame, nw.LazyFrame)): + try: + data = nw.from_native( + data, eager_or_interchange_only=False + ) + except TypeError: + data = None + if data is not None: + data_implementation = data.implementation + break + schema_mod = ( + type(err.schema).__module__ if err.schema is not None else "" + ) + if schema_mod.startswith( + ("pandera.api.pandas", "pandera._pandas_deprecated") + ): + data_implementation = nw.Implementation.PANDAS + break + for err in schema_errors: check_identifier = _check_identifier(err) @@ -309,7 +461,13 @@ def failure_cases_metadata( except TypeError: pass - if isinstance(fc, (nw.LazyFrame, nw.DataFrame)) and _is_lazy(fc): + # pandas-like frames take the narwhals-native builder as well: + # the eager builder below is a polars-specific path (Arrow + # roundtrip into pl.DataFrame) that would convert pandas failure + # cases to polars and leak the pandas index into the output. + if isinstance(fc, (nw.LazyFrame, nw.DataFrame)) and ( + _is_lazy(fc) or _is_pandas_like(fc) + ): failure_case_collection.append( self._build_lazy_failure_case(fc, err, check_identifier) ) @@ -322,7 +480,9 @@ def failure_cases_metadata( self._build_scalar_failure_case(err, check_identifier) ) - failure_cases = _concat_failure_cases(failure_case_collection) + failure_cases = _concat_failure_cases( + failure_case_collection, data_implementation + ) error_handler = ErrorHandler() # Only collect errors with a valid reason_code; errors without one @@ -378,8 +538,10 @@ def _build_lazy_failure_case(fc, err: SchemaError, check_identifier): nw.lit(err.schema.__class__.__name__).alias("schema_context"), nw.lit(err.schema.name).alias("column"), nw.lit(check_identifier).alias("check"), - nw.lit(err.check_index).cast(nw.Int32).alias("check_number"), - nw.lit(None).cast(nw.Int32).alias("index"), + _lit_nullable_int32(err.check_index, fc.implementation).alias( + "check_number" + ), + _lit_nullable_int32(None, fc.implementation).alias("index"), ) # Return narwhals-wrapped frame — _concat_failure_cases dispatches on # item.implementation to handle PySpark vs ibis vs polars without @@ -446,8 +608,13 @@ def _build_eager_failure_case(fc, err: SchemaError, check_identifier): @staticmethod def _build_scalar_failure_case(err: SchemaError, check_identifier): - """Build a failure-case frame for Python scalars/strings/None.""" - assert pl is not None, "polars is required for scalar failure_cases" + """Build a failure-case frame for Python scalars/strings/None. + + Returns a ``pl.DataFrame`` when polars is installed (the historical + behavior; downstream ``_concat_failure_cases`` branches convert as + needed). On polars-free installs (e.g. pandas + narwhals only) a + ``pandas.DataFrame`` is built instead. + """ scalar_failure_cases: dict = defaultdict(list) scalar_failure_cases["failure_case"].append(err.failure_cases) scalar_failure_cases["schema_context"].append( @@ -457,6 +624,12 @@ def _build_scalar_failure_case(err: SchemaError, check_identifier): scalar_failure_cases["check"].append(check_identifier) scalar_failure_cases["check_number"].append(err.check_index) scalar_failure_cases["index"].append(None) + if pl is None: + # polars-free install (e.g. pandas + narwhals only): build the + # scalar failure-case frame with pandas instead. + import pandas as pd + + return pd.DataFrame(dict(scalar_failure_cases)) return pl.DataFrame(scalar_failure_cases).cast( { "check_number": pl.Int32, @@ -486,15 +659,18 @@ def drop_invalid_rows(self, check_obj, error_handler): if not errors: return check_obj - # Collect (col_name, pass_expr) pairs where pass_expr returns True for valid rows. + # Collect (col_name, pass_expr, check, selector) tuples where + # pass_expr returns True for valid rows and selector identifies the + # checked input column (None for dataframe-level checks). pass_exprs = [] for i, err in enumerate(errors): co = err.check_output col_name = f"__check_output_{i}__" + selector = getattr(err.schema, "selector", None) if isinstance(co, nw.Expr): # DATAFRAME_CHECK path: True=pass. Apply ignore_na is handled later. - pass_exprs.append((col_name, co, err.check)) + pass_exprs.append((col_name, co, err.check, selector)) elif ( isinstance(co, (nw.LazyFrame, nw.DataFrame)) and CHECK_OUTPUT_KEY in co.collect_schema().names() @@ -503,34 +679,36 @@ def drop_invalid_rows(self, check_obj, error_handler): and hasattr(err.schema, "selector") ): # check_nullable path: True=null (failing). Reconstruct as ~is_null(). - selector = err.schema.selector not_null_expr = ~nw.col(selector).is_null() - pass_exprs.append((col_name, not_null_expr, None)) + pass_exprs.append((col_name, not_null_expr, None, selector)) if not pass_exprs: return check_obj frame = nw.from_native(check_obj, eager_or_interchange_only=False) - bool_cols = [col_name for col_name, _, _ in pass_exprs] + bool_cols = [col_name for col_name, _, _, _ in pass_exprs] # Build wide frame: single with_columns call for all exprs. wide = frame.with_columns( - [expr.alias(col_name) for col_name, expr, _ in pass_exprs] + [expr.alias(col_name) for col_name, expr, _, _ in pass_exprs] ) # Apply ignore_na at column level for expr-based checks (avoids ibis SQL issues). - ignore_na_cols = [ - col_name - for col_name, _, check in pass_exprs + ignore_na_specs = [ + (col_name, selector) + for col_name, _, check, selector in pass_exprs if check is not None and getattr(check, "ignore_na", False) ] - if ignore_na_cols: - wide = wide.with_columns( - [ - (nw.col(c) | nw.col(c).is_null()).alias(c) - for c in ignore_na_cols - ] - ) + if ignore_na_specs: + na_exprs = [] + for c, selector in ignore_na_specs: + na_pass = nw.col(c) | nw.col(c).is_null() + if _use_input_nullness(True, wide, selector): + # pandas-like: NaN comparisons yield False, not null — + # OR in the input column's nullness. + na_pass = na_pass | nw.col(selector).is_null() + na_exprs.append(na_pass.alias(c)) + wide = wide.with_columns(na_exprs) filtered = wide.filter( nw.all_horizontal(*[nw.col(c) for c in bool_cols]) diff --git a/pandera/backends/narwhals/checks.py b/pandera/backends/narwhals/checks.py index a5b83c915..90e4661c7 100644 --- a/pandera/backends/narwhals/checks.py +++ b/pandera/backends/narwhals/checks.py @@ -9,9 +9,32 @@ from pandera.api.base.checks import CheckResult from pandera.api.checks import Check from pandera.api.narwhals.types import NarwhalsData +from pandera.api.narwhals.utils import _EAGER_PANDAS_LIKE_IMPLEMENTATIONS from pandera.backends.base import BaseCheckBackend from pandera.constants import CHECK_OUTPUT_KEY +# Temporary column carrying the input column's nullness for pandas-like +# ignore_na handling (see _use_input_nullness). +_INPUT_NULL_KEY = f"{CHECK_OUTPUT_KEY}_input_null" + + +def _use_input_nullness(ignore_na: bool, frame: Any, key: Any) -> bool: + """Whether ``ignore_na`` must also consult the input column's nullness. + + On polars/ibis, null inputs propagate to null check outputs, so ORing the + output with its own nullness suffices. On pandas-like backends, NaN + inputs produce ``False`` comparison outputs (never null), so the input + column's nullness must be ORed in as well. Only column-level checks + (``key`` not None/"*") can attribute nullness to an input column. + """ + if not ignore_na or not key or key == "*": + return False + return ( + getattr(frame, "implementation", None) + in _EAGER_PANDAS_LIKE_IMPLEMENTATIONS + ) + + try: import ibis # noqa: F401 import ibis.expr.types as ir @@ -84,6 +107,28 @@ def _is_ibis_native(frame: Any) -> bool: return isinstance(frame, ibis.Table) +# Root modules of pandas-like libraries. Compared against the first dotted +# segment of ``type(obj).__module__``: pandas < 3 reports e.g. +# "pandas.core.frame" while pandas >= 3 reports just "pandas" for its public +# classes, so a bare root comparison covers both. +_PANDAS_LIKE_MODULE_ROOTS = frozenset({"pandas", "modin", "cudf"}) + + +def _is_pandas_like_module(mod: str) -> bool: + """True if a ``__module__`` string belongs to a pandas-like library.""" + return mod.split(".", 1)[0] in _PANDAS_LIKE_MODULE_ROOTS + + +def _is_pandas_like_native(frame: Any) -> bool: + """Cheap pandas-like (pandas, modin, cuDF) detection. + + Mirrors ``_is_polars_native``: module duck-typing so that pandas + (and modin/cudf) never need to be imported on installs without them. + """ + mod = getattr(type(frame), "__module__", "") or "" + return _is_pandas_like_module(mod) + + def _wrap_native_frame_with_key(native_frame: Any, key: str | None) -> Any: """Wrap ``(native_frame, key)`` into a polars/ibis-style data container. @@ -113,6 +158,16 @@ def _wrap_native_frame_with_key(native_frame: Any, key: str | None) -> Any: return IbisData(table=native_frame, key=key) + if _is_pandas_like_native(native_frame): + # pandas-style user check functions receive exactly what the native + # pandas backend passes them: the Series for a column-level check and + # the DataFrame for a dataframe-level check. This keeps existing + # pandas checks (``pa.Check(lambda s: s > 0)``) working unchanged + # under the narwhals backend. + if key is None or key == "*": + return native_frame + return native_frame[key] + return None @@ -126,9 +181,91 @@ def __init__(self, check: Check): self.check = check self.check_fn = partial(check._check_fn, **check._check_kwargs) - def groupby(self, check_obj: nw.LazyFrame): - """Implements groupby behavior for check object.""" - raise NotImplementedError + def groupby(self, check_obj): + """Implements groupby behavior for check object. + + Column-check-groups (`groupby=`) are a pandas-semantic feature: the + user check function receives a ``dict`` mapping group key to the native + pandas ``Series``/``DataFrame`` for that group. Narwhals has no + equivalent, so for pandas-like eager frames we group the *native* frame + with pandas and build the same dict the native pandas backend produces. + + :param check_obj: a native pandas-like frame (already unwrapped). + """ + assert self.check.groupby is not None, "Check.groupby must be set." + if isinstance(self.check.groupby, (str, list)): + return check_obj.groupby(self.check.groupby) + return self.check.groupby(check_obj) + + @staticmethod + def _format_groupby_input(groupby_obj, groups): + """Format a pandas groupby object into a dict of group -> Series/df. + + Ported from :class:`~pandera.backends.pandas.checks.PandasCheckBackend` + so the narwhals backend runs column-check-groups without dispatching to + the pandas check backend. + """ + if groups is None: + return { + (k if isinstance(k, bool) else k[0] if len(k) == 1 else k): v + for k, v in groupby_obj + } + group_keys = {k[0] if len(k) == 1 else k for k, _ in groupby_obj} + invalid_groups = [g for g in groups if g not in group_keys] + if invalid_groups: + raise KeyError( + f"groups {invalid_groups} provided in `groups` argument not a " + f"valid group key. Valid group keys: {group_keys}" + ) + output = {} + for group_key, group in groupby_obj: + if isinstance(group_key, tuple) and len(group_key) == 1: + group_key = group_key[0] + if group_key in groups: + output[group_key] = group + return output + + def apply_groupby(self, check_obj: NarwhalsData): + """Run a `groupby=` check by building the pandas group dict. + + Only supported for pandas-like eager frames. The user check function + receives the same ``dict`` it would under the native pandas backend and + typically returns a scalar boolean (or a per-group Series/dict of + booleans, which is reduced to a single pass/fail). + """ + native = nw.to_native(check_obj.frame) + if hasattr(native, "collect"): # pragma: no cover — lazy safety net + native = native.collect() + if not _is_pandas_like_native(native): + raise NotImplementedError( + "groupby checks (column check groups) are only supported for " + "pandas-like frames under the narwhals backend." + ) + key = check_obj.key + grouped = self.groupby(native) + if key and key != "*": + grouped = grouped[key] + group_dict = self._format_groupby_input(grouped, self.check.groups) + return self._reduce_groupby_output(self.check_fn(group_dict)) + + @staticmethod + def _reduce_groupby_output(out): + """Reduce a groupby-check output to a scalar boolean. + + Accepts a Python/numpy scalar bool, a numpy boolean array, a pandas + ``Series`` of booleans, or a ``dict`` of group -> bool, and returns a + single pass/fail bool (a group check passes only if every group passes). + """ + if isinstance(out, dict): + return all(bool(v) for v in out.values()) + mod = getattr(type(out), "__module__", "") or "" + if mod.startswith("numpy"): + if getattr(out, "ndim", 0) == 0: + return bool(out) + return bool(out.all()) + if _is_pandas_like_module(mod) and type(out).__name__ == "Series": + return bool(out.all()) + return out def query(self, check_obj: nw.LazyFrame): """Implements querying behavior to produce subset of check object.""" @@ -147,6 +284,11 @@ def apply(self, check_obj: NarwhalsData): frame = check_obj.frame key = check_obj.key + # Column-check-groups: build the pandas group dict and run the check + # on it (narwhals-native handling for pandas-like frames). + if self.check.groupby is not None: + return self.apply_groupby(check_obj) + if self.check.element_wise: selector = nw.col(key or "*") try: @@ -297,6 +439,50 @@ def _normalize_native_output(out, check_obj: NarwhalsData): native.with_columns(bool_col), eager_only=True ) + # Handle pandas-like native return types (pandas, modin, cuDF) from + # native=True checks: boolean Series / DataFrame outputs are attached + # to the original frame as CHECK_OUTPUT_KEY, mirroring the polars + # handling above. Detection is module based so pandas is not + # imported on pandas-free installs. + if _is_pandas_like_module(out_mod): + out_type_name = type(out).__name__ + native = nw.to_native(check_obj.frame) + + if out_type_name == "Series": + bool_series = out + elif out_type_name == "DataFrame": + if CHECK_OUTPUT_KEY in out.columns: + bool_series = out[CHECK_OUTPUT_KEY] + else: + # Multi-column boolean output — AND-reduce to a single + # check-output column (matches the native pandas backend + # and the polars handling above). + bool_series = out.all(axis=1) + else: # pragma: no cover — unexpected pandas-like type + return out + + native = native.assign(**{CHECK_OUTPUT_KEY: bool_series}) + return nw.from_native(native, eager_only=True) + + # Handle numpy return types from pandas-style checks: + # - 0-d bool scalars (e.g. ``(series > 0).all()`` returns np.bool_) + # → plain Python bool, handled by postprocess_bool_output. + # - 1-d boolean arrays → attached positionally as CHECK_OUTPUT_KEY. + if out_mod.startswith("numpy"): + out_type_name = type(out).__name__ + # numpy 1.x names the scalar bool type "bool_"; numpy 2.x "bool". + if out_type_name in ("bool_", "bool"): + return bool(out) + if ( + out_type_name == "ndarray" + and getattr(out, "dtype", None) is not None + and out.dtype.kind == "b" + ): + native = nw.to_native(check_obj.frame) + if hasattr(native, "assign"): + native = native.assign(**{CHECK_OUTPUT_KEY: out}) + return nw.from_native(native, eager_only=True) + return out # bool or other scalar — handled by postprocess_bool_output def postprocess(self, check_obj: NarwhalsData, check_output): @@ -334,15 +520,30 @@ def postprocess_expr_output( as expr | expr.is_null() — the latter causes ibis to produce incorrect SQL (isnull() on an expression before binding returns True for all rows on some SQL backends, because the expression is treated as nullable). + + For pandas-like frames, ignore_na additionally ORs in the *input* + column's nullness: pandas comparisons on NaN yield False instead of + propagating null (unlike polars/ibis), so output-nullness alone would + treat missing values as check failures. """ frame = check_obj.frame + key = check_obj.key # Evaluate expr to a single-column frame, then apply ignore_na on # the concrete column values where is_null() works correctly. - check_col = frame.select(expr.alias(CHECK_OUTPUT_KEY)) + select_exprs = [expr.alias(CHECK_OUTPUT_KEY)] + include_input_null = _use_input_nullness( + self.check.ignore_na, frame, key + ) + if include_input_null: + select_exprs.append(nw.col(key).is_null().alias(_INPUT_NULL_KEY)) + check_col = frame.select(*select_exprs) if self.check.ignore_na: - check_col = check_col.with_columns( + na_pass = ( nw.col(CHECK_OUTPUT_KEY) | nw.col(CHECK_OUTPUT_KEY).is_null() ) + if include_input_null: + na_pass = na_pass | nw.col(_INPUT_NULL_KEY) + check_col = check_col.with_columns(na_pass.alias(CHECK_OUTPUT_KEY)) passed = check_col.select(nw.col(CHECK_OUTPUT_KEY).all()) return CheckResult( check_output=expr, # Store ONLY the expr — failure_cases deferred @@ -359,9 +560,19 @@ def postprocess_lazyframe_output( """Postprocesses LazyFrame check output into a CheckResult.""" # check_output is the wide table (frame + CHECK_OUTPUT_KEY column). Stay lazy. if self.check.ignore_na: - check_output = check_output.with_columns( + na_pass = ( nw.col(CHECK_OUTPUT_KEY) | nw.col(CHECK_OUTPUT_KEY).is_null() ) + if _use_input_nullness( + self.check.ignore_na, check_output, check_obj.key + ): + # pandas-like: NaN comparisons yield False, not null — OR in + # the input column's nullness (the wide table still carries + # the original columns). + na_pass = na_pass | nw.col(check_obj.key).is_null() + check_output = check_output.with_columns( + na_pass.alias(CHECK_OUTPUT_KEY) + ) passed = check_output.select(nw.col(CHECK_OUTPUT_KEY).all()) failure_cases = check_output.filter(~nw.col(CHECK_OUTPUT_KEY)) diff --git a/pandera/backends/narwhals/components.py b/pandera/backends/narwhals/components.py index c58a60541..3ee5c509a 100644 --- a/pandera/backends/narwhals/components.py +++ b/pandera/backends/narwhals/components.py @@ -11,6 +11,8 @@ from pandera.api.base.error_handler import get_error_category from pandera.api.narwhals.error_handler import ErrorHandler from pandera.api.narwhals.utils import ( + _EAGER_PANDAS_LIKE_IMPLEMENTATIONS, + _is_pandas_like, _materialize, _to_native, _unwrap_failure_cases, @@ -140,7 +142,18 @@ def validate( return check_lf def get_regex_columns(self, schema, check_obj) -> Iterable: - """Get column names matching a regex pattern.""" + """Get column names matching a regex pattern. + + Accepts either a Narwhals frame or a native frame: native pandas frames + arrive here when native pandas backend code (e.g. ``coerce_dtype``) + dispatches regex handling through this backend under the narwhals + override, since ``Column`` is registered to this backend for + ``pd.DataFrame``. + """ + if not isinstance(check_obj, (nw.DataFrame, nw.LazyFrame)): + check_obj = nw.from_native( + check_obj, eager_or_interchange_only=False + ) frame_cols = check_obj.collect_schema().names() return [c for c in frame_cols if re.search(schema.selector, c)] @@ -282,6 +295,17 @@ def check_dtype(self, check_obj, schema) -> list[CoreCheckResult]: except ImportError: uses_pyspark_dtype = False + try: + from pandera.engines import numpy_engine as _numpy_engine + from pandera.engines import pandas_engine as _pandas_engine + + uses_pandas_dtype = isinstance( + schema.dtype, + (_pandas_engine.DataType, _numpy_engine.DataType), + ) + except ImportError: # pragma: no cover — pandas-free installs only + uses_pandas_dtype = False + results = [] schema_obj = check_obj.select(schema.selector).collect_schema() @@ -289,6 +313,25 @@ def check_dtype(self, check_obj, schema) -> list[CoreCheckResult]: nw.to_native(check_obj).schema if uses_pyspark_dtype else None ) + # Schema configured with a pandas/numpy dtype on a pandas-like frame: + # compare against the *native* pandas dtypes instead of the narwhals + # ones. Narwhals normalizes pandas dtypes (object and string[python] + # both map to nw.String; int64 and nullable Int64 both map to + # nw.Int64), which would erase distinctions the pandas engine + # preserves — e.g. Column(str) must accept object columns, and native + # semantics for nullable extension dtypes must be kept. Like the + # pyspark branch above, this dispatch is schema-driven (isinstance on + # schema.dtype); the frame check guards the cross-engine case where a + # pandas dtype is used with a non-pandas frame. + native_pandas_dtypes = None + if ( + uses_pandas_dtype + and check_obj.implementation in _EAGER_PANDAS_LIKE_IMPLEMENTATIONS + ): + native_pandas_dtypes = nw.to_native( + check_obj.select(schema.selector) + ).dtypes + for column, nw_dtype in zip(schema_obj.names(), schema_obj.dtypes()): if uses_pyspark_dtype: # Schema configured with a PySpark dtype (e.g. T.IntegerType()) — @@ -317,9 +360,41 @@ def check_dtype(self, check_obj, schema) -> list[CoreCheckResult]: if not passed else None ), - failure_cases=pyspark_dtype_str - if not passed - else None, + failure_cases=( + pyspark_dtype_str if not passed else None + ), + ) + ) + continue + + if native_pandas_dtypes is not None: + # pandas-like frame with a pandas/numpy schema dtype: compare + # native dtypes through the pandas engine so native semantics + # are preserved (see native_pandas_dtypes comment above). + native_dtype = native_pandas_dtypes[column] + try: + actual_dtype = _pandas_engine.Engine.dtype(native_dtype) + except TypeError: # pragma: no cover — unknown native dtype + actual_dtype = native_dtype + dtype_check_results = schema.dtype.check(actual_dtype) + if isinstance(dtype_check_results, bool): + passed = dtype_check_results + else: + passed = all(dtype_check_results) + results.append( + CoreCheckResult( + passed=bool(passed), + check=f"dtype('{schema.dtype}')", + reason_code=SchemaErrorReason.WRONG_DATATYPE, + message=( + f"expected column '{column}' to have type " + f"{schema.dtype}, got {native_dtype}" + if not passed + else None + ), + failure_cases=( + str(native_dtype) if not passed else None + ), ) ) continue @@ -369,11 +444,30 @@ def run_checks(self, check_obj, schema) -> list[CoreCheckResult]: check_results: list[CoreCheckResult] = [] for check_index, check in enumerate(schema.checks): try: - check_results.append( - self.run_check( - check_obj, schema, check, check_index, schema.selector + if self.is_native_delegated_check(check) and _is_pandas_like( + check_obj + ): + # Hypothesis / groupby checks on pandas frames run through + # the native pandas backend (see run_native_check). + check_results.append( + self.run_native_check( + check_obj, + schema, + check, + check_index, + schema.name, + ) + ) + else: + check_results.append( + self.run_check( + check_obj, + schema, + check, + check_index, + schema.selector, + ) ) - ) except Exception as err: err_msg = f'"{err.args[0]}"' if err.args else "" msg = f"{err.__class__.__name__}({err_msg})" diff --git a/pandera/backends/narwhals/container.py b/pandera/backends/narwhals/container.py index 709534a7b..6c0393a45 100644 --- a/pandera/backends/narwhals/container.py +++ b/pandera/backends/narwhals/container.py @@ -3,6 +3,7 @@ from __future__ import annotations import copy +import functools import re import traceback import warnings @@ -14,6 +15,7 @@ from pandera.api.base.error_handler import get_error_category from pandera.api.narwhals.error_handler import ErrorHandler from pandera.api.narwhals.utils import ( + _is_pandas_like, _materialize, _to_native, _unwrap_failure_cases, @@ -42,6 +44,40 @@ from pandera.validation_depth import validate_scope +def _native_is_pandas_like(obj) -> bool: + """True if a *native* (un-wrapped) frame is a pandas-like DataFrame. + + Detects pandas / modin / cudf frames from the class module prefix so we + can decide — before wrapping into Narwhals — whether the pandas-specific + pre-wrap steps (custom parsers, index coercion, column-name uniqueness) + should run. Narwhals wrapping raises on duplicate column labels, so this + must not rely on wrapping the frame. + """ + module = type(obj).__module__ + return module.startswith(("pandas", "modin", "cudf")) + + +def _narwhals_target_dtype(pandera_dtype): + """Map a pandera ``DataType`` to a Narwhals dtype for ``cast``. + + Returns the Narwhals dtype (e.g. ``nw.Int64``) or ``None`` when the pandera + dtype has no Narwhals equivalent (e.g. some pandas extension dtypes). A + ``None`` result means the column is left as-is — the subsequent dtype check + reports any mismatch. This is the documented fidelity limitation of the + Narwhals-native coerce path. + """ + if pandera_dtype is None: + return None + from pandera.engines import narwhals_engine + + for probe in (pandera_dtype, str(pandera_dtype)): + try: + return narwhals_engine.Engine.dtype(probe).type + except (TypeError, ValueError, SystemError): + continue + return None + + def _to_lazy_nw(check_obj) -> nw.LazyFrame: """Wrap any supported native frame as a Narwhals LazyFrame.""" wrapped = nw.from_native(check_obj, eager_or_interchange_only=False) @@ -100,14 +136,77 @@ def validate( # Capture the input type so we can return the same type return_type = type(check_obj) - # Convert to Narwhals LazyFrame — all parsers operate on LazyFrame - check_lf = _to_lazy_nw(check_obj) - if inplace: warnings.warn("setting inplace=True will have no effect.") error_handler = ErrorHandler(lazy) + is_pandas_native = _native_is_pandas_like(check_obj) + + # --- pandas-only pre-wrap steps --------------------------------------- + # These operate on the *native* pandas frame before it is wrapped into + # Narwhals, because Narwhals has no parser step and rejects duplicate + # column labels at wrap time. + + # Run custom parsers (schema.parsers) on the native pandas frame + # (self-contained — the parser fn is applied directly here, no dispatch + # to the pandas schema backend). Narwhals preserves the resulting frame + # through wrapping. + if is_pandas_native and getattr(schema, "parsers", None): + check_obj = self.run_native_parsers(check_obj, schema) + + # Index/MultiIndex coercion is the one delegation to the native pandas + # backend (Narwhals has no index concept). Coerce the index on the + # native frame pre-wrap so the coerced index propagates through + # Narwhals (which preserves the pandas index) to the output. + if is_pandas_native and getattr(schema, "index", None) is not None: + check_obj = self.coerce_native_index( + check_obj, schema, error_handler + ) + + # add_missing_columns, set_default, and column/schema coercion are + # applied as Narwhals-native core parsers (see below) — no delegation + # to the pandas schema backend. + + # Column-name uniqueness must be detected on the native frame: Narwhals + # raises ``DuplicateError`` when wrapping a frame with duplicate labels, + # so the check cannot run after wrapping. + if is_pandas_native: + dup_error = self.check_native_column_names_unique( + check_obj, schema + ) + if dup_error is not None: + # collect_error raises immediately in non-lazy mode; in lazy + # mode we raise below because the frame cannot be wrapped. + error_handler.collect_error( + get_error_category(dup_error.reason_code), + dup_error.reason_code, + dup_error, + ) + raise SchemaErrors( + schema=schema, + schema_errors=error_handler.schema_errors, + data=check_obj, + ) + + # Convert to Narwhals LazyFrame — all parsers operate on LazyFrame + check_lf = _to_lazy_nw(check_obj) + + # Index/MultiIndex validation is a pandas-only concept. For eager + # pandas-like frames it is delegated to the native Index/MultiIndex + # backends in ``run_index_checks`` (Narwhals preserves the pandas + # index through its operations, so the parsed frame still carries it). + # For non-pandas frames that somehow carry an index component, warn and + # skip — Narwhals has no notion of an index there. + if getattr(schema, "index", None) is not None and not is_pandas_native: + warnings.warn( + "index validation is not supported by the narwhals backend " + "for non-pandas frames. The `index` component of this schema " + "will not be validated.", + SchemaWarning, + stacklevel=5, + ) + column_info = self.collect_column_info(check_lf, schema) if getattr(schema, "drop_invalid_rows", False) and not lazy: @@ -115,11 +214,13 @@ def validate( "When drop_invalid_rows is True, lazy must be set to True." ) - # The parsers list wires strict_filter_columns and coerce_dtype - # (for row-wise dtypes like PydanticModel). - # add_missing_columns and set_default are deferred to later releases. + # Core parsers, mirroring the native pandas ordering: + # add_missing_columns → strict_filter_columns → set_defaults → + # coerce_dtype. All are Narwhals-native (operate on the LazyFrame). core_parsers: list[tuple[Callable[..., Any], tuple[Any, ...]]] = [ + (self.add_missing_columns, (schema, column_info)), (self.strict_filter_columns, (schema, column_info)), + (self.set_defaults, (schema,)), (self.coerce_dtype, (schema,)), ] @@ -135,6 +236,10 @@ def validate( except SchemaErrors as exc: error_handler.collect_errors(exc.schema_errors) + # add_missing_columns may have added columns — regenerate column info + # so downstream presence checks and components see the current frame. + column_info = self.collect_column_info(check_lf, schema) + # collect schema components components = self.collect_schema_components( check_lf, schema, column_info @@ -162,6 +267,7 @@ def validate( self.run_schema_component_checks, (sample_lf, schema, components, lazy), ), + (self.run_index_checks, (check_lf, schema, lazy)), (self.run_checks, (sample_lf, schema)), ] @@ -232,9 +338,20 @@ def run_checks( check_results: list[CoreCheckResult] = [] for check_index, check in enumerate(schema.checks): try: - check_results.append( - self.run_check(check_obj, schema, check, check_index) - ) + if self.is_native_delegated_check(check) and _is_pandas_like( + check_obj + ): + # Hypothesis / groupby dataframe-level checks on pandas + # frames run through the native pandas backend. + check_results.append( + self.run_native_check( + check_obj, schema, check, check_index + ) + ) + else: + check_results.append( + self.run_check(check_obj, schema, check, check_index) + ) except SchemaDefinitionError: raise except Exception as err: @@ -299,6 +416,131 @@ def run_schema_component_checks( ) return check_results + def run_native_parsers(self, check_obj, schema): + """Run custom ``schema.parsers`` on the native pandas frame. + + Narwhals has no parser step, and custom parsers are arbitrary user code + written against the native pandas frame. The parser function is applied + directly here (self-contained — no dispatch to the pandas schema + backend): dataframe-level parsers receive the whole frame, and + ``element_wise`` parsers are applied row-wise. + """ + for parser in schema.parsers: + parser_fn = functools.partial( + parser._parser_fn, **parser._parser_kwargs + ) + if getattr(parser, "element_wise", False): + check_obj = check_obj.apply(parser_fn, axis=1) + else: + check_obj = parser_fn(check_obj) + return check_obj + + def coerce_native_index(self, check_obj, schema, error_handler): + """Coerce the pandas index dtype via the native Index component. + + Index/MultiIndex is the one pandas concept Narwhals cannot express, so + index coercion is delegated to the native pandas Index backend. Done on + the native frame before wrapping so the coerced index propagates + through Narwhals (which preserves the pandas index) to the output. + """ + index = schema.index + if not ( + getattr(index, "coerce", False) or getattr(schema, "coerce", False) + ): + return check_obj + + check_obj = check_obj.copy() + try: + check_obj.index = index.coerce_dtype(check_obj.index) + except SchemaError as exc: + error_handler.collect_error( + get_error_category(exc.reason_code), exc.reason_code, exc + ) + except SchemaErrors as exc: + error_handler.collect_errors(exc.schema_errors) + return check_obj + + def check_native_column_names_unique(self, check_obj, schema): + """Return a ``SchemaError`` if duplicate column labels are present. + + Mirrors the native pandas ``check_column_names_are_unique``. Runs on + the native pandas frame (before Narwhals wrapping) because Narwhals + rejects duplicate column labels at construction time. Returns ``None`` + when the check passes or ``unique_column_names`` is not set. + """ + if not getattr(schema, "unique_column_names", False): + return None + + columns = getattr(check_obj, "columns", None) + if columns is None: + return None + + failed = columns[columns.duplicated()] + if not failed.any(): + return None + + return SchemaError( + schema, + data=check_obj, + message=( + "dataframe contains multiple columns with label(s): " + f"{failed.tolist()}" + ), + failure_cases=failed, + check="dataframe_column_labels_unique", + reason_code=SchemaErrorReason.DUPLICATE_COLUMN_LABELS, + ) + + def run_index_checks( + self, + check_obj, + schema, + lazy: bool, + ) -> list[CoreCheckResult]: + """Validate the pandas ``index`` / ``MultiIndex`` component. + + Delegates to the native pandas Index/MultiIndex backends (which stay + registered even when the Narwhals backend is active). Narwhals + preserves the pandas index through its operations, so ``_to_native`` + of the parsed Narwhals frame still carries the original index. + + For non-pandas frames this is a no-op — those schemas have no ``index`` + component, and the ``validate`` method already warned if one is present. + """ + index_component = getattr(schema, "index", None) + if index_component is None: + return [] + + native_obj = _to_native(check_obj) + if not _native_is_pandas_like(native_obj): + return [] + + results: list[CoreCheckResult] = [] + try: + # inplace=True: the index is read-only for validation purposes here; + # any coercion applies to a copy the native backend manages. + index_component.validate(native_obj, lazy=lazy, inplace=True) + except SchemaError as err: + results.append( + CoreCheckResult( + passed=False, + check="index", + reason_code=err.reason_code, + schema_error=err, + ) + ) + except SchemaErrors as err: + results.extend( + CoreCheckResult( + passed=False, + check="index", + reason_code=schema_error.reason_code, + schema_error=schema_error, + ) + for schema_error in err.schema_errors + ) + return results + def collect_column_info(self, check_obj, schema): """Collect column metadata for the dataframe.""" # Use collect_schema().names() — lazy-safe Narwhals equivalent of @@ -342,42 +584,191 @@ def collect_column_info(self, check_obj, schema): ) def coerce_dtype(self, check_obj, schema): - """Coerce dataframe to schema.dtype for row-wise dtypes (e.g. PydanticModel). + """Coerce dtypes to the schema (Narwhals-native). + + Two cases are handled: + + * **Row-wise auto_coerce dtypes** (e.g. ``PydanticModel``): coerced by + the dtype engine itself over the whole frame — works for any backend. + * **Column- and schema-level dtypes**: coerced with ``nw.cast`` for + eager pandas-like frames. Narwhals normalizes pandas dtypes, so + native pandas dtype fidelity (nullable ``Int64``, ``Categorical``, + tz-aware datetimes) is not guaranteed. Cast failures are reported as + ``DATATYPE_COERCION`` errors. - Column-level dtypes are handled via schema components; this method only - acts when schema.dtype.auto_coerce is True (i.e. the dtype applies - row-wise and handles its own coercion over the whole frame). + Column-level coercion remains a no-op for non-pandas Narwhals backends + (a known gap). Accepts and returns either a Narwhals frame (validate + path) or a native frame (direct ``schema.coerce_dtype(df)`` calls). """ + was_wrapped = isinstance(check_obj, (nw.DataFrame, nw.LazyFrame)) + + # Row-wise auto_coerce dtypes (e.g. PydanticModel): the dtype engine + # coerces the whole frame. Self-contained (no pandas schema backend). if ( - schema.dtype is None - or not schema.coerce - or not getattr(schema.dtype, "auto_coerce", False) + schema.dtype is not None + and schema.coerce + and getattr(schema.dtype, "auto_coerce", False) ): + config_ctx = get_config_context(validation_depth_default=None) + coerce_fn = ( + "try_coerce" + if config_ctx.validation_depth + in (ValidationDepth.SCHEMA_AND_DATA, ValidationDepth.DATA_ONLY) + else "coerce" + ) + native_obj = _to_native(check_obj) + try: + coerced = getattr(schema.dtype, coerce_fn)(native_obj) + except ParserError as exc: + raise SchemaError( + schema=schema, + data=native_obj, + message=exc.args[0], + check=f"coerce_dtype('{schema.dtype}')", + reason_code=SchemaErrorReason.DATATYPE_COERCION, + failure_cases=exc.failure_cases, + check_output=exc.parser_output, + ) from exc + return _to_lazy_nw(coerced) if was_wrapped else coerced + + # Column-/schema-level coercion — pandas-like frames only. Hybrid: + # plain numpy dtypes are cast with ``nw.cast`` (Narwhals-native), while + # pandas extension dtypes (nullable Int64, Categorical, tz-aware + # datetimes, ``string``) fall back to the pandas dtype engine so native + # semantics are preserved. A Narwhals cast that raises also falls back. + if not _native_is_pandas_like(_to_native(check_obj)): return check_obj - config_ctx = get_config_context(validation_depth_default=None) - coerce_fn = ( - "try_coerce" - if config_ctx.validation_depth - in (ValidationDepth.SCHEMA_AND_DATA, ValidationDepth.DATA_ONLY) - else "coerce" - ) + lf = check_obj if was_wrapped else _to_lazy_nw(check_obj) + frame_cols = lf.collect_schema().names() - native_obj = _to_native(check_obj) - try: - coerced = getattr(schema.dtype, coerce_fn)(native_obj) - except ParserError as exc: - raise SchemaError( + # Build (column, pandera_dtype) pairs to coerce. + targets = self._collect_coerce_targets(schema, frame_cols) + if not targets: + return check_obj + + error_handler = ErrorHandler(lazy=True) + # Columns that must be coerced through the pandas dtype engine (either + # extension dtypes, or numpy dtypes whose Narwhals cast raised). + pandas_fallback: list[tuple[Any, Any]] = [] + + for col_name, col_dtype in targets: + target_nw = ( + _narwhals_target_dtype(col_dtype) + if self._prefers_narwhals_cast(col_dtype) + else None + ) + if target_nw is None: + pandas_fallback.append((col_name, col_dtype)) + continue + try: + casted = lf.with_columns(nw.col(col_name).cast(target_nw)) + # Force evaluation to surface cast errors (pandas is eager, so + # this is a cheap materialization). + _materialize(casted.select(nw.col(col_name))) + lf = casted + except Exception: # noqa: BLE001 — narwhals raises ValueError + # Narwhals could not cast faithfully — fall back to the pandas + # dtype engine, which produces a proper DATATYPE_COERCION error + # (with failing values) or the correct native dtype. + pandas_fallback.append((col_name, col_dtype)) + + if pandas_fallback: + lf = self._coerce_via_pandas_engine( + lf, pandas_fallback, schema, error_handler + ) + + if error_handler.collected_errors: + raise SchemaErrors( schema=schema, - data=native_obj, - message=exc.args[0], - check=f"coerce_dtype('{schema.dtype}')", - reason_code=SchemaErrorReason.DATATYPE_COERCION, - failure_cases=exc.failure_cases, - check_output=exc.parser_output, - ) from exc + schema_errors=error_handler.schema_errors, + data=_to_native(_materialize(lf)), + ) + + return lf if was_wrapped else _to_native(_materialize(lf)) - return _to_lazy_nw(coerced) + @staticmethod + def _prefers_narwhals_cast(pandera_dtype) -> bool: + """True if a dtype round-trips faithfully through ``nw.cast``. + + Plain numpy dtypes (int, float, str, bool) are cast Narwhals-native. + pandas extension dtypes (nullable ``Int64``, ``Categorical``, tz-aware + datetimes, ``string``) are handled by the pandas dtype engine instead. + """ + from pandera.engines import numpy_engine + + return isinstance(pandera_dtype, numpy_engine.DataType) + + def _collect_coerce_targets(self, schema, frame_cols): + """Resolve ``(column_name, pandera_dtype)`` pairs to coerce.""" + targets: list[tuple[Any, Any]] = [] + seen: set = set() + for col_name, col_schema in schema.columns.items(): + if not ( + getattr(col_schema, "coerce", False) + or getattr(schema, "coerce", False) + ): + continue + col_dtype = ( + schema.dtype if schema.dtype is not None else col_schema.dtype + ) + if col_dtype is None: + continue + if getattr(col_schema, "regex", False): + matches = [ + c + for c in frame_cols + if re.search(col_schema.selector, str(c)) + ] + elif col_name in frame_cols: + matches = [col_name] + else: + matches = [] + for c in matches: + if c not in seen: + targets.append((c, col_dtype)) + seen.add(c) + + # Schema-level dtype with no explicit columns: coerce every column. + if schema.dtype is not None and schema.coerce and not schema.columns: + for c in frame_cols: + if c not in seen: + targets.append((c, schema.dtype)) + seen.add(c) + return targets + + def _coerce_via_pandas_engine(self, lf, columns, schema, error_handler): + """Coerce ``columns`` through the pandas dtype engine (native fidelity). + + Used for pandas extension dtypes and as a fallback when ``nw.cast`` + cannot coerce a column. Operates on the native pandas frame (the + Narwhals frame is materialized first — pandas is eager, so this is + cheap) and preserves the index. Coercion failures are collected as + ``DATATYPE_COERCION`` errors with the offending values. + """ + native = nw.to_native(_materialize(lf)).copy() + for col_name, col_dtype in columns: + # ``try_coerce`` raises ``ParserError`` (with failing values) on + # failure, unlike ``coerce`` which surfaces a raw backend error. + try: + native[col_name] = col_dtype.try_coerce(native[col_name]) + except ParserError as exc: + error_handler.collect_error( + get_error_category(SchemaErrorReason.DATATYPE_COERCION), + SchemaErrorReason.DATATYPE_COERCION, + SchemaError( + schema=schema, + data=native, + message=( + f"Error while coercing '{col_name}' to type " + f"{col_dtype}: {exc}" + ), + failure_cases=exc.failure_cases, + check=f"coerce_dtype('{col_dtype}')", + reason_code=SchemaErrorReason.DATATYPE_COERCION, + ), + ) + return _to_lazy_nw(native) def collect_schema_components( self, @@ -430,10 +821,14 @@ def collect_schema_components( col.dtype = schema.dtype # type: ignore # Warn once per column when coerce=True was requested but will - # not be applied — the narwhals ColumnBackend has no coerce_dtype - # step, so column-level coerce=True is a no-op that would otherwise - # silently produce a WRONG_DATATYPE error. - if getattr(col, "coerce", False): + # not be applied — for non-pandas Narwhals backends the + # ColumnBackend has no coerce step, so column-level coerce=True + # is a no-op that would otherwise silently produce a + # WRONG_DATATYPE error. Pandas frames are coerced Narwhals-native + # in ``coerce_dtype``, so no warning is needed there. + if getattr(col, "coerce", False) and not _is_pandas_like( + check_obj + ): warn_col_name = getattr(col, "name", None) or getattr( col, "selector", col_name ) @@ -456,6 +851,150 @@ def collect_schema_components( # Parsers # ########### + @staticmethod + def _has_default(col_schema) -> bool: + """True if the column schema declares a (non-null) default value.""" + default = col_schema.default + # None or NaN (a float that is not equal to itself) means "no default". + return default is not None and not ( + isinstance(default, float) and default != default + ) + + def add_missing_columns(self, check_obj, schema, column_info: ColumnInfo): + """Add schema columns missing from the frame. + + Absent columns must either declare a default value or be nullable; + otherwise an ``ADD_MISSING_COLUMN_NO_DEFAULT`` error is raised. Missing + columns are inserted in schema order relative to the existing columns. + + Column construction is hybrid, mirroring the coerce path: plain numpy + dtypes with a concrete default are built Narwhals-native + (``nw.lit(value).cast(...)``); extension dtypes (nullable ``Int64`` etc.) + and null-valued columns are given their schema dtype through the pandas + dtype engine, since ``nw.cast`` cannot represent them (e.g. a null + integer). + """ + if not ( + column_info.absent_column_names + and getattr(schema, "add_missing_columns", False) + ): + return check_obj + + original_cols = check_obj.collect_schema().names() + + absent_errors: list[SchemaError] = [] + add_exprs = [] + # (col_name, pandera_dtype) for columns whose dtype must be fixed + # through the pandas dtype engine after the Narwhals literals are added. + native_dtype_fix: list[tuple[Any, Any]] = [] + for col_name in column_info.absent_column_names: + col_schema = schema.columns[col_name] + has_default = self._has_default(col_schema) + if not has_default and not col_schema.nullable: + absent_errors.append( + SchemaError( + schema=schema, + data=_to_native(check_obj), + message=( + f"column '{col_name}' in " + f"{schema.__class__.__name__} {schema.columns} " + "requires a default value when non-nullable " + "add_missing_columns is enabled" + ), + failure_cases=col_name, + check="add_missing_has_default", + reason_code=( + SchemaErrorReason.ADD_MISSING_COLUMN_NO_DEFAULT + ), + ) + ) + continue + value = col_schema.default if has_default else None + expr = nw.lit(value) + target_nw = ( + _narwhals_target_dtype(col_schema.dtype) + if value is not None + and self._prefers_narwhals_cast(col_schema.dtype) + else None + ) + if target_nw is not None: + expr = expr.cast(target_nw) + elif col_schema.dtype is not None: + # Extension dtype or null value: add the literal now, then give + # it the correct native dtype below. + native_dtype_fix.append((col_name, col_schema.dtype)) + add_exprs.append(expr.alias(col_name)) + + if absent_errors: + raise SchemaErrors( + schema=schema, + schema_errors=absent_errors, + data=_to_native(check_obj), + ) + + if add_exprs: + check_obj = check_obj.with_columns(add_exprs) + ordered = self._order_columns_with_missing( + original_cols, schema, column_info + ) + check_obj = check_obj.select(ordered) + + if native_dtype_fix: + native = nw.to_native(_materialize(check_obj)).copy() + for col_name, col_dtype in native_dtype_fix: + native[col_name] = col_dtype.try_coerce(native[col_name]) + check_obj = _to_lazy_nw(native) + return check_obj + + @staticmethod + def _order_columns_with_missing(original_cols, schema, column_info): + """Compute column order after inserting missing columns. + + Ports the native pandas ordering: missing schema columns are inserted + at their schema position relative to the existing columns, without + disturbing the order of columns already present. + """ + absent = column_info.absent_column_names + schema_cols: dict[Any, None] = {} + for col_name, col_schema in schema.columns.items(): + if col_name in original_cols or col_schema.required: + schema_cols[col_name] = None + + ordered: list[Any] = [] + for col_name in original_cols: + pop_cols = [] + for next_col_name in iter(schema_cols): + if next_col_name in absent and next_col_name not in ordered: + ordered.append(next_col_name) + pop_cols.append(next_col_name) + else: + for pop_col in pop_cols: + schema_cols.pop(pop_col) + break + ordered.append(col_name) + schema_cols.pop(col_name, None) + + for col_name in absent: + if col_name not in ordered: + ordered.append(col_name) + return ordered + + def set_defaults(self, check_obj, schema): + """Fill null values in columns that declare a default (Narwhals-native).""" + frame_cols = check_obj.collect_schema().names() + exprs = [] + for col_name, col_schema in schema.columns.items(): + if not self._has_default(col_schema) or col_name not in frame_cols: + continue + expr = nw.col(col_name).fill_null(col_schema.default) + target = _narwhals_target_dtype(col_schema.dtype) + if target is not None: + expr = expr.cast(target) + exprs.append(expr.alias(col_name)) + if exprs: + check_obj = check_obj.with_columns(exprs) + return check_obj + def strict_filter_columns( self, check_obj, diff --git a/pandera/backends/narwhals/register.py b/pandera/backends/narwhals/register.py index cbbdc2d18..c01926b7a 100644 --- a/pandera/backends/narwhals/register.py +++ b/pandera/backends/narwhals/register.py @@ -1,8 +1,9 @@ """Narwhals backend registration utilities. -Registration of Narwhals backends for Polars, Ibis, and PySpark frame types is -handled by ``register_polars_backends()``, ``register_ibis_backends()``, and -``register_pyspark_backends()`` when ``PANDERA_USE_NARWHALS_BACKEND=True`` (or +Registration of Narwhals backends for Polars, Ibis, PySpark, and pandas frame +types is handled by ``register_polars_backends()``, +``register_ibis_backends()``, ``register_pyspark_backends()``, and +``register_pandas_backends()`` when ``PANDERA_USE_NARWHALS_BACKEND=True`` (or ``pandera.config.CONFIG.use_narwhals_backend`` is ``True``). """ @@ -80,6 +81,42 @@ def clear_narwhals_compatible_backend_registry() -> None: except ImportError: pass + try: + import pandas as pd + + from pandera._pandas_deprecated import ( + DataFrameSchema as _PandasDataFrameSchemaDeprecated, + ) + from pandera.api.pandas.components import Column as PandasColumn + from pandera.api.pandas.container import ( + DataFrameSchema as PandasDataFrameSchema, + ) + + # Only the entries the narwhals flag can override are popped. The + # native entries for SeriesSchema/Index/MultiIndex and the other + # pandas-like frame types are never swapped, so they stay put. + pandas_keys: list[tuple[Any, Any]] = [ + (PandasDataFrameSchema, pd.DataFrame), + (_PandasDataFrameSchemaDeprecated, pd.DataFrame), + (PandasColumn, pd.DataFrame), + ] + try: + import narwhals.stable.v1 as nw + + pandas_keys.extend( + [ + (Check, nw.LazyFrame), + (Check, nw.DataFrame), + ] + ) + except ImportError: # pragma: no cover — narwhals is co-installed + pass + + for schema_cls, frame_type in pandas_keys: + _pop_registry_key(schema_cls, frame_type) + except ImportError: # pragma: no cover — pandas is co-installed + pass + try: import pyspark.sql as pyspark_sql @@ -131,8 +168,16 @@ def _narwhals_compatible_registration_state() -> dict[str, bool]: "polars": False, "ibis": False, "pyspark": False, + "pandas": False, } + try: + from pandera.backends.pandas.register import register_pandas_backends + + state["pandas"] = register_pandas_backends.cache_info().currsize > 0 + except ImportError: # pragma: no cover — pandas is co-installed + pass + try: from pandera.backends.polars.register import register_polars_backends @@ -181,6 +226,13 @@ def _get_register_functions() -> dict[str, Any]: except ImportError: pass + try: + from pandera.backends.pandas.register import register_pandas_backends + + register_functions["pandas"] = register_pandas_backends + except ImportError: # pragma: no cover — pandas is co-installed + pass + return register_functions @@ -209,5 +261,18 @@ def reregister_narwhals_compatible_backends( ) for name, register_fn in register_functions.items(): - if previously_registered.get(name): + if not previously_registered.get(name): + continue + if name == "pandas": + # register_pandas_backends is parameterized by check-class fqn; + # re_register_pandas_backends replays each previously-registered + # fqn with the new flag value. + from pandera.backends.pandas.register import ( + re_register_pandas_backends, + ) + + re_register_pandas_backends( + use_narwhals_backend=use_narwhals_backend + ) + else: register_fn(use_narwhals_backend=use_narwhals_backend) diff --git a/pandera/backends/pandas/register.py b/pandera/backends/pandas/register.py index 2bbcc9453..a02b771ec 100644 --- a/pandera/backends/pandas/register.py +++ b/pandera/backends/pandas/register.py @@ -14,19 +14,43 @@ from pandera.backends.pandas.hypotheses import PandasHypothesisBackend from pandera.backends.pandas.parsers import PandasParserBackend +# Fully qualified check-class names that have been registered in this process. +# Used by re_register_pandas_backends() to replay registrations after the +# use_narwhals_backend config flag is toggled at runtime. +_registered_check_cls_fqns: set[str] = set() + @lru_cache def register_pandas_backends( check_cls_fqn: str | None = None, + use_narwhals_backend: bool = False, ): """Register pandas backends. This function is called at schema initialization in the _register_*_backends method. - :param framework_name: name of the framework to register backends for. - Allowable types are "pandas", "dask", "modin", "pyspark", and - "geopandas". + The native pandas backends are always registered — they serve + ``SeriesSchema``, ``Index``, ``MultiIndex``, ``Parser``, and ``Hypothesis`` + validation as well as the pandas-like frame types (dask, modin, geopandas, + pyspark.pandas). When ``use_narwhals_backend`` is ``True`` (from + ``PANDERA_USE_NARWHALS_BACKEND`` or ``pandera.config.CONFIG``), the + ``pandas.DataFrame`` registry entries for ``DataFrameSchema`` and + ``Column`` (plus Narwhals-frame ``Check`` dispatch) are overridden with + the Narwhals backend implementations, mirroring + :func:`pandera.backends.polars.register.register_polars_backends`. + + Decorated with @lru_cache to prevent duplicate registrations across repeated + validate() calls. The backend choice is part of the cache key; programmatic + changes to ``CONFIG.use_narwhals_backend`` after registration trigger + automatic re-registration via ``pandera.config.set_config``. + + :param check_cls_fqn: fully qualified name of the class of the object to + be validated, e.g. "pandas.core.frame.DataFrame". Determines which + framework's types (pandas, dask, modin, pyspark.pandas, geopandas) + are registered. + :param use_narwhals_backend: if True, route ``pd.DataFrame`` validation + through the Narwhals backend. """ from pandera._patch_numpy2 import _patch_numpy2 @@ -53,6 +77,8 @@ def register_pandas_backends( ) backend_types = get_backend_types(check_cls_fqn) + _registered_check_cls_fqns.add(check_cls_fqn) + from pandera.backends.pandas import builtin_checks, builtin_hypotheses for t in backend_types.check_backend_types: @@ -80,3 +106,73 @@ def register_pandas_backends( for t in backend_types.multiindex_datatypes: MultiIndex.register_backend(t, MultiIndexBackend) + + if use_narwhals_backend: + _register_narwhals_pandas_overrides() + + +def _register_narwhals_pandas_overrides(): + """Route ``pd.DataFrame`` validation through the Narwhals backend. + + Overrides only the ``pandas.DataFrame`` registry entries for + ``DataFrameSchema`` and ``Column`` (plus Narwhals-frame ``Check`` + dispatch). Everything else — ``SeriesSchema``, ``Index``, ``MultiIndex``, + ``Parser``, ``Hypothesis``, and the other pandas-like frame types (dask, + modin, geopandas, pyspark.pandas) — stays on the native pandas backends. + """ + try: + import narwhals.stable.v1 as nw + except ImportError as exc: # pragma: no cover + raise ImportError( + "The Narwhals backend is enabled but the 'narwhals' " + "package is not installed. Install it with: " + "pip install 'pandera[narwhals]'" + ) from exc + + import pandas as pd + + import pandera.backends.narwhals.builtin_checks # noqa: F401 + from pandera._pandas_deprecated import ( + DataFrameSchema as _DataFrameSchemaDeprecated, + ) + from pandera.api.checks import Check + from pandera.api.pandas.components import Column + from pandera.api.pandas.container import DataFrameSchema + from pandera.backends.narwhals.checks import NarwhalsCheckBackend + from pandera.backends.narwhals.components import ( + ColumnBackend as NarwhalsColumnBackend, + ) + from pandera.backends.narwhals.container import ( + DataFrameSchemaBackend as NarwhalsDataFrameSchemaBackend, + ) + + DataFrameSchema.register_backend( + pd.DataFrame, NarwhalsDataFrameSchemaBackend, force=True + ) + _DataFrameSchemaDeprecated.register_backend( + pd.DataFrame, NarwhalsDataFrameSchemaBackend, force=True + ) + Column.register_backend(pd.DataFrame, NarwhalsColumnBackend, force=True) + # pandas frames are eager under narwhals: the narwhals backend wraps them + # as nw.LazyFrame for validation, so checks dispatch on both wrapper + # types. The (Check, pd.DataFrame) entry intentionally stays on the + # native PandasCheckBackend — direct Check calls on native pandas frames + # are unchanged. + Check.register_backend(nw.LazyFrame, NarwhalsCheckBackend, force=True) + Check.register_backend(nw.DataFrame, NarwhalsCheckBackend, force=True) + + +def re_register_pandas_backends(*, use_narwhals_backend: bool): + """Re-register pandas backends after toggling ``use_narwhals_backend``. + + Unlike the polars/ibis/pyspark register functions, + :func:`register_pandas_backends` is parameterized by the check class fqn, + so re-registration replays every fqn registered so far in this process + with the new flag value. + """ + fqns = sorted(_registered_check_cls_fqns) + register_pandas_backends.cache_clear() + for fqn in fqns: + register_pandas_backends( + fqn, use_narwhals_backend=use_narwhals_backend + ) diff --git a/pandera/config.py b/pandera/config.py index ff290e5d4..f6cfcfd5c 100644 --- a/pandera/config.py +++ b/pandera/config.py @@ -39,8 +39,8 @@ class PanderaConfig: export PANDERA_USE_NARWHALS_BACKEND=True export SILENCE_WARNING_PYDANTIC_MODEL=true - ``use_narwhals_backend``: when ``True``, Polars, Ibis, and PySpark SQL use - the Narwhals-powered validation backend. Backends register lazily on first + ``use_narwhals_backend``: when ``True``, Polars, Ibis, PySpark SQL, and + pandas use the Narwhals-powered validation backend. Backends register lazily on first schema use; changing this flag via :func:`~pandera.config.set_config` re-registers backends that were already registered in the current process. """ @@ -163,8 +163,9 @@ def set_config( silenced_warnings: List of warning names to silence (default: None) Note: - Changing ``use_narwhals_backend`` re-registers Polars, Ibis, and PySpark - validation backends that were already registered in the current process. + Changing ``use_narwhals_backend`` re-registers Polars, Ibis, PySpark, + and pandas validation backends that were already registered in the + current process. Backends that have not yet been registered pick up the new value on first schema use. See the Narwhals backend documentation for details. """ diff --git a/pyproject.toml b/pyproject.toml index d104380ab..07436c9e5 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -189,6 +189,7 @@ xfail_strict = true markers = [ "polars: tests for the Polars backend", "ibis: tests for the Ibis backend", + "pandas: tests for the pandas backend", ] [tool.ruff] diff --git a/tests/narwhals/backends/conftest.py b/tests/narwhals/backends/conftest.py index dff218462..91c8243f5 100644 --- a/tests/narwhals/backends/conftest.py +++ b/tests/narwhals/backends/conftest.py @@ -47,6 +47,28 @@ def collect(self, result) -> pd.DataFrame: return result.to_pandas() +class PandasBackend(BackendFixture): + name = "pandas" + + @property + def DataFrameSchema(self): + import pandera.pandas as pa_pd + + return pa_pd.DataFrameSchema + + @property + def Column(self): + import pandera.pandas as pa_pd + + return pa_pd.Column + + def make_frame(self, data: dict): + return pd.DataFrame(data) + + def collect(self, result) -> pd.DataFrame: + return result + + class IbisBackend(BackendFixture): name = "ibis" @@ -85,6 +107,7 @@ def collect(self, result) -> pd.DataFrame: BACKENDS = [ pytest.param(PolarsBackend(), id="polars", marks=pytest.mark.polars), pytest.param(IbisBackend(), id="ibis", marks=pytest.mark.ibis), + pytest.param(PandasBackend(), id="pandas", marks=pytest.mark.pandas), ] diff --git a/tests/narwhals/test_pandas_e2e.py b/tests/narwhals/test_pandas_e2e.py new file mode 100644 index 000000000..c576d2a50 --- /dev/null +++ b/tests/narwhals/test_pandas_e2e.py @@ -0,0 +1,851 @@ +"""End-to-end tests for the narwhals backend on pandas DataFrames. + +Cross-backend behavior shared with polars/ibis is covered by the parametrized +suite in tests/narwhals/backends/. This module covers pandas-specific +behavior: native dtype semantics, pandas-style check functions, NaN-based +ignore_na handling, failure-case frame types, and the index-validation gap. + +Requires PANDERA_USE_NARWHALS_BACKEND=True (set by the narwhals nox session); +the fixture below enforces it for local runs. +""" + +import warnings + +import pandas as pd +import pytest + +pytest.importorskip("narwhals") + +import pandera.pandas as pa +from pandera.backends.narwhals.container import ( + DataFrameSchemaBackend as NarwhalsDataFrameSchemaBackend, +) +from pandera.errors import SchemaError, SchemaErrors, SchemaWarning + + +@pytest.fixture(autouse=True) +def use_narwhals_backend(): + """Force narwhals-backed pandas registration for every test here.""" + from pandera.backends.pandas.register import register_pandas_backends + from pandera.config import CONFIG + + original_flag = CONFIG.use_narwhals_backend + CONFIG.use_narwhals_backend = True + register_pandas_backends.cache_clear() + yield + CONFIG.use_narwhals_backend = original_flag + register_pandas_backends.cache_clear() + from pandera.backends.narwhals.register import ( + clear_narwhals_compatible_backend_registry, + ) + + clear_narwhals_compatible_backend_registry() + + +@pytest.fixture +def df() -> pd.DataFrame: + return pd.DataFrame( + {"x": [1, 2, 3], "s": ["a", "b", "c"], "f": [0.5, 1.5, 2.5]} + ) + + +def test_narwhals_backend_is_used(df): + schema = pa.DataFrameSchema({"x": pa.Column(int)}) + assert isinstance(schema.get_backend(df), NarwhalsDataFrameSchemaBackend) + + +def test_validate_returns_pandas_dataframe(df): + schema = pa.DataFrameSchema({"x": pa.Column(int)}) + out = schema.validate(df) + assert isinstance(out, pd.DataFrame) + pd.testing.assert_frame_equal(out, df) + + +def test_validate_preserves_index(): + df = pd.DataFrame({"x": [1, 2]}, index=["a", "b"]) + out = pa.DataFrameSchema({"x": pa.Column(int)}).validate(df) + assert list(out.index) == ["a", "b"] + + +# --------------------------------------------------------------------------- +# dtype semantics — native pandas engine comparisons +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize( + "dtype,data", + [ + (int, [1, 2]), + ("int64", [1, 2]), + (float, [1.0, 2.0]), + (str, ["a", "b"]), # object dtype must satisfy str schema + (object, ["a", 1]), + (bool, [True, False]), + ("datetime64[ns]", pd.to_datetime(["2021-01-01", "2021-01-02"])), + ("category", pd.Categorical(["a", "b"])), + ("Int64", pd.array([1, 2], dtype="Int64")), + ], +) +def test_dtype_passes(dtype, data): + frame = pd.DataFrame({"col": data}) + out = pa.DataFrameSchema({"col": pa.Column(dtype)}).validate(frame) + assert isinstance(out, pd.DataFrame) + + +@pytest.mark.parametrize( + "dtype,data", + [ + (int, ["a", "b"]), + (str, [1, 2]), + ("int32", [1, 2]), # int64 data — width must be respected + (float, [1, 2]), + ("datetime64[ns]", [1, 2]), + ], +) +def test_dtype_mismatch_raises(dtype, data): + frame = pd.DataFrame({"col": data}) + with pytest.raises(SchemaError, match="expected column 'col'"): + pa.DataFrameSchema({"col": pa.Column(dtype)}).validate(frame) + + +def test_dtype_failure_cases_report_native_dtype(): + with pytest.raises(SchemaError) as exc_info: + pa.DataFrameSchema({"x": pa.Column(str)}).validate( + pd.DataFrame({"x": [1, 2]}) + ) + assert exc_info.value.failure_cases == "int64" + + +# --------------------------------------------------------------------------- +# pandas-style check functions keep working under the narwhals backend +# --------------------------------------------------------------------------- + + +def test_series_check_passes(df): + schema = pa.DataFrameSchema( + {"x": pa.Column(int, pa.Check(lambda s: s > 0))} + ) + assert isinstance(schema.validate(df), pd.DataFrame) + + +def test_series_check_fails_with_pandas_failure_cases(df): + schema = pa.DataFrameSchema( + {"x": pa.Column(int, pa.Check(lambda s: s > 1))} + ) + with pytest.raises(SchemaError) as exc_info: + schema.validate(df) + failure_cases = exc_info.value.failure_cases + assert isinstance(failure_cases, pd.DataFrame) + assert failure_cases["x"].tolist() == [1] + + +def test_aggregate_check_returning_numpy_bool(df): + schema = pa.DataFrameSchema( + {"x": pa.Column(int, pa.Check(lambda s: (s > 0).all()))} + ) + assert isinstance(schema.validate(df), pd.DataFrame) + + failing = pa.DataFrameSchema( + {"x": pa.Column(int, pa.Check(lambda s: (s > 1).all()))} + ) + with pytest.raises(SchemaError): + failing.validate(df) + + +def test_check_returning_numpy_array(df): + schema = pa.DataFrameSchema( + {"x": pa.Column(int, pa.Check(lambda s: s.to_numpy() > 0))} + ) + assert isinstance(schema.validate(df), pd.DataFrame) + + +def test_dataframe_level_check(df): + schema = pa.DataFrameSchema( + {"x": pa.Column(int)}, + checks=pa.Check(lambda d: d["x"] > 0), + ) + assert isinstance(schema.validate(df), pd.DataFrame) + + failing = pa.DataFrameSchema( + {"x": pa.Column(int)}, + checks=pa.Check(lambda d: d["x"] > 1), + ) + with pytest.raises(SchemaError): + failing.validate(df) + + +def test_element_wise_check(df): + schema = pa.DataFrameSchema( + {"x": pa.Column(int, pa.Check(lambda v: v > 0, element_wise=True))} + ) + assert isinstance(schema.validate(df), pd.DataFrame) + + +def test_narwhals_expression_check(df): + """native=False checks receive a narwhals column expression.""" + schema = pa.DataFrameSchema( + {"x": pa.Column(int, pa.Check(lambda col: col < 100, native=False))} + ) + assert isinstance(schema.validate(df), pd.DataFrame) + + +def test_dataframe_check_returning_bool_dataframe(df): + """Multi-column boolean DataFrame outputs are AND-reduced per row.""" + schema = pa.DataFrameSchema( + {"x": pa.Column(int)}, + checks=pa.Check(lambda d: d[["x", "f"]] > 0), + ) + assert isinstance(schema.validate(df), pd.DataFrame) + + failing = pa.DataFrameSchema( + {"x": pa.Column(int)}, + checks=pa.Check(lambda d: d[["x", "f"]] > 1), + ) + with pytest.raises(SchemaError): + failing.validate(df) + + +def test_check_returning_dataframe_with_check_output_key(df): + """DataFrame outputs carrying CHECK_OUTPUT_KEY use that column directly.""" + from pandera.constants import CHECK_OUTPUT_KEY + + schema = pa.DataFrameSchema( + { + "x": pa.Column( + int, + pa.Check(lambda s: pd.DataFrame({CHECK_OUTPUT_KEY: s > 0})), + ) + } + ) + assert isinstance(schema.validate(df), pd.DataFrame) + + +def test_builtin_checks(df): + schema = pa.DataFrameSchema( + { + "x": pa.Column(int, [pa.Check.ge(0), pa.Check.isin([1, 2, 3])]), + "s": pa.Column(str, pa.Check.str_matches(r"^[abc]$")), + "f": pa.Column(float, pa.Check.in_range(0, 3)), + } + ) + assert isinstance(schema.validate(df), pd.DataFrame) + + +# --------------------------------------------------------------------------- +# ignore_na semantics with NaN-based missing values +# --------------------------------------------------------------------------- + + +def test_ignore_na_passes_nan_rows(): + """NaN rows pass checks by default (ignore_na=True), like the native backend.""" + frame = pd.DataFrame({"a": [None, 1.0, 2.0]}) + schema = pa.DataFrameSchema( + {"a": pa.Column(float, pa.Check.ge(0), nullable=True)} + ) + assert isinstance(schema.validate(frame), pd.DataFrame) + + +def test_ignore_na_false_fails_nan_rows(): + frame = pd.DataFrame({"a": [None, 1.0, 2.0]}) + schema = pa.DataFrameSchema( + {"a": pa.Column(float, pa.Check.ge(0, ignore_na=False), nullable=True)} + ) + with pytest.raises(SchemaError): + schema.validate(frame) + + +def test_ignore_na_with_nullable_extension_dtype(): + frame = pd.DataFrame({"a": pd.array([None, 1, 2], dtype="Int64")}) + schema = pa.DataFrameSchema( + {"a": pa.Column("Int64", pa.Check.ge(0), nullable=True)} + ) + assert isinstance(schema.validate(frame), pd.DataFrame) + + +def test_drop_invalid_rows_nullable_violation(): + """drop_invalid_rows removes rows violating nullable=False.""" + frame = pd.DataFrame({"a": [1.0, None, 2.0]}) + schema = pa.DataFrameSchema( + {"a": pa.Column(float, nullable=False)}, + drop_invalid_rows=True, + ) + result = schema.validate(frame, lazy=True) + assert result["a"].tolist() == [1.0, 2.0] + + +def test_drop_invalid_rows_keeps_nan_rows(): + """drop_invalid_rows must not drop NaN rows when the check ignores NA.""" + frame = pd.DataFrame({"a": [None, -1.0, 0.0, 1.0]}) + schema = pa.DataFrameSchema( + {"a": pa.Column(float, pa.Check.ge(0), nullable=True)}, + drop_invalid_rows=True, + ) + result = schema.validate(frame, lazy=True) + assert result["a"].isna().sum() == 1 + assert result["a"].dropna().tolist() == [0.0, 1.0] + + +# --------------------------------------------------------------------------- +# failure cases and lazy validation +# --------------------------------------------------------------------------- + + +def test_lazy_failure_cases_are_pandas(df): + """Aggregated failure_cases frame is pandas, even for scalar-only errors.""" + schema = pa.DataFrameSchema( + { + "x": pa.Column(str), # scalar failure case (wrong dtype) + "s": pa.Column(str, pa.Check.isin(["a"])), # data failure cases + "missing": pa.Column(int), # scalar failure case (absent column) + } + ) + with pytest.raises(SchemaErrors) as exc_info: + schema.validate(df, lazy=True) + + err = exc_info.value + assert len(err.schema_errors) == 3 + assert isinstance(err.failure_cases, pd.DataFrame) + assert set(err.failure_cases.columns) == { + "failure_case", + "schema_context", + "column", + "check", + "check_number", + "index", + } + # data-level failure cases contain the failing values, not JSON structs + isin_cases = err.failure_cases.query("column == 's'") + assert sorted(isin_cases["failure_case"]) == ["b", "c"] + + +def test_lazy_failure_cases_scalar_only(df): + """All-scalar failure cases (dtype errors) also produce a pandas frame.""" + schema = pa.DataFrameSchema({"x": pa.Column(str), "s": pa.Column(int)}) + with pytest.raises(SchemaErrors) as exc_info: + schema.validate(df, lazy=True) + failure_cases = exc_info.value.failure_cases + assert isinstance(failure_cases, pd.DataFrame) + # string columns are dtype "object" on pandas < 3 and "str" on pandas >= 3 + assert sorted(failure_cases["failure_case"]) == sorted( + ["int64", str(df["s"].dtype)] + ) + + +def test_schema_error_failure_cases_no_index_leak(df): + """Failure-case frames must not contain pandas index artifacts.""" + schema = pa.DataFrameSchema({"s": pa.Column(str, pa.Check.isin(["a"]))}) + with pytest.raises(SchemaErrors) as exc_info: + schema.validate(df, lazy=True) + failure_cases = exc_info.value.failure_cases + assert not any( + "__index_level_" in str(v) for v in failure_cases["failure_case"] + ) + + +# --------------------------------------------------------------------------- +# pandas-specific schema features +# --------------------------------------------------------------------------- + + +def test_index_validation(df): + """The narwhals backend validates pandas Index components (delegated to + the native Index backend) and preserves the index in the output.""" + schema = pa.DataFrameSchema( + {"x": pa.Column(int)}, index=pa.Index(str, name="s") + ) + out = schema.validate(df.set_index("s")) + assert isinstance(out, pd.DataFrame) + assert out.index.name == "s" + assert list(out.index) == list(df["s"]) + + +def test_index_validation_failure(df): + schema = pa.DataFrameSchema( + {"x": pa.Column(int)}, + index=pa.Index(int, pa.Check.ge(0), name="idx"), + ) + bad = df.copy() + bad.index = pd.Index([-1, 0, 1], name="idx") + with pytest.raises(SchemaError): + schema.validate(bad) + with pytest.raises(pa.errors.SchemaErrors) as exc_info: + schema.validate(bad, lazy=True) + # the failure is reported against the index check with the offending value + fcs = exc_info.value.failure_cases["failure_case"].astype(str).str.cat() + assert "-1" in fcs + assert any( + "greater_than_or_equal_to" in str(c) + for c in exc_info.value.failure_cases["check"] + ) + + +def test_multiindex_validation(): + schema = pa.DataFrameSchema( + {"a": pa.Column(int)}, + index=pa.MultiIndex( + [ + pa.Index(int, name="i0"), + pa.Index(str, pa.Check.isin(["x", "y"]), name="i1"), + ] + ), + ) + mi = pd.MultiIndex.from_tuples([(0, "x"), (1, "y")], names=["i0", "i1"]) + out = schema.validate(pd.DataFrame({"a": [1, 2]}, index=mi)) + assert isinstance(out.index, pd.MultiIndex) + + mi_bad = pd.MultiIndex.from_tuples( + [(0, "x"), (1, "z")], names=["i0", "i1"] + ) + with pytest.raises(pa.errors.SchemaErrors): + schema.validate(pd.DataFrame({"a": [1, 2]}, index=mi_bad), lazy=True) + + +def test_column_coerce(df): + """Column-level coerce=True coerces dtypes for pandas frames (delegated to + the native pandas backend), without emitting a no-op warning.""" + schema = pa.DataFrameSchema({"x": pa.Column(float, coerce=True)}) + with warnings.catch_warnings(): + warnings.simplefilter("error", SchemaWarning) + out = schema.validate(df) + assert out["x"].dtype == "float64" + assert out["x"].tolist() == [1.0, 2.0, 3.0] + + +def test_column_coerce_failure(): + schema = pa.DataFrameSchema({"x": pa.Column(int, coerce=True)}) + bad = pd.DataFrame({"x": ["a", "2"]}) + with pytest.raises(pa.errors.SchemaErrors): + schema.validate(bad, lazy=True) + + +def test_schema_level_coerce(): + schema = pa.DataFrameSchema( + {"a": pa.Column(float), "b": pa.Column(str)}, coerce=True + ) + out = schema.validate(pd.DataFrame({"a": [1, 2], "b": [10, 20]})) + assert out["a"].dtype == "float64" + # str coercion yields object on pandas < 3 and pd.StringDtype on + # pandas >= 3; derive the expected dtype from the engine so the + # assertion is version-agnostic (matches the native pandas backend). + from pandera.engines import pandas_engine + + expected_b_dtype = ( + pandas_engine.Engine.dtype(str).coerce(pd.Series([10, 20])).dtype + ) + assert out["b"].dtype == expected_b_dtype + + +def test_coerce_numpy_dtype_via_narwhals_cast(): + """Plain numpy dtypes are coerced Narwhals-native via ``nw.cast``.""" + schema = pa.DataFrameSchema({"a": pa.Column("int64", coerce=True)}) + out = schema.validate(pd.DataFrame({"a": ["1", "2", "3"]})) + assert out["a"].dtype == "int64" + assert out["a"].tolist() == [1, 2, 3] + + +def test_coerce_nullable_int_falls_back_to_pandas_engine(): + """Nullable ``Int64`` coercion falls back to the pandas dtype engine so + ```` values are preserved (Narwhals cast would drop the nullability). + """ + schema = pa.DataFrameSchema( + {"a": pa.Column("Int64", coerce=True, nullable=True)} + ) + out = schema.validate(pd.DataFrame({"a": [1.0, None, 3.0]})) + assert str(out["a"].dtype) == "Int64" + assert out["a"].isna().tolist() == [False, True, False] + + +def test_coerce_categorical_and_tz_datetime_fidelity(): + """Extension dtypes (category, tz-aware datetime) coerce with native + pandas fidelity via the pandas dtype engine fallback.""" + cat = pa.DataFrameSchema({"a": pa.Column("category", coerce=True)}) + assert str( + cat.validate(pd.DataFrame({"a": ["x", "y", "x"]}))["a"].dtype + ) == ("category") + + tz = pa.DataFrameSchema( + {"a": pa.Column("datetime64[ns, UTC]", coerce=True)} + ) + out = tz.validate( + pd.DataFrame({"a": pd.to_datetime(["2021-01-01", "2021-01-02"])}) + ) + assert str(out["a"].dtype) == "datetime64[ns, UTC]" + + +def test_coerce_failure_reports_offending_values(): + """A failed coercion reports the offending value (pandas-engine fallback).""" + schema = pa.DataFrameSchema({"a": pa.Column(int, coerce=True)}) + with pytest.raises(pa.errors.SchemaErrors) as exc_info: + schema.validate(pd.DataFrame({"a": ["x", "2"]}), lazy=True) + coerce_fc = exc_info.value.failure_cases.query( + "check == \"coerce_dtype('int64')\"" + ) + assert "x" in coerce_fc["failure_case"].astype(str).str.cat() + + +def test_index_coerce(): + schema = pa.DataFrameSchema( + {"a": pa.Column(int)}, + index=pa.Index(int, coerce=True, name="idx"), + ) + out = schema.validate( + pd.DataFrame({"a": [1, 2]}, index=pd.Index(["0", "1"], name="idx")) + ) + assert out.index.dtype == "int64" + + +def test_index_coerce_failure(): + schema = pa.DataFrameSchema( + {"a": pa.Column(int)}, + index=pa.Index(int, coerce=True, name="idx"), + ) + bad = pd.DataFrame({"a": [1, 2]}, index=pd.Index(["x", "y"], name="idx")) + with pytest.raises((SchemaError, SchemaErrors)): + schema.validate(bad, lazy=True) + + +def test_multiindex_coerce(): + schema = pa.DataFrameSchema( + {"a": pa.Column(int)}, + index=pa.MultiIndex( + [ + pa.Index(int, coerce=True, name="i0"), + pa.Index(str, name="i1"), + ] + ), + ) + mi = pd.MultiIndex.from_tuples( + [("0", "x"), ("1", "y")], names=["i0", "i1"] + ) + out = schema.validate(pd.DataFrame({"a": [1, 2]}, index=mi)) + assert out.index.get_level_values("i0").dtype == "int64" + + +def test_custom_parsers(): + """Custom parsers= run on the native frame inside the narwhals backend.""" + schema = pa.DataFrameSchema( + {"a": pa.Column(int, pa.Check.ge(0))}, + parsers=pa.Parser(lambda d: d.assign(a=d["a"].abs())), + ) + # The check ge(0) would fail on the raw negative data; it passes only + # because the parser ran first. + out = schema.validate(pd.DataFrame({"a": [-1, -2, 3]})) + assert out["a"].tolist() == [1, 2, 3] + + +def test_custom_parsers_chained(): + schema = pa.DataFrameSchema( + {"a": pa.Column(int)}, + parsers=[ + pa.Parser(lambda d: d.assign(a=d["a"] + 1)), + pa.Parser(lambda d: d.assign(a=d["a"] * 2)), + ], + ) + out = schema.validate(pd.DataFrame({"a": [1, 2]})) + assert out["a"].tolist() == [4, 6] # (x+1)*2 + + +def test_custom_parsers_element_wise(): + schema = pa.DataFrameSchema( + {"a": pa.Column(int), "b": pa.Column(int)}, + parsers=pa.Parser(lambda row: row * 2, element_wise=True), + ) + out = schema.validate(pd.DataFrame({"a": [1, 2], "b": [3, 4]})) + assert out["a"].tolist() == [2, 4] + assert out["b"].tolist() == [6, 8] + + +def test_add_missing_columns(): + schema = pa.DataFrameSchema( + {"a": pa.Column(int), "b": pa.Column(float, default=1.5)}, + add_missing_columns=True, + ) + out = schema.validate(pd.DataFrame({"a": [1, 2, 3]})) + assert out["b"].tolist() == [1.5, 1.5, 1.5] + assert out["b"].dtype == "float64" + assert list(out.columns) == ["a", "b"] + + +def test_add_missing_columns_ordering_and_multiple(): + """Missing columns are inserted at their schema position, not appended.""" + schema = pa.DataFrameSchema( + { + "a": pa.Column(int, default=0), + "b": pa.Column(int), + "c": pa.Column(int, default=0), + }, + add_missing_columns=True, + ) + out = schema.validate(pd.DataFrame({"b": [1]})) + assert list(out.columns) == ["a", "b", "c"] + + +def test_add_missing_columns_extension_dtype_fidelity(): + """Added extension-dtype columns get the correct native dtype (not numpy): + nullable Int64 with a default, and nullable-no-default (all ).""" + schema = pa.DataFrameSchema( + { + "a": pa.Column(int), + "b": pa.Column("Int64", default=5, nullable=True), + "c": pa.Column("Int64", nullable=True), + }, + add_missing_columns=True, + ) + out = schema.validate(pd.DataFrame({"a": [1, 2]})) + assert str(out["b"].dtype) == "Int64" + assert out["b"].tolist() == [5, 5] + assert str(out["c"].dtype) == "Int64" + assert out["c"].isna().all() + + +def test_add_missing_columns_requires_default(): + schema = pa.DataFrameSchema( + {"a": pa.Column(int), "b": pa.Column(int)}, + add_missing_columns=True, + ) + with pytest.raises(pa.errors.SchemaErrors): + schema.validate(pd.DataFrame({"a": [1]}), lazy=True) + + +def test_set_defaults(): + schema = pa.DataFrameSchema( + {"a": pa.Column(float, default=0.0, nullable=False)} + ) + out = schema.validate(pd.DataFrame({"a": [1.0, None, 3.0]})) + # only the null value is replaced by the default; others are untouched. + assert out["a"].tolist() == [1.0, 0.0, 3.0] + + +def test_set_defaults_no_default_is_noop(): + schema = pa.DataFrameSchema({"a": pa.Column(float, nullable=True)}) + out = schema.validate(pd.DataFrame({"a": [1.0, None, 3.0]})) + assert out["a"].isna().tolist() == [False, True, False] + + +def test_unique_column_names(): + schema = pa.DataFrameSchema( + {"a": pa.Column(int)}, unique_column_names=True + ) + dup = pd.DataFrame([[1, 2], [3, 4]], columns=["a", "a"]) + with pytest.raises(SchemaError, match="multiple columns with label"): + schema.validate(dup) + with pytest.raises(SchemaErrors) as exc_info: + schema.validate(dup, lazy=True) + assert ( + exc_info.value.schema_errors[0].reason_code + == pa.errors.SchemaErrorReason.DUPLICATE_COLUMN_LABELS + ) + + +def test_groupby_check(): + """Column check groups (groupby=) run Narwhals-native via apply_groupby.""" + schema = pa.DataFrameSchema( + { + "height": pa.Column( + float, + pa.Check( + lambda g: g["M"].mean() > g["F"].mean(), groupby="sex" + ), + ), + "sex": pa.Column(str), + } + ) + ok = pd.DataFrame( + {"height": [6.0, 5.9, 5.4, 5.5], "sex": ["M", "M", "F", "F"]} + ) + schema.validate(ok) + + bad = pd.DataFrame( + {"height": [5.0, 5.1, 6.4, 6.5], "sex": ["M", "M", "F", "F"]} + ) + with pytest.raises(SchemaErrors): + schema.validate(bad, lazy=True) + + +def test_groupby_check_returns_dict(): + """A groupby check returning a per-group dict of bools is reduced to + pass only when every group passes.""" + schema = pa.DataFrameSchema( + { + "val": pa.Column( + int, + pa.Check( + lambda g: {k: (v > 0).all() for k, v in g.items()}, + groupby="grp", + ), + ), + "grp": pa.Column(str), + } + ) + ok = pd.DataFrame({"val": [1, 2, 3, 4], "grp": ["a", "a", "b", "b"]}) + schema.validate(ok) + + bad = pd.DataFrame({"val": [1, 2, -3, 4], "grp": ["a", "a", "b", "b"]}) + with pytest.raises(SchemaErrors): + schema.validate(bad, lazy=True) + + +def test_groupby_check_with_groups_filter(): + """The groups= kwarg restricts which groups the check runs on.""" + schema = pa.DataFrameSchema( + { + "val": pa.Column( + int, + pa.Check( + lambda g: g["a"].mean() > 0, groupby="grp", groups=["a"] + ), + ), + "grp": pa.Column(str), + } + ) + # group "b" is negative but excluded by groups=["a"], so validation passes. + df = pd.DataFrame({"val": [1, 2, -5, -6], "grp": ["a", "a", "b", "b"]}) + schema.validate(df) + + +def test_hypothesis_check(): + """Hypothesis checks run through the native pandas hypothesis backend.""" + pytest.importorskip("scipy") + schema = pa.DataFrameSchema( + { + "height": pa.Column( + float, + pa.Hypothesis.two_sample_ttest( + sample1="M", + sample2="F", + groupby="sex", + relationship="greater_than", + alpha=0.05, + ), + ), + "sex": pa.Column(str), + } + ) + ok = pd.DataFrame( + { + "height": [7.0, 6.9, 6.8, 5.4, 5.5, 5.3], + "sex": ["M", "M", "M", "F", "F", "F"], + } + ) + schema.validate(ok) + + bad = pd.DataFrame( + { + "height": [5.0, 5.1, 5.2, 6.4, 6.5, 6.6], + "sex": ["M", "M", "M", "F", "F", "F"], + } + ) + with pytest.raises(SchemaErrors): + schema.validate(bad, lazy=True) + + +def test_series_schema_stays_native(): + """SeriesSchema validation is unaffected by the narwhals backend flag.""" + schema = pa.SeriesSchema(int, pa.Check.ge(0)) + out = schema.validate(pd.Series([1, 2, 3])) + assert isinstance(out, pd.Series) + + +def test_dataframe_model(df): + class Model(pa.DataFrameModel): + x: int + s: str + f: float + + out = Model.validate(df) + assert isinstance(out, pd.DataFrame) + + class FailingModel(pa.DataFrameModel): + x: str + + with pytest.raises(SchemaError): + FailingModel.validate(df) + + +def test_regex_columns(): + frame = pd.DataFrame({"col_a": [1], "col_b": [2], "other": ["x"]}) + schema = pa.DataFrameSchema( + {"^col_.*$": pa.Column(int, pa.Check.ge(0), regex=True)} + ) + out = schema.validate(frame) + assert isinstance(out, pd.DataFrame) + + with pytest.raises(SchemaError): + schema.validate(pd.DataFrame({"col_a": [-1], "other": ["x"]})) + + +def test_ordered_schema(): + frame = pd.DataFrame({"b": [1], "a": [2]}) + schema = pa.DataFrameSchema( + {"a": pa.Column(int), "b": pa.Column(int)}, ordered=True + ) + with pytest.raises(SchemaError, match="out-of-order"): + schema.validate(frame) + + +def test_scalar_failure_cases_without_polars(df, monkeypatch): + """Scalar failure cases build pandas frames on polars-free installs.""" + import pandera.backends.narwhals.base as narwhals_base + + monkeypatch.setattr(narwhals_base, "pl", None) + schema = pa.DataFrameSchema({"x": pa.Column(str)}) + with pytest.raises(SchemaErrors) as exc_info: + schema.validate(df, lazy=True) + failure_cases = exc_info.value.failure_cases + assert isinstance(failure_cases, pd.DataFrame) + assert failure_cases["failure_case"].tolist() == ["int64"] + + +def test_failure_cases_metadata_with_non_frame_error_data(df): + """Non-frame err.data falls back to schema-module implementation detection.""" + from pandera.errors import SchemaErrorReason + + schema = pa.DataFrameSchema({"x": pa.Column(int)}) + backend = schema.get_backend(df) + err = SchemaError( + schema=pa.Column(int, name="x"), + data="not-a-frame", + message="boom", + failure_cases="int64", + reason_code=SchemaErrorReason.WRONG_DATATYPE, + ) + metadata = backend.failure_cases_metadata("schema", [err]) + assert isinstance(metadata.failure_cases, pd.DataFrame) + + +def test_check_dtype_iterable_result(df): + """Iterable dtype-check results are AND-reduced (native pandas semantics).""" + import narwhals.stable.v1 as nw + + from pandera.backends.narwhals.components import ColumnBackend + from pandera.engines import numpy_engine + + class IterableResultInt64(numpy_engine.Int64): + def check(self, pandera_dtype, data_container=None): + return iter([True, True]) + + column = pa.Column(int, name="x") + column._dtype = IterableResultInt64() + lf = nw.from_native(df).lazy() + results = ColumnBackend().check_dtype(lf, column) + assert all(result.passed for result in results) + + +def test_is_pandas_like_helper(df): + """_is_pandas_like distinguishes narwhals pandas frames from other values.""" + import narwhals.stable.v1 as nw + + from pandera.api.narwhals.utils import _is_pandas_like + + assert _is_pandas_like(nw.from_native(df)) + assert _is_pandas_like(nw.from_native(df).lazy()) + assert not _is_pandas_like("not a frame") + assert not _is_pandas_like(df) + + +def test_pandas_column_selector_property(): + """The pandas Column exposes the selector property used by the backend.""" + assert pa.Column(int, name="a").selector == "a" + assert pa.Column(int, name="a", regex=True).selector == "^a$" + assert pa.Column(int, name="^a.*$", regex=True).selector == "^a.*$" + assert pa.Column(int, name=("a", "b")).selector == ("a", "b") diff --git a/tests/narwhals/test_pandas_narwhals_register.py b/tests/narwhals/test_pandas_narwhals_register.py new file mode 100644 index 000000000..b32cb82f8 --- /dev/null +++ b/tests/narwhals/test_pandas_narwhals_register.py @@ -0,0 +1,242 @@ +"""Tests for register_pandas_backends() narwhals activation. + +Requires both pandas and narwhals. Mirrors +tests/pyspark/test_pyspark_narwhals_register.py and +tests/ibis/test_ibis_narwhals_register.py. +""" + +import pandas as pd +import pytest + +pytest.importorskip("narwhals") + +PANDAS_DATAFRAME_FQN = "pandas.core.frame.DataFrame" + + +@pytest.fixture +def restore_pandas_registry(request): + """Snapshot and restore the pd.DataFrame registry entries and cache.""" + from pandera._pandas_deprecated import ( + DataFrameSchema as DataFrameSchemaDeprecated, + ) + from pandera.api.checks import Check + from pandera.api.pandas.components import Column + from pandera.api.pandas.container import DataFrameSchema + from pandera.backends.pandas.register import register_pandas_backends + + registry_keys = [ + (DataFrameSchema, (DataFrameSchema, pd.DataFrame)), + ( + DataFrameSchemaDeprecated, + (DataFrameSchemaDeprecated, pd.DataFrame), + ), + (Column, (Column, pd.DataFrame)), + (Check, (Check, pd.DataFrame)), + ] + saved = [ + (cls, key, cls.BACKEND_REGISTRY.pop(key, None)) + for cls, key in registry_keys + ] + + def restore(): + register_pandas_backends.cache_clear() + for cls, key, backend in saved: + if backend is None: + cls.BACKEND_REGISTRY.pop(key, None) + else: + cls.BACKEND_REGISTRY[key] = backend + + request.addfinalizer(restore) + register_pandas_backends.cache_clear() + + +def test_pandas_narwhals_activated_when_opted_in(restore_pandas_registry): + """register_pandas_backends() registers narwhals backends when opted in.""" + import narwhals.stable.v1 as nw + + from pandera._pandas_deprecated import ( + DataFrameSchema as DataFrameSchemaDeprecated, + ) + from pandera.api.checks import Check + from pandera.api.pandas.components import Column + from pandera.api.pandas.container import DataFrameSchema + from pandera.backends.narwhals.checks import NarwhalsCheckBackend + from pandera.backends.narwhals.components import ( + ColumnBackend as NarwhalsColumnBackend, + ) + from pandera.backends.narwhals.container import ( + DataFrameSchemaBackend as NarwhalsDataFrameSchemaBackend, + ) + from pandera.backends.pandas.checks import PandasCheckBackend + from pandera.backends.pandas.register import register_pandas_backends + + register_pandas_backends(PANDAS_DATAFRAME_FQN, use_narwhals_backend=True) + + backend = DataFrameSchema.get_backend(check_type=pd.DataFrame) + assert isinstance(backend, NarwhalsDataFrameSchemaBackend) + + deprecated_backend = DataFrameSchemaDeprecated.get_backend( + check_type=pd.DataFrame + ) + assert isinstance(deprecated_backend, NarwhalsDataFrameSchemaBackend) + + column_backend = Column.get_backend(check_type=pd.DataFrame) + assert isinstance(column_backend, NarwhalsColumnBackend) + + # Narwhals-wrapped frames dispatch to the narwhals check backend; the + # native pd.DataFrame entry stays on the native pandas check backend so + # direct Check calls on native frames are unchanged. + assert ( + Check.BACKEND_REGISTRY[(Check, nw.LazyFrame)] is NarwhalsCheckBackend + ) + assert ( + Check.BACKEND_REGISTRY[(Check, nw.DataFrame)] is NarwhalsCheckBackend + ) + assert Check.BACKEND_REGISTRY[(Check, pd.DataFrame)] is PandasCheckBackend + + +def test_pandas_native_unchanged_when_flag_off( + monkeypatch, restore_pandas_registry +): + """register_pandas_backends() registers native backends when opted out.""" + from pandera.api.pandas.container import DataFrameSchema + from pandera.backends.pandas.container import ( + DataFrameSchemaBackend as NativeBackend, + ) + from pandera.backends.pandas.register import register_pandas_backends + from pandera.config import CONFIG + + # get_backend() re-invokes registration reading CONFIG, so the flag must + # be off there as well. + monkeypatch.setattr(CONFIG, "use_narwhals_backend", False) + register_pandas_backends(PANDAS_DATAFRAME_FQN, use_narwhals_backend=False) + backend = DataFrameSchema.get_backend(check_type=pd.DataFrame) + assert isinstance(backend, NativeBackend) + + +def test_pandas_series_and_index_backends_stay_native( + restore_pandas_registry, +): + """SeriesSchema/Index/MultiIndex stay on the native pandas backends.""" + from pandera.api.pandas.array import SeriesSchema + from pandera.api.pandas.components import Index, MultiIndex + from pandera.backends.pandas.array import SeriesSchemaBackend + from pandera.backends.pandas.components import ( + IndexBackend, + MultiIndexBackend, + ) + from pandera.backends.pandas.register import register_pandas_backends + + register_pandas_backends(PANDAS_DATAFRAME_FQN, use_narwhals_backend=True) + + assert isinstance( + SeriesSchema.get_backend(check_type=pd.Series), SeriesSchemaBackend + ) + assert isinstance(Index.get_backend(check_type=pd.DataFrame), IndexBackend) + assert isinstance( + MultiIndex.get_backend(check_type=pd.DataFrame), MultiIndexBackend + ) + + +def test_pandas_register_is_idempotent(): + """Calling register_pandas_backends() twice does not raise or corrupt state.""" + from pandera.backends.pandas.register import register_pandas_backends + from pandera.config import CONFIG + + register_pandas_backends( + PANDAS_DATAFRAME_FQN, + use_narwhals_backend=CONFIG.use_narwhals_backend, + ) + register_pandas_backends( + PANDAS_DATAFRAME_FQN, + use_narwhals_backend=CONFIG.use_narwhals_backend, + ) + + +def test_re_register_pandas_backends_replays_fqns( + monkeypatch, restore_pandas_registry +): + """re_register_pandas_backends() replays registrations with the new flag.""" + from pandera.api.pandas.container import DataFrameSchema + from pandera.backends.narwhals.container import ( + DataFrameSchemaBackend as NarwhalsDataFrameSchemaBackend, + ) + from pandera.backends.pandas.container import ( + DataFrameSchemaBackend as NativeBackend, + ) + from pandera.backends.pandas.register import ( + re_register_pandas_backends, + register_pandas_backends, + ) + from pandera.config import CONFIG + + # get_backend() re-invokes registration reading CONFIG, so keep CONFIG in + # sync with each re-registration below. + monkeypatch.setattr(CONFIG, "use_narwhals_backend", False) + register_pandas_backends(PANDAS_DATAFRAME_FQN, use_narwhals_backend=False) + assert isinstance( + DataFrameSchema.get_backend(check_type=pd.DataFrame), NativeBackend + ) + + monkeypatch.setattr(CONFIG, "use_narwhals_backend", True) + re_register_pandas_backends(use_narwhals_backend=True) + assert isinstance( + DataFrameSchema.get_backend(check_type=pd.DataFrame), + NarwhalsDataFrameSchemaBackend, + ) + + # Toggling back re-registers the native backend. Registry entries from + # the narwhals registration must be cleared first — mirroring what + # set_config() does via reregister_narwhals_compatible_backends(). + from pandera.backends.narwhals.register import ( + clear_narwhals_compatible_backend_registry, + ) + + monkeypatch.setattr(CONFIG, "use_narwhals_backend", False) + clear_narwhals_compatible_backend_registry() + re_register_pandas_backends(use_narwhals_backend=False) + assert isinstance( + DataFrameSchema.get_backend(check_type=pd.DataFrame), NativeBackend + ) + + +def test_set_config_reregisters_pandas_backends(restore_pandas_registry): + """set_config(use_narwhals_backend=...) swaps registered pandas backends.""" + import pandera + from pandera.api.pandas.container import DataFrameSchema + from pandera.backends.narwhals.container import ( + DataFrameSchemaBackend as NarwhalsDataFrameSchemaBackend, + ) + from pandera.backends.pandas.container import ( + DataFrameSchemaBackend as NativeBackend, + ) + from pandera.backends.pandas.register import register_pandas_backends + from pandera.config import CONFIG + + original_flag = CONFIG.use_narwhals_backend + + try: + pandera.set_config(use_narwhals_backend=False) + register_pandas_backends( + PANDAS_DATAFRAME_FQN, use_narwhals_backend=False + ) + assert isinstance( + DataFrameSchema.get_backend(check_type=pd.DataFrame), + NativeBackend, + ) + + with pytest.warns(UserWarning, match="Re-registered"): + pandera.set_config(use_narwhals_backend=True) + assert isinstance( + DataFrameSchema.get_backend(check_type=pd.DataFrame), + NarwhalsDataFrameSchemaBackend, + ) + + with pytest.warns(UserWarning, match="Re-registered"): + pandera.set_config(use_narwhals_backend=False) + assert isinstance( + DataFrameSchema.get_backend(check_type=pd.DataFrame), + NativeBackend, + ) + finally: + pandera.set_config(use_narwhals_backend=original_flag) diff --git a/tests/polars/test_polars_container.py b/tests/polars/test_polars_container.py index ddf3ec8b0..751e0fdd9 100644 --- a/tests/polars/test_polars_container.py +++ b/tests/polars/test_polars_container.py @@ -448,11 +448,6 @@ def test_drop_invalid_rows_nullable( assert validated_data.collect().equals(expected_valid_data.collect()) -@pytest.mark.xfail( - condition=CONFIG.use_narwhals_backend, - reason="set_default not implemented in Narwhals backend", - strict=True, -) def test_set_defaults(ldf_basic, ldf_schema_basic): ldf_schema_basic.columns["int_col"].default = 1 ldf_schema_basic.columns["string_col"].default = "a" diff --git a/tests/pyspark/test_schemas_on_pyspark_pandas.py b/tests/pyspark/test_schemas_on_pyspark_pandas.py index 49eb30e80..04fbeeb32 100644 --- a/tests/pyspark/test_schemas_on_pyspark_pandas.py +++ b/tests/pyspark/test_schemas_on_pyspark_pandas.py @@ -487,6 +487,19 @@ def test_dtype_coercion_nullable_reports_invalid_values(): } +def _failure_case_values(failure_cases): + """Extract the failing values from either failure-cases shape. + + The native pandas backend reshapes failure cases into a frame with a + ``failure_case`` column; the narwhals backend + (``PANDERA_USE_NARWHALS_BACKEND=True``) returns the failing rows with + the original column names. + """ + if "failure_case" in failure_cases.columns: + return failure_cases["failure_case"] + return failure_cases["field"] + + @pytest.mark.parametrize("dtype", [float, int, str, bool]) @hypothesis.given(st.data()) def test_failure_cases(dtype, data): @@ -504,7 +517,7 @@ def test_failure_cases(dtype, data): try: schema(sample) except pa.errors.SchemaError as exc: - assert (exc.failure_cases.failure_case != value).all() + assert (_failure_case_values(exc.failure_cases) != value).all() # make sure reporting a limited number of failure cases works correctly updated_schema = schema.update_column( @@ -513,7 +526,7 @@ def test_failure_cases(dtype, data): try: updated_schema(sample) except pa.errors.SchemaError as exc: - assert (exc.failure_cases.failure_case != value).all() + assert (_failure_case_values(exc.failure_cases) != value).all() assert exc.failure_cases.shape[0] == 2