feat(narwhals): support the pandas schema API in the narwhals backend

docs/source/configuration.md

@@ -38,7 +38,7 @@ This can be achieved by setting the environment variable
 
 Pandera ships an optional
 [Narwhals](https://narwhals-dev.github.io/narwhals/)-powered backend that
-unifies the Polars, Ibis, and PySpark SQL validation paths. It is **opt-in**; by
+unifies the Polars, Ibis, PySpark SQL, and pandas validation paths. It is **opt-in**; by
 default the native backends are used.
 
 Install the `narwhals` extra and enable the backend with either:

docs/source/narwhals_backend.md

@@ -4,13 +4,14 @@
 
 As of *0.32.0*, Pandera ships an optional
 [Narwhals](https://narwhals-dev.github.io/narwhals/)-based validation
-backend that powers the {ref}`Polars <polars>`, {ref}`Ibis <ibis>`, and
-{ref}`PySpark SQL <native-pyspark>` integrations behind a single unified code
-path. The Narwhals backend is **opt-in**: by default Pandera continues to use
-the native Polars, Ibis, and PySpark backends. The public API
+backend that powers the {ref}`Polars <polars>`, {ref}`Ibis <ibis>`,
+{ref}`PySpark SQL <native-pyspark>`, and {ref}`pandas <dataframeschemas>`
+integrations behind a single unified code path. The Narwhals backend is
+**opt-in**: by default Pandera continues to use the native Polars, Ibis,
+PySpark, and pandas backends. The public API
 (`import pandera.polars as pa`, `import pandera.ibis as pa`,
-`import pandera.pyspark as pa`) is unchanged regardless of which backend is
-active.
+`import pandera.pyspark as pa`, `import pandera.pandas as pa`) is unchanged
+regardless of which backend is active.
 
 ## Enabling the Narwhals backend
 
@@ -21,6 +22,7 @@ backend(s) you use:
 pip install 'pandera[narwhals,polars]'   # Polars
 pip install 'pandera[narwhals,ibis]'     # Ibis
 pip install 'pandera[narwhals,pyspark]'  # PySpark SQL
+pip install 'pandera[narwhals,pandas]'   # pandas
 ```
 
 Then enable it using **either** of the following options.
@@ -39,7 +41,7 @@ This value is read when `pandera.config` is first imported.
 ### Programmatic configuration
 
 Call {func}`~pandera.set_config` at any point — before or after importing
-`pandera.polars`, `pandera.ibis`, or `pandera.pyspark`:
+`pandera.polars`, `pandera.ibis`, `pandera.pyspark`, or `pandera.pandas`:
 
 ```python
 import pandera
@@ -66,7 +68,18 @@ register_polars_backends(use_narwhals_backend=True)
 ```
 
 The same pattern applies to `register_ibis_backends` and
-`register_pyspark_backends`.
+`register_pyspark_backends`. The pandas register function is additionally
+parameterized by the fully qualified name of the frame class being
+validated:
+
+```python
+from pandera.backends.pandas.register import register_pandas_backends
+
+register_pandas_backends.cache_clear()
+register_pandas_backends(
+    "pandas.core.frame.DataFrame", use_narwhals_backend=True
+)
+```
 
 If `PANDERA_USE_NARWHALS_BACKEND=True` but `narwhals` is not installed,
 schema construction raises an `ImportError` directing you to install
@@ -85,18 +98,20 @@ Two behaviours govern how that mapping is established and updated at runtime.
 
 ### Lazy registration
 
-Validation backends for Polars, Ibis, and PySpark SQL are registered
+Validation backends for Polars, Ibis, PySpark SQL, and pandas are registered
 **lazily** — not when you import a pandera backend module, but the first time
 a schema needs a backend. Concretely, registration runs when you:
 
 - construct a {py:class}`~pandera.api.polars.container.DataFrameSchema`,
   {py:class}`~pandera.api.ibis.container.DataFrameSchema`, or
   {py:class}`~pandera.api.pyspark.container.DataFrameSchema`, or
 - call `validate()` on a column or schema component that triggers backend
-  lookup.
+  lookup (for pandas schemas, registration always happens at validation
+  time, when the type of the validated object is known).
 
-Until one of those happens, importing ``pandera.polars``, ``pandera.ibis``, or
-``pandera.pyspark`` has no effect on which validation backend is active:
+Until one of those happens, importing ``pandera.polars``, ``pandera.ibis``,
+``pandera.pyspark``, or ``pandera.pandas`` has no effect on which validation
+backend is active:
 
 ```python
 import pandera.polars as pa
@@ -128,8 +143,8 @@ backends have already been registered, pandera **re-registers** them
 automatically:
 
 1. The global ``CONFIG.use_narwhals_backend`` value is updated.
-2. Pandera detects which of the Polars / Ibis / PySpark register functions had
-   already run.
+2. Pandera detects which of the Polars / Ibis / PySpark / pandas register
+   functions had already run.
 3. Registration caches are cleared and existing registry entries for those
    backends are removed.
 4. Only the backends that were previously registered are registered again, now
@@ -154,8 +169,8 @@ registered backend from the global registry on each ``validate()`` call.
 
 Re-registration applies only to backends that had already been registered in
 the current process. If you call ``set_config(use_narwhals_backend=True)``
-before constructing any Polars/Ibis/PySpark schema, no re-registration occurs
-— the first lazy registration picks up the updated config silently.
+before constructing any Polars/Ibis/PySpark/pandas schema, no re-registration
+occurs — the first lazy registration picks up the updated config silently.
 
 :::{note}
 Runtime re-registration is triggered by {func}`~pandera.set_config`, which
@@ -177,12 +192,13 @@ it executed natively by each supported engine.
 
 ## What it changes for you
 
-* **Unified checks across Polars, Ibis, and PySpark SQL.** Built-in checks
-  (`isin`, `in_range`, `str_matches`, etc.) are implemented as Narwhals
-  expressions and run unchanged on Polars LazyFrames, Ibis tables, and
-  PySpark SQL DataFrames when the Narwhals backend is enabled. PySpark SQL
-  is a SQL-lazy backend: element-wise checks are not supported, and row
-  sampling (`sample=` / `tail=` parameters) is not supported.
+* **Unified checks across Polars, Ibis, PySpark SQL, and pandas.** Built-in
+  checks (`isin`, `in_range`, `str_matches`, etc.) are implemented as
+  Narwhals expressions and run unchanged on Polars LazyFrames, Ibis tables,
+  PySpark SQL DataFrames, and pandas DataFrames when the Narwhals backend is
+  enabled. PySpark SQL is a SQL-lazy backend: element-wise checks are not
+  supported, and row sampling (`sample=` / `tail=` parameters) is not
+  supported.
 * **Lazy validation stays lazy.** For Polars LazyFrames, Ibis tables, and
   PySpark SQL DataFrames, Pandera threads validation through the native lazy
   API: no full-frame `.collect()` / `.execute()` is triggered during
@@ -235,12 +251,82 @@ backend:
   `dataframe.pandera.errors` accessor, use the native PySpark backend
   (see {ref}`Opting out <narwhals-opting-out>`).
 
+(narwhals-pandas-differences)=
+
+## pandas: differences from the native backend
+
+When the Narwhals backend is enabled, `pandas.DataFrame` validation through
+{py:class}`~pandera.api.pandas.container.DataFrameSchema` (and
+`DataFrameModel`) is routed through the shared Narwhals code path. Because
+pandas frames are **eager**, the Narwhals pandas path reaches feature parity
+with the native pandas backend. Almost everything is handled **Narwhals-native**
+(parsers, `add_missing_columns`, `set_default`, column/schema coercion,
+`groupby` checks, and all cross-backend checks). The **only** steps delegated to
+the native pandas backend are **Index/MultiIndex validation** (and its
+coercion) — because Narwhals has no index concept — and **`Hypothesis`**
+checks, which rely on scipy statistical tests. A few behavioural notes:
+
+- **Only `pd.DataFrame` validation is rerouted.** `SeriesSchema` validation —
+  and the other pandas-like frame types (dask, modin, geopandas,
+  `pyspark.pandas`) — always use the native pandas backends, regardless of
+  the flag. `DataFrameSchema` `Index`/`MultiIndex` **components** are
+  validated (see below).
+- **Index and MultiIndex components are validated (delegated).** Narwhals
+  preserves the pandas index through its operations, so a schema with an
+  `index=` component is validated by delegating to the native Index/MultiIndex
+  backends; index-level coercion is applied and the index is preserved in the
+  output. (For non-pandas frames — polars/ibis/pyspark — Narwhals has no index
+  concept, so an `index=` component there still emits a ``SchemaWarning`` and
+  is skipped.)
+- **Column dtype checks compare native pandas dtypes.** Dtype *checks* still
+  use the pandas dtype engine, so `Column(str)` accepts `object` columns, and
+  nullable extension dtypes (`Int64`), categoricals, and tz-aware datetimes are
+  recognised as under the native backend.
+- **pandas-style check functions keep working.** Column-level checks receive
+  the `pd.Series` for the column (e.g. `pa.Check(lambda s: s > 0)`), and
+  dataframe-level checks receive the `pd.DataFrame` — the same convention as
+  the native backend. Checks may return boolean Series/DataFrames, numpy
+  boolean arrays, or scalar booleans. Setting `native=False` passes a
+  Narwhals column expression instead, making the check portable across all
+  Narwhals-backed integrations.
+- **`coerce=True` uses a hybrid strategy.** Column- and schema-level coercion
+  cast plain numpy dtypes (`int`, `float`, `str`, `bool`) Narwhals-native via
+  ``nw.cast``, and **fall back to the pandas dtype engine for pandas extension
+  dtypes** — nullable ``Int64``, ``Categorical``, tz-aware datetimes, and
+  ``string`` — so their native semantics are preserved (e.g. a nullable
+  ``Int64`` column keeps its ``<NA>`` values). A Narwhals cast that raises also
+  falls back to the pandas engine, which reports the offending values in the
+  ``DATATYPE_COERCION`` error. (Index-level coercion is delegated to the native
+  Index backend.)
+- **`parsers`, `add_missing_columns`, and `set_default` are applied
+  Narwhals-native.** Custom `parsers=` run on the native frame inside the
+  Narwhals backend; `add_missing_columns=True` and per-`Column` `default=`
+  values are built from Narwhals expressions (`nw.lit`, `fill_null`, `cast`).
+  Added columns follow the same hybrid dtype rule as coercion: plain numpy
+  dtypes are cast Narwhals-native, while extension-dtype or null-valued added
+  columns get their dtype from the pandas dtype engine (so a missing nullable
+  `Int64` column is added as `Int64` with `<NA>`, not `object`).
+- **`groupby` column-check-groups are Narwhals-native; `Hypothesis` checks are
+  delegated.** `groupby=` checks are handled by the Narwhals check backend
+  (building the pandas group dict for pandas-like frames). `Hypothesis` checks
+  are delegated to the native pandas hypothesis backend (scipy). Custom checks
+  that expect the `pd.Series`/`pd.DataFrame` keep working.
+- **Data synthesis strategies** (`schema.strategy()` / `schema.example()`)
+  are unaffected by the backend flag — they operate on the pandas schema API
+  and generate pandas data directly.
+- **`failure_cases` are pandas DataFrames** (including when every failure
+  case is a scalar, e.g. dtype errors). Unlike the native backend, the
+  ``index`` column of the aggregated failure-cases frame is null for
+  column/dataframe-level checks — the Narwhals code path does not report
+  failing row positions (index-component failures do report the failing
+  index value).
+
 (narwhals-opting-out)=
 
 ## Opting out
 
 The Narwhals backend is **off by default**, so no action is needed to
-continue using the native Polars, Ibis, and PySpark backends. If you
+continue using the native Polars, Ibis, PySpark, and pandas backends. If you
 previously opted in and want to switch back, unset the environment variable
 (or set it to `False`):
 
@@ -273,19 +359,36 @@ backend. Follow-up milestones track each of the gaps below:
   frame. This is because scalar Polars frames cannot be converted to PySpark
   without a live ``SparkSession`` at the error-collection site; this gap is
   tracked for a future release.
-* Column-level `coerce=True` is currently a no-op for **all** Narwhals backends
-  (Polars, Ibis, PySpark SQL). Pandera emits a one-time ``SchemaWarning`` per
-  column so the subsequent ``WRONG_DATATYPE`` error is understandable rather than
-  silent. Full column-level coercion support is tracked as a follow-up.
+* Column-level `coerce=True` is currently a no-op for the **non-pandas**
+  Narwhals backends (Polars, Ibis, PySpark SQL). Pandera emits a one-time
+  ``SchemaWarning`` per column so the subsequent ``WRONG_DATATYPE`` error is
+  understandable rather than silent. Full column-level coercion support for
+  those backends is tracked as a follow-up. (The **pandas** Narwhals backend
+  applies column-level coercion via a hybrid of ``nw.cast`` and the pandas
+  dtype engine — preserving native semantics for extension dtypes; see
+  {ref}`pandas differences <narwhals-pandas-differences>`.)
+* The ``index`` column of aggregated ``failure_cases`` frames is null for
+  column/dataframe-level checks — failing row positions are not reported
+  (SQL-lazy backends have no row order; the pandas path follows the same
+  convention). Index-**component** failures do report the failing index value.
 * `coerce` for the Ibis backend (deferred; `Ibis` coerces eagerly today)
-* `add_missing_columns` parser and `set_default` for `Column` fields
-* `group_by`-based checks beyond element-wise and column-wise expressions
+* `add_missing_columns` parser and `set_default` for `Column` fields on the
+  **non-pandas** Narwhals backends (both are Narwhals-native on the pandas
+  backend)
+* `group_by`-based checks beyond element-wise and column-wise expressions on
+  the **non-pandas** Narwhals backends (`groupby` column-check-groups are
+  Narwhals-native on the pandas backend — see
+  ``NarwhalsCheckBackend.apply_groupby``)
 * Element-wise checks for SQL-lazy backends (Ibis and PySpark SQL). As a consequence,
   the shared built-in check suite in ``tests/common/`` does not run for the PySpark
   Narwhals backend (all shared checks are element-wise; running them would produce only
   skips with no useful coverage signal).
-* Schema IO (YAML/JSON) for Narwhals-backed schemas
-* Hypothesis data-synthesis strategies
+* Schema IO (YAML/JSON) for Narwhals-backed schemas. (Unaffected for the
+  pandas backend: `to_yaml`/`from_yaml` operate on the pandas schema API.)
+* Hypothesis checks and hypothesis-based data-synthesis strategies for the
+  **non-pandas** Narwhals backends. (On the pandas backend, `Hypothesis`
+  checks are delegated to the native pandas backend and `schema.strategy()` /
+  `schema.example()` work through the pandas schema API.)
 * `sample=` / `tail=` row sampling for SQL-lazy backends (Ibis and PySpark SQL)
 * `check_unique` (column-level uniqueness) does not produce a per-row boolean
   `check_output`, so `drop_invalid_rows=True` cannot filter rows that fail a

docs/source/reference/narwhals.rst

@@ -6,7 +6,7 @@ Narwhals Backend
 *new in 0.32.0*
 
 Opt-in `Narwhals <https://narwhals-dev.github.io/narwhals/>`__-powered
-validation backend that powers the Polars, Ibis, and PySpark SQL
+validation backend that powers the Polars, Ibis, PySpark SQL, and pandas
 integrations behind a single unified code path. Requires the ``narwhals``
 extra. Enable with ``PANDERA_USE_NARWHALS_BACKEND=True`` or
 :func:`~pandera.set_config`. Backends register lazily on first schema use;

docs/source/supported_libraries.md

@@ -99,7 +99,7 @@ are expressed in the native library's API.
 :widths: 25 75
 
 * - {ref}`Narwhals <narwhals-backend>`
-  - Unified opt-in backend for Polars, Ibis, and PySpark SQL. Enable with ``PANDERA_USE_NARWHALS_BACKEND=True`` or {func}`~pandera.set_config`.
+  - Unified opt-in backend for Polars, Ibis, PySpark SQL, and pandas. Enable with ``PANDERA_USE_NARWHALS_BACKEND=True`` or {func}`~pandera.set_config`.
 :::
 
 ```{toctree}

noxfile.py

@@ -340,6 +340,13 @@ def tests_narwhals_backend(session: Session, extra: str) -> None:
     or tests/ibis/ suite then exercises the narwhals backend rather than the
     native one. Tests that expose narwhals backend gaps should be marked
     xfail in the test files.
+
+    pandas is intentionally NOT in this matrix: the full tests/pandas/ suite
+    exercises features the narwhals backend does not support (column-level
+    coerce, index/multiindex validation, parsers, hypothesis strategies).
+    Narwhals-backed pandas coverage lives in tests/narwhals/ (test_pandas_e2e,
+    test_pandas_narwhals_register, and the backends/ parity suite), which runs
+    in the unit-tests-narwhals CI job.
     """
     deps = PYPROJECT["project"]["optional-dependencies"]
     requirements = [

pandera/api/narwhals/utils.py

@@ -24,6 +24,25 @@
     }
 )
 
+# Eager pandas-like implementations. These are wrapped as ``nw.DataFrame``
+# (then ``.lazy()``-ed by the backend) and unwrap back to a pandas-API frame.
+# They share pandas dtype semantics (e.g. numpy object columns, NaN-based
+# missing integers) that the backend special-cases in a few places.
+_EAGER_PANDAS_LIKE_IMPLEMENTATIONS: frozenset = frozenset(
+    {
+        nw.Implementation.PANDAS,
+        nw.Implementation.MODIN,
+        nw.Implementation.CUDF,
+    }
+)
+
+
+def _is_pandas_like(frame) -> bool:
+    """True if frame is backed by an eager pandas-like implementation."""
+    if not isinstance(frame, (nw.DataFrame, nw.LazyFrame)):
+        return False
+    return frame.implementation in _EAGER_PANDAS_LIKE_IMPLEMENTATIONS
+
 
 def _to_native(frame):
     """Convert a Narwhals frame to its native backend frame.

pandera/api/pandas/array.py

@@ -45,15 +45,24 @@ def dtype(self, value: PandasDtypeInputTypes | None) -> None:
     @staticmethod
     def register_default_backends(check_obj_cls: type):
         from pandera.backends.pandas.register import register_pandas_backends
+        from pandera.config import CONFIG
+
+        use_narwhals_backend = CONFIG.use_narwhals_backend
 
         _cls = check_obj_cls
         try:
-            register_pandas_backends(f"{_cls.__module__}.{_cls.__name__}")
+            register_pandas_backends(
+                f"{_cls.__module__}.{_cls.__name__}",
+                use_narwhals_backend=use_narwhals_backend,
+            )
         except BackendNotFoundError:
             for base_cls in _cls.__bases__:
                 base_cls_name = f"{base_cls.__module__}.{base_cls.__name__}"
                 try:
-                    register_pandas_backends(base_cls_name)
+                    register_pandas_backends(
+                        base_cls_name,
+                        use_narwhals_backend=use_narwhals_backend,
+                    )
                 except BackendNotFoundError:
                     pass
 

pandera/api/pandas/components.py

@@ -13,6 +13,7 @@
 from pandera.api.pandas.types import PandasDtypeInputTypes
 from pandera.dtypes import UniqueSettings
 from pandera.import_utils import strategy_import_error
+from pandera.utils import is_regex
 
 
 class Column(ArraySchema[pd.DataFrame]):
@@ -115,6 +116,23 @@ def _allow_groupby(self) -> bool:
         """Whether the schema or schema component allows groupby operations."""
         return True
 
+    @property
+    def selector(self):
+        """Column selector used by the Narwhals validation backend.
+
+        Mirrors the ``selector`` property on the polars/ibis/pyspark
+        ``Column`` components: returns a regex pattern when ``regex=True``
+        and the plain column name otherwise. Non-string names (e.g. tuples
+        for MultiIndex columns) are returned as-is.
+        """
+        if (
+            isinstance(self.name, str)
+            and not is_regex(self.name)
+            and self.regex
+        ):
+            return f"^{self.name}$"
+        return self.name
+
     @property
     def properties(self) -> dict[str, Any]:
         """Get column properties."""