feat(pyarrow): add first-class pyarrow.Table schema validation - #2427
Open
sravankumarkunadi wants to merge 2 commits into
Open
feat(pyarrow): add first-class pyarrow.Table schema validation#2427sravankumarkunadi wants to merge 2 commits into
sravankumarkunadi wants to merge 2 commits into
Conversation
Signed-off-by: Sravan Kumar Kunadi <96312788+sravankumarkunadi@users.noreply.github.com>
sravankumarkunadi
force-pushed
the
feature/pyarrow-table-support
branch
from
July 26, 2026 02:45
947ebbe to
760f5e2
Compare
Author
|
@cosmicBboy kindly review the changes and let me know if anything should be adjusted |
Collaborator
|
hi @sravankumarkunadi please resolve the conflict with |
Signed-off-by: Sravan Kumar Kunadi <96312788+sravankumarkunadi@users.noreply.github.com>
Author
|
Conflict resolved. Kindly review when you get a chance. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #2264
Adds support for validating
pyarrow.Tabledirectly, so Arrow tables get dtype checks, value checks and class-based models instead of callers comparingTable.schemaby hand.validate()returns apyarrow.Table, so a schema can be dropped into an existing pipeline without changing types.Approach
The narwhals backends are already frame-agnostic, so this reuses them rather than adding another validation engine. Most of the diff is the API layer and registry wiring:
pandera/api/pyarrow/DataFrameSchema,Column,DataFrameModel,BaseConfig,PyArrowData, dtype/validation-depth helperspandera/pyarrow.pypandera/ibis.pypandera/typing/pyarrow.pyTable[Model]generic forcheck_typespandera/backends/pyarrow/register.pypyarrow.Tableonto the narwhals backendsChanges to existing modules
Four modules outside the new package needed changes:
narwhals_engine.py— registers the builtinsint,float,strandboolas dtype equivalents. They previously raisedTypeError: data type not understood, which madepa.Column(int)unusable.polars_enginealready registers all four, so this brings narwhals to parity.backends/narwhals/checks.py— normalises pyarrow return values fromnative=Truechecks (Table,*Scalar,*Array/ChunkedArray), and hands single-argument check functions aPyArrowData, matching how polars getsPolarsDataand ibis getsIbisData.backends/narwhals/base.py— the eager failure-case builder round-trips through polars, which a polars-free install cannot do, and the concat step dispatches on.union(), whichpyarrow.Tabledoes not have. Adds pyarrow builders for the eager and scalar paths plus a pyarrow branch in the concat. Routing is byImplementation.PYARROWrather than by whether polars is importable, sofailure_casescomes back as apyarrow.Tableeither way instead of changing type depending on what else is installed. Column layout and values match the polars output exactly.backends/register_checks.py—register_default_check_backendswas eagerly registering the polars, ibis and pyspark backends for anynarwhals.*check object. That primes each library's registrationlru_cache, so a later wipe of the sharedBACKEND_REGISTRYleaves those libraries unregistered for the rest of the process. It surfaced as 63 ibis failures once pyarrow started routing through narwhals. There is now a fast path that returns once a check backend for the narwhals type is registered.Two details worth review
Validation depth. A
pyarrow.Tableis always fully materialised, unlike apl.LazyFrameor anibis.Table, so it defaults toSCHEMA_AND_DATA. Without that,Check.gt(0)on[1, -2, 3]passes silently, which is worse than erroring. Seetest_data_level_check_failure_is_raised.Dtype translation. Only some pyarrow types stringify into something the narwhals engine resolves —
pa.int64()renders as"int64"and works, butpa.float64()renders as"double"andpa.date32()as"date32[day]", which do not.pyarrow_dtype_to_narwhalsround-trips a zero-length array through narwhals instead, which also covers parametrised types (timestamps, decimals, lists, structs).Known limitation
coerce=Trueis a no-op: it emits aSchemaWarningand reports aWRONG_DATATYPEerror. This is inherited rather than introduced — coercion isn't implemented in the narwhals backend at all (polars behaves the same way under it, andtests/narwhals/test_parity.pymarks coercexfail(strict=True)with "coerce is a v2 feature"). Since pyarrow is served only by narwhals, it ships without coercion for now.test_column_coerce_is_not_supportedpins the current behaviour so it fails once coerce lands.Tests
76 new tests in
tests/pyarrow/, covering the container, dtypes, checks, the model API and backend registration.tests/pyarrow tests/polars tests/ibis tests/base tests/coretests/narwhalswithPANDERA_USE_NARWHALS_BACKEND=Truetests/pandas tests/xarray tests/dask tests/geopandastests/pyarrowon a wheel-built venv with only thepyarrowextraThe two failures not listed above are pre-existing and behave identically on
main: 5 pyspark tests needingPYSPARK_PYTHON, andtest_ibis_backend_is_narwhalsneedingPANDERA_USE_NARWHALS_BACKEND=True.That last row is a venv built from the wheel with narwhals and pyarrow only — no pandas, polars, ibis or numpy — to confirm the extra stands on its own. The single skip needs ibis.
prek run --all-filesis clean, both new doctests run, and the docs build (sphinx -W -b=doctest) reports no warnings from the new page.Packaging
Adds a
pyarrowextra pinned topyarrow >= 13, the floor already used inenvironment.ymlandrequirements.txt, so no new dependency is introduced. Also adds the extra to the noxfile'sDATAFRAME_EXTRASand to the CI matrix, plus adocs/source/pyarrow.mdpage.