Skip to content

feat(io): add ParquetFile with row-group statistics()#1

Open
yohplala wants to merge 5 commits into
mainfrom
claude/arro3-feature-planning-jet5w
Open

feat(io): add ParquetFile with row-group statistics()#1
yohplala wants to merge 5 commits into
mainfrom
claude/arro3-feature-planning-jet5w

Conversation

@yohplala

Copy link
Copy Markdown
Owner

Summary

Adds a new arro3.io.ParquetFile class that reads only the Parquet footer
(no data pages) and exposes per-row-group statistics as an Arrow RecordBatch,
implementing the design sketched in kylebarron#315.

The class wraps parquet::arrow::arrow_reader::ArrowReaderMetadata and uses
StatisticsConverter from the parquet crate to produce a three-column
RecordBatch (min, max, null_count), one row per row group. The min
and max columns take the Arrow data type of the source column; null_count
is UInt64.

from arro3.io import ParquetFile

pf = ParquetFile.open("data.parquet")
pf.num_rows, pf.num_row_groups, pf.num_columns

stats = pf.statistics("timestamp")  # arro3.core.RecordBatch
lo = stats.column("min")[0].as_py()     # first row group's min
hi = stats.column("max")[-1].as_py()    # last row group's max

Design notes

  • Footer-only. ParquetFile.open calls ArrowReaderMetadata::load, which
    parses the Parquet footer and merges the embedded ARROW:schema metadata.
    No data pages are read, so construction is cheap.
  • Class shape mirrors upstream PR Expand Parquet API: async, filtering, projection pushdown kylebarron/arro3#313. #[pyclass(module = "arro3.io", name = "ParquetFile", subclass, frozen)], #[classmethod] fn open(...),
    inner field meta: ArrowReaderMetadata, getter named schema_arrow
    (disambiguates from a future Parquet-schema accessor and matches pyarrow's
    pq.ParquetFile.schema_arrow). When PR Expand Parquet API: async, filtering, projection pushdown kylebarron/arro3#313 lands, this class should merge
    cleanly into its file layout.
  • page_index=True is Optional, not Required. The upstream From<bool>
    impl maps true to PageIndexPolicy::Required, which errors if the page
    index is missing — a footgun for users opting in on files that lack one.
    We explicitly map true → Optional ("load if present, skip otherwise").
  • null_count field nullability is !missing_null_counts_as_zero: when
    the default zero-fill path is taken the array is guaranteed to have no
    nulls, so the field is marked non-nullable; when the flag is False the
    field is nullable.
  • Struct columns are not supported by the converter yet (upstream limit,
    Parquet StatisticsConverter does not work for struct columns apache/arrow-rs#7364). Documented on the method.
  • Error handling flows through Arro3IoResult, matching the async path
    already in parquet.rs. No new .unwrap()s introduced.
  • No dependency changesparquet 58 already ships StatisticsConverter
    and ArrowReaderMetadata.

Changes

  • arro3-io/src/parquet.rs — new PyParquetFile pyclass with open,
    num_rows, num_row_groups, num_columns, schema_arrow, __repr__,
    and statistics.
  • arro3-io/src/lib.rs — register the class.
  • arro3-io/python/arro3/io/_parquet.pyi — typed ParquetFile stub.
  • arro3-io/python/arro3/io/_io.pyi — re-export ParquetFile (missing this
    would break type checkers and mkdocstrings).
  • docs/api/io/parquet.md — docs entry.
  • tests/io/test_parquet.py — 9 new tests covering accessors, int + float
    statistics (with pyarrow footer metadata as ground truth), nullable float
    column, first/last row-group bounds pattern, unknown column error,
    file-like object input, missing_null_counts_as_zero=False branch (field
    nullability), skip_arrow_metadata=True kwarg, and page_index=True on
    a file without a page index (pins the Optional-vs-Required choice).

Test plan

  • uv run pytest tests/io/test_parquet.py — 12/12 pass (3 pre-existing + 9 new)
  • uv run pytest tests/ — 170/170 pass
  • cargo check -p arro3-io — clean
  • cargo fmt -p arro3-io --check — clean
  • Manual spot check: ParquetFile.open("data.parquet").statistics("timestamp")
    returns an arro3.core.RecordBatch with the expected schema, slicing
    first/last row group gives correct min/max, works with both a file
    path and a BytesIO file-like object.

Refs: kylebarron#315, apache/arrow-rs#7364

claude added 5 commits April 14, 2026 21:29
Expose per-row-group Parquet statistics (min/max/null_count) as an Arrow
RecordBatch, via a new `arro3.io.ParquetFile` class. The class loads only
the Parquet footer (no data pages) via `ArrowReaderMetadata::load` and
wraps `arrow-rs`'s `StatisticsConverter`.

Implements the design sketched in kylebarron#315. Struct columns
are not yet supported upstream (apache/arrow-rs#7364).
Add a companion test to `test_parquet_file_statistics_first_last_row_group_bounds`
that exercises `stats.column("min")[0].as_py()` / `stats.column("max")[-1].as_py()`
directly on the arro3 RecordBatch, with no pyarrow wrapping on the result
side. This locks down the recommended downstream usage pattern for cheap
footer-only min/max reads.
- Drop `pa.record_batch(...)` wrapping on statistics results; use
  `stats.column(name)` / `.num_rows` / `.schema.names` / `.to_pylist()`
  directly on the returned arro3 RecordBatch.
- Inline `write_parquet(..., max_row_group_size=4)` calls and remove the
  `_write_multi_row_group` helper.
- Merge the two bounds tests (the pyarrow-wrapped and arro3-native
  variants were testing the same thing) into a single arro3-native test.
`pytest.raises(Exception)` is too loose — it catches any error and would
mask unrelated failures. Add `match="not_a_column"` so we verify the
expected "column not found" error, not just any exception.
- Export `ParquetFile` from `_io.pyi` (both the import line and `__all__`)
  so type checkers and mkdocstrings can see it.
- Rename `schema` getter to `schema_arrow` to disambiguate from a future
  Parquet-schema accessor and match pyarrow's `pq.ParquetFile.schema_arrow`.
- Map `page_index=True` to `PageIndexPolicy::Optional` (load if present)
  instead of `Required` (error if missing) — the latter is a footgun for
  users opting in on files that happen to lack a page index.
- Add `__repr__` showing `num_rows`, `num_row_groups`, `num_columns`, in
  line with other arro3 pyclasses.
- Mark the `null_count` field as non-nullable when the default
  `missing_null_counts_as_zero=True` path is taken (the array is
  guaranteed to have no nulls), and nullable otherwise.
- Add `subclass` to the pyclass attribute to match pyo3-arrow convention.
- Tests:
  - Cover the `missing_null_counts_as_zero=False` branch.
  - Cover the `skip_arrow_metadata=True` kwarg.
  - Cover the `page_index=True` kwarg on a file without a page index,
    asserting it does not raise (pins the Optional-vs-Required choice).
  - Cover `__repr__`.
- Minor: generalize a test docstring ("Downstream use case:" instead of
  "The downstream use case:").
@yohplala yohplala force-pushed the claude/arro3-feature-planning-jet5w branch 2 times, most recently from 6347601 to 81cbcf8 Compare April 14, 2026 21:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants