feat(io): add ParquetFile with row-group statistics()#1
Open
yohplala wants to merge 5 commits into
Open
Conversation
Expose per-row-group Parquet statistics (min/max/null_count) as an Arrow RecordBatch, via a new `arro3.io.ParquetFile` class. The class loads only the Parquet footer (no data pages) via `ArrowReaderMetadata::load` and wraps `arrow-rs`'s `StatisticsConverter`. Implements the design sketched in kylebarron#315. Struct columns are not yet supported upstream (apache/arrow-rs#7364).
Add a companion test to `test_parquet_file_statistics_first_last_row_group_bounds`
that exercises `stats.column("min")[0].as_py()` / `stats.column("max")[-1].as_py()`
directly on the arro3 RecordBatch, with no pyarrow wrapping on the result
side. This locks down the recommended downstream usage pattern for cheap
footer-only min/max reads.
- Drop `pa.record_batch(...)` wrapping on statistics results; use `stats.column(name)` / `.num_rows` / `.schema.names` / `.to_pylist()` directly on the returned arro3 RecordBatch. - Inline `write_parquet(..., max_row_group_size=4)` calls and remove the `_write_multi_row_group` helper. - Merge the two bounds tests (the pyarrow-wrapped and arro3-native variants were testing the same thing) into a single arro3-native test.
`pytest.raises(Exception)` is too loose — it catches any error and would mask unrelated failures. Add `match="not_a_column"` so we verify the expected "column not found" error, not just any exception.
- Export `ParquetFile` from `_io.pyi` (both the import line and `__all__`)
so type checkers and mkdocstrings can see it.
- Rename `schema` getter to `schema_arrow` to disambiguate from a future
Parquet-schema accessor and match pyarrow's `pq.ParquetFile.schema_arrow`.
- Map `page_index=True` to `PageIndexPolicy::Optional` (load if present)
instead of `Required` (error if missing) — the latter is a footgun for
users opting in on files that happen to lack a page index.
- Add `__repr__` showing `num_rows`, `num_row_groups`, `num_columns`, in
line with other arro3 pyclasses.
- Mark the `null_count` field as non-nullable when the default
`missing_null_counts_as_zero=True` path is taken (the array is
guaranteed to have no nulls), and nullable otherwise.
- Add `subclass` to the pyclass attribute to match pyo3-arrow convention.
- Tests:
- Cover the `missing_null_counts_as_zero=False` branch.
- Cover the `skip_arrow_metadata=True` kwarg.
- Cover the `page_index=True` kwarg on a file without a page index,
asserting it does not raise (pins the Optional-vs-Required choice).
- Cover `__repr__`.
- Minor: generalize a test docstring ("Downstream use case:" instead of
"The downstream use case:").
6347601 to
81cbcf8
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a new
arro3.io.ParquetFileclass that reads only the Parquet footer(no data pages) and exposes per-row-group statistics as an Arrow
RecordBatch,implementing the design sketched in kylebarron#315.
The class wraps
parquet::arrow::arrow_reader::ArrowReaderMetadataand usesStatisticsConverterfrom theparquetcrate to produce a three-columnRecordBatch(min,max,null_count), one row per row group. Theminand
maxcolumns take the Arrow data type of the source column;null_countis
UInt64.Design notes
ParquetFile.opencallsArrowReaderMetadata::load, whichparses the Parquet footer and merges the embedded
ARROW:schemametadata.No data pages are read, so construction is cheap.
#[pyclass(module = "arro3.io", name = "ParquetFile", subclass, frozen)],#[classmethod] fn open(...),inner field
meta: ArrowReaderMetadata, getter namedschema_arrow(disambiguates from a future Parquet-schema accessor and matches pyarrow's
pq.ParquetFile.schema_arrow). When PR Expand Parquet API: async, filtering, projection pushdown kylebarron/arro3#313 lands, this class should mergecleanly into its file layout.
page_index=Trueis Optional, not Required. The upstreamFrom<bool>impl maps
truetoPageIndexPolicy::Required, which errors if the pageindex is missing — a footgun for users opting in on files that lack one.
We explicitly map
true → Optional("load if present, skip otherwise").null_countfield nullability is!missing_null_counts_as_zero: whenthe default zero-fill path is taken the array is guaranteed to have no
nulls, so the field is marked non-nullable; when the flag is
Falsethefield is nullable.
Parquet StatisticsConverter does not work for struct columns apache/arrow-rs#7364). Documented on the method.
Arro3IoResult, matching the async pathalready in
parquet.rs. No new.unwrap()s introduced.parquet 58already shipsStatisticsConverterand
ArrowReaderMetadata.Changes
arro3-io/src/parquet.rs— newPyParquetFilepyclass withopen,num_rows,num_row_groups,num_columns,schema_arrow,__repr__,and
statistics.arro3-io/src/lib.rs— register the class.arro3-io/python/arro3/io/_parquet.pyi— typedParquetFilestub.arro3-io/python/arro3/io/_io.pyi— re-exportParquetFile(missing thiswould break type checkers and mkdocstrings).
docs/api/io/parquet.md— docs entry.tests/io/test_parquet.py— 9 new tests covering accessors, int + floatstatistics (with pyarrow footer metadata as ground truth), nullable float
column, first/last row-group bounds pattern, unknown column error,
file-like object input,
missing_null_counts_as_zero=Falsebranch (fieldnullability),
skip_arrow_metadata=Truekwarg, andpage_index=Trueona file without a page index (pins the Optional-vs-Required choice).
Test plan
uv run pytest tests/io/test_parquet.py— 12/12 pass (3 pre-existing + 9 new)uv run pytest tests/— 170/170 passcargo check -p arro3-io— cleancargo fmt -p arro3-io --check— cleanParquetFile.open("data.parquet").statistics("timestamp")returns an
arro3.core.RecordBatchwith the expected schema, slicingfirst/last row group gives correct min/max, works with both a file
path and a
BytesIOfile-like object.Refs: kylebarron#315, apache/arrow-rs#7364