feat(scan): the parquet ingestible honors a per-file byte range - #1700
Draft
aocsa wants to merge 1 commit into
Draft
feat(scan): the parquet ingestible honors a per-file byte range#1700aocsa wants to merge 1 commit into
aocsa wants to merge 1 commit into
Conversation
parquet_ingestible_table_info carries resolved_file_ranges parallel to resolved_file_paths; build_file_scan_info keeps only the row groups the range owns (row_groups_in_byte_range) after all_row_groups and before stats pruning, so a range and a filter compose. (0,0) reads the whole file, a range inside one row group is a valid empty split, and a partial range/path pairing is refused at construction. A ranged scan neither serves nor is served by the pinned cache (cache_entry_info::has_byte_ranges). Inert until the FFI layer supplies ranges. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This was referenced Sep 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Layer 2 of the scan stack. The base is
stacked/scan-byte-range-rule(#1696), which providesparquet_byte_range.hppanddetail::row_groups_in_byte_range. This layer teaches the parquet ingestible and the scan cache to honor a per-file byte range. Nothing in this PR populates the range, so with this PR alone every scan still reads whole files and the cache behaves as before.parquet_ingestible_table_infogainsresolved_file_ranges, astd::vector<std::pair<std::uint64_t, std::uint64_t>>parallel toresolved_file_paths. Each entry is(start, length). An empty vector means every file is read whole. A(0,0)entry means that one file is read whole.has_byte_ranges()is true when any entry is something other than(0,0).The
parquet_gpu_ingestibleconstructor refuses a partial pairing. Ifresolved_file_rangesis non-empty and its size differs fromresolved_file_paths, it throwssirius::invalid_input_exceptionwith the message "parquet scan carries {} byte ranges for {} files; a partial pairing would make row-group ownership ambiguous". I would rather fail at construction than guess which file a range belongs to.Row-group selection.
next_split_providerlooks up the file's range by index and passes it tobuild_file_scan_info, which takes a newbyte_rangeparameter. When the range is not(0,0),build_file_scan_inforeplaces the result ofreader.all_row_groups(opts)withdetail::row_groups_in_byte_range(metadata, start, length). Only row groups whose start offset falls inside[start, start+length)survive. This happens before stats pruning, so filter pushdown then prunes within the owned set. A range that owns no row group is a valid empty split. It flows through the existing all-pruned fallback and never turns into a whole-file read. ASIRIUS_LOG_DEBUGline reports how many row groups each range owns.Cache.
cache_entry_infogainsbool has_byte_ranges, whichcache_entry_info::fromcopies from the table info.can_serve_with_columnsreturns a miss when either side has byte ranges, before it compares file sets at all. The cache identifies a parquet scan by its file set, and a ranged scan holds only a fraction of each file's rows. A whole-file pin would hand a ranged scan extra rows. A pin built from a ranged scan would hand a whole-file scan missing rows. Missing in both directions is the simple safe answer, and I did not try to key the cache on ranges.Two tests live in
test/cpp/scan/test_parquet_byte_range.cpp, tagged[parquet_byte_range][scan].row_group_size_rowsto a 5000-row floor, so that is 10 real row groups. For each half of the file it drives the production path:make_ingestible, thennext_split_provider, thencreate_batch_coalescerpush and flush, with akvikio_contextas the io context. The row-group indices come out ofparquet_split_info::rg_slices. Row counts come from the footer metadata, no decode. It asserts both halves are non-empty and disjoint and that their rows sum to 50000. A(0,0)range yields all 50000 rows and the same row-group count as the two halves together. The range(10, 5), which sits inside one row group, yields zero row groups and zero rows.make_ingestibleand expectssirius::invalid_input_exception.One more test lands in
test/cpp/scan/test_can_serve_with_columns.cppunder[scan][can_serve].(0, 4096). A pin built from that ranged scan does not serve a whole-file scan. A control pin built from the whole-file scan does serve it with projection{0}, so the miss comes from the range and not the file set. A(0,0)entry still hits.Run them with:
How I tested it. On a GB200 box (aarch64) I did an incremental build and ran the Catch2 tags
[parquet_byte_range],[can_serve]and the full[scan]suite on a GPU; the whole scan suite passes, including the two new byte-range cases and the newcan_serveguard test. These are GPU-only tests, so CI does not run them.Intentionally not handled here. The pin-table file-subset serve path (
chunk_file_pathsprovenance,matches_parquet_file_set,allowed_chunks, the coalescer file-boundary mode) is the next scan layer. CarryingFileOrFiles.start/lengthfrom the Substrait plan intoresolved_file_rangesis the ffi stack. Until that lands, this PR changes no behavior on its own.What I want eyes on: the ordering in
build_file_scan_info, range selection first and stats pruning second, and whether a blanket cache miss for any ranged scan is the right call for now.Checklist
References
has_byte_rangescoverage; that draft is closed).sirius_scan_manager.cppthat this PR's hunks sit beside without touching.