feat(scan): serve pinned parquet tables to file-subset scans - #1717
Draft
aocsa wants to merge 1 commit into
Draft
Conversation
A pinned parquet table only served scans whose file set equaled the pinned set, so a whole-glob pin never matched StarRocks' per-CN file subsets or a DuckDB read_parquet over some of the pinned files. The pin path now coalesces within file boundaries and records each chunk's canonical source files; the serve path classifies a scan as exact, strict subset, or miss and serves a subset from the covered chunks only, restricted before zone-map pruning. Entries without provenance and byte-range scans keep exact-only matching. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Draft
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
A pinned parquet table only served a scan whose resolved file set equaled the pinned set. StarRocks hands each compute node a per-query subset of a table's files, so a whole-glob pin on each CN never matched and served nothing while still holding the memory. The plain DuckDB path has the same gap. Pin a glob, query two of its three files, and the read falls through to disk.
This PR makes a pin serve any strict subset of its files. It is one unit from pin time to serve time.
Pin time.
build_parquet_pin_infoturns on a new coalescer mode,batch_within_file_boundaries, so no pinned chunk bundles two files.materialize_pin_batchesrecords each chunk's source files (canonicalized, sorted, deduplicated) and the three sinks push that vector unconditionally so it stays parallel to the chunks. The threeinsert_pinned_entry*paths store it aspinned_entry::chunk_file_pathsand throw on a length mismatch. The GPU merge path adopts provenance an existing entry lacks and throws when the two disagree.Serve time.
cache_entry_info::matches_parquet_file_setclassifies a scan as exact, strict subset, or miss. It is now the one parquet matcher behind bothcan_serve_with_columnsand the plan-time residency gate;matches_parquet_filesstays as an exact-only wrapper with no production caller.try_match_cached_entryselects the chunks whose provenance the scan covers and hands them tobuild_cached_scan_planasallowed_chunks. I apply that restriction before zone-map pruning so the all-pruned sentinel can only pick an allowed chunk; a sentinel outside the set would return another file's rows.find_pinned_entry_for_parquet_filesaccepts a subset under the same provenance condition, so the plan-time residency gate and the serve path agree. The header doc oncan_serve_with_columnsspells out that a non-empty projection on the subset branch is necessary but not sufficient; the caller still has to checkchunk_file_paths.Subset matching requires duplicate-free canonical sets on both sides. A duplicated pinned path means that file's chunks were materialized twice, and serving them to a scan that names the file once would double its rows. Exact matching stays duplicate-symmetric, so existing pins behave as before. Entries without provenance (pins made before this change, duckdb-native pins) keep exact-only matching. Byte-range scans never serve or get served; the guard from #1700 in
can_serve_with_columnsstays above the new check.Tests added, all Catch2:
test/cpp/scan/test_can_serve_with_columns.cpp:matches_parquet_file_setover exact, subset, miss, superset, empty, duplicate poisoning, and a duckdb entry.test/cpp/scan_manager/test_cached_serving_hardening.cpp:build_cached_scan_planwith an allowed-chunks restriction (identity, out-of-range indices dropped, empty set);insert_pinned_entryprovenance on all three insert paths (arity throwsstd::invalid_argument, merge adopts, merge mismatch throwsstd::runtime_errorand leaves the entry intact), plus the residency gate refusing a subset probe until provenance exists.test/cpp/scan/test_parquet_scan_sizing.cpp: the coalescer bundles three files into one split by default and emits three single-file splits withbatch_within_file_boundaries.test/cpp/integration/test_pin_table_file_subset.cpp(new, registered inCMakeLists.txt): pins a three-file glob on the gpu and host tiers, checks one file per chunk in the recorded provenance, then runs two-file, one-file, exact, superset, filtered, and selects-nothing scans against the unpinned read and asserts theserves operator ... as a file subsetlog line where a subset was served.How I tested it. On a GB200 box (aarch64) I did a full release build in a worktree, then ran the scan, cache and pin_table Catch2 suites on a GPU, plus the new end-to-end test. Everything passes and pre-commit is clean. These are GPU tests that CI does not run.
The source branch measured this on 2x RTX PRO 6000, SF100 lineitem, 2 CNs, q06 shape, at 0.63 s unpinned and 0.09 s pinned, byte-identical to the DuckDB oracle. I did not re-measure here.
Not handled. A superset scan is a miss. Duckdb-native pins record no provenance. A pin over many tiny files now yields at least one chunk per file. The FE lever (
files_query_whole_file_ranges), the FFIpin_table, and the CN admin channel are later PRs in this series.Layer 3 of the scan stack; base stacked/scan-byte-range-ingestible (#1700).
Checklist
References