fix(cn): infer a FILES() schema across every assigned range - #1705
Draft
aocsa wants to merge 1 commit into
Draft
Conversation
On dev the CN reads the FILES() schema from the first broker range and rejects a get_file_schema request carrying more than one, so a query over a multi-file table fails at planning. The FE sends one request with one range per file. parquet_files_schema infers from the first file and makes every other file agree on column count, ASCII case-insensitive name and type, failing closed because the scan reads every file with that schema. Type promotion across files and non-parquet formats stay out of scope. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Draft
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Motivation. The FE answers a FILES() query by sending one
get_file_schemarequest to asingle CN, with one
TBrokerRangeDescper matched file (TableFunctionTable.getGetFileSchemaRequestin the pinned StarRocks tree). On
devthe CN reads the schema from range 0 and refuses anyrequest with more than one range with
multi-file FILES() schema inference is not supported yet (N files); use a single file path. So a FILES() query over a table written as one parquet fileper generator chunk, which is the layout a distributed scan wants, fails at planning time.
What changed (one reviewable unit across two existing files).
experimental/starrocks/src/file_schema.rs: newparquet_files_schema(paths). The first filewins the spelling. Every other file must have the same column count, the same names under ASCII
case-insensitive comparison, and the same
slot_type. Positional slot fields (id,slot_idx,column_pos) are per-file bookkeeping and are not compared. Each mismatch error names theoffending file and the file it disagrees with. The parquet fixture writer moved from the test
module into
pub(crate) mod test_supportso the service tests can reuse it.experimental/starrocks/src/compute_node_service.rs:file_schema_from_attachmentrejects anempty range list, rejects a non-parquet range with an error that names both the format and the
path, collects every path and calls
parquet_files_schema. Signature and placement areunchanged.
I chose whole-set agreement over what native StarRocks does. Its
FileScanner::sample_schemareads
schema_sample_file_countfiles andmerge_schemapromotes conflicting types. That is areasonable contract when the scanner re-reads each file's own footer at scan time. Here the scan
reads every file with the inferred schema, so a file that disagrees would be misread rather than
promoted. Failing closed at planning, with the path in the error, is the honest behaviour for
this engine. Sampling would also make the answer depend on which files happen to be sampled.
Tests. Eight new tests, all in CI's
--no-default-featuresjob. Six infile_schema.rscover the agreeing case and each mismatch kind, among them
multi_file_rejects_column_type_mismatchandmulti_file_accepts_case_differing_column_names.Two in
compute_node_service.rs, among themget_file_schema_attachment_infers_across_multiple_ranges, build a realTGetFileSchemaRequestthe way the FE does and run it through the service function.
How I tested it. On a GB200 box (aarch64) I ran the CI trio: cargo fmt, clippy with warnings as errors, and the CN test suite without the engine feature. All 179 tests pass, including the eight new ones. Nothing here needs a GPU, so CI covers the same ground.
Intentionally not handled. No type promotion across files; a
bigintcolumn in one file anda
doublein another is an error, not a widened column. Non-parquet formats are still rejected.No configuration or documentation changes; the only user-visible text is the error messages.
Self-contained fork PR against
dev. The CN dispatch rewrite that follows moves this functioninto its
ServiceCore; landing the fix first means that PR only relocates an existing call.Checklist
References
aocsa/feat/pin-table-cnumbrella (draft fix(starrocks): demo don't merge #1686, being closed and re-cut).