feat(fuse): support partitioned table layout#20143
Conversation
github-actions
Bot
added
the
pr-feature
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a9fc8177e0
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d19ad30b9c
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3cc8d455b7
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
@codex review |
|
Codex Review: Didn't find any major issues. You're on a roll. Reviewed commit: ℹ️ About Codex in GitHubCodex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback". |
|
Findings P3 statement.rs (line 1177): table options before PARTITION BY are rejected only when ENGINE=FUSE is explicit. If the engine is omitted, the parser allows CREATE TABLE t(a INT) ROW_PER_BLOCK=1 PARTITION BY(a), and the binder later defaults that same table to Fuse (table.rs (line 588)). The same semantic table is rejected when written as ENGINE=FUSE ROW_PER_BLOCK=1 PARTITION BY(a). Either allow this order for Fuse consistently or reject it after resolving the default engine. |
| } | ||
| let partition_key_indices: Arc<[_]> = | ||
| cluster_stats_gen.cluster_key_index[..cluster_stats_gen.partition_key_count].into(); | ||
| if !partition_key_indices.is_empty() { |
There was a problem hiding this comment.
TransformPartitionBy runs after TransformSortPartial, which only sorts each incoming block. Therefore inserts with many input blocks can emit many small blocks per partition because equal partition values from different input blocks are never merged or shuffled together. This preserves the no-cross-partition invariant, but does not provide a partitioned write layout and can amplify small files/blocks for high-cardinality or interleaved inputs. Either add a partition-key exchange/merge stage, or document this as a local block-splitting invariant and add tests covering fragmented multi-block input.
I suggest as:
input columns
-> add partition expr columns
-> add cluster expr columns
DataBlock
-> HashPartitionExchange(partition_key_columns, N)
-> each lane owns a subset of partition values
For distributed:
NodeToNodeHash(partition_key_exprs)
partition A rows -> blocks -> segments
partition B rows -> blocks -> segments
There was a problem hiding this comment.
I investigated adding partition-aware exchange and
merge stages. i agree that the current pipeline can produce multiple fragments
for the same physical partition when rows arrive in different input blocks,
pipeline lanes, or distributed nodes.
For this PR, however, the intended guarantee is that a block or segment never
spans physical partitions, rather than that all rows of one partition are
globally coalesced during ingestion. Providing the stronger layout guarantee
would require more than a local hash exchange: the exchange must use the
evaluated partition expressions, and each writer must maintain separate bounded
buffers for all partition values assigned to it. The distributed APPEND,
REPLACE, UPDATE, MERGE, and multi-table-insert paths would also need to preserve
their mutation metadata across that routing. High-cardinality partition keys
would additionally require memory limits, eviction, or spill, and can still
produce small blocks even after a perfect exchange.
This is also consistent with the general approach documented by StarRocks. It
routes data through partitions and buckets/tablets, but asks users to choose an
appropriate partition granularity based on data volume and query patterns, and
uses compaction to merge data files produced by loads:
- https://docs.starrocks.io/docs/table_design/data_distribution/#how-to-choose-partitioning-columns-and-granularity
- https://docs.starrocks.io/docs/table_design/data_distribution/#hash-bucketing
- https://docs.starrocks.io/docs/administration/management/compaction/
I therefore chose the documented-invariant option for this PR. The transform
now explicitly states that partition splitting is local to each input block,
and the SQL regression test covers fragmented multi-block input. Existing
partition-aware COMPACT/RECLUSTER paths can merge fragments later without
crossing physical partition boundaries. If ingestion fragmentation proves to
be a practical bottleneck, partition-aware distributed write routing should be
designed separately with bounded buffering, spill/backpressure, and coverage
for all DML paths.
There was a problem hiding this comment.
This is a great approach overall.
However, interleaved partition fragments cannot currently be consolidated by compaction. This can happen within a single parallel or distributed write, or accumulate across multiple writes.
For example, A, B, A, where both A segments belong to the same partition. Since compaction only groups adjacent segments from the same partition, these fragments may persist and increase segment and metadata overhead.
Maybe we need a way to consolidate non-adjacent fragments from the same partition while preserving the required segment order.
# Conflicts: # src/query/ast/src/parser/statement.rs # src/query/service/src/interpreters/interpreter_table_rename_column.rs # src/query/service/src/interpreters/interpreter_table_show_create.rs # src/query/sql/src/planner/binder/ddl/table.rs # src/query/storages/common/table_meta/src/table/table_keys.rs # src/query/storages/fuse/src/fuse_table.rs # src/query/storages/fuse/src/operations/append.rs # src/query/storages/fuse/src/operations/common/meta/mutation_log.rs # src/query/storages/fuse/src/operations/common/processors/transform_block_writer.rs # src/query/storages/fuse/src/operations/common/processors/transform_mutation_aggregator.rs # src/query/storages/fuse/src/operations/common/processors/transform_serialize_block.rs # src/query/storages/fuse/src/operations/recluster.rs # src/query/storages/fuse/src/pruning/fuse_pruner.rs
# Conflicts: # src/query/storages/fuse/src/operations/common/processors/transform_mutation_aggregator.rs
- sqllogictest: verify SET OPTIONS partition_by rejection, MODIFY COLUMN partition-key rejection, unrelated-filter no-pruning, DELETE+compact boundary preservation, and CTAS partition boundary on initial write - unit: partition_values/same_partition fallback cases (None stats, mismatched key id, min!=max prefix) - unit: PartitionPruner::should_keep returns true conservatively when segment has no cluster statistics
4 participants
I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/
Summary
Fuse tables currently have no table-level partition layout, so data rewrites cannot preserve partition boundaries and pruning cannot use exact partition values. This PR adds
PARTITION BYfor Fuse tables.Tests
Coverage includes parser and validation behavior, partition layout preservation, compaction and mutation behavior, exact expression pruning, strict and inclusive ranges, reversed comparisons, OR predicates, multiple partition keys, pruning counts, and query result correctness.
Type of change
This change is