feat(query): optimize ordered top-k serving

[dbsid]

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

Supersedes #20031.

This draft combines storage, planner, executor, and MySQL-path changes for the Web3 ORDER BY ... LIMIT serving workload without adding a query-result cache or a new planner cache:

• persist composite/prefix page zone maps in Fuse block metadata and use them for Parquet page-range reads;
• prove linear cluster-key ordering, preserve ordered partitions, use Sort(PresortedMerge), and allow ordered top-k early stop;
• make lazy row fetch page-aware;
• reduce short-query session, pipeline, logging, privilege-check, Decimal, and MySQL result-encoding overhead.

The standalone planner-cache commit previously included in this PR has been removed. The remaining five commits are intentionally layered for review.

Rollout guards

• max_cluster_key_ordered_topk_overlap=0 by default and is the single ordered-top-k off switch.
• cluster_key_ordered_topk_lazy_read_threshold=1000 only applies when max overlap is nonzero; setting it to 0 disables the specialized lazy path.
• New page statistics are optional. Readers fall back when they are absent.
• Query/profile logging keeps existing behavior when configured on; the fast path only skips work when those logs are configured off.
• enable_query_result_cache=0 was explicit in every benchmark session.
• The pre-existing mainline enable_planner_cache=1 was also explicit and identical for baseline and PR runs. The removed PR-local setting was enable_standalone_fast_planner_cache; that symbol and its plan metadata no longer exist in the final diff.

Implementation

1. Adds composite cluster-stat pages and prefix min/max evaluation for page pruning.
2. Adds Parquet page-range reads and mixed full-block/page-range table-data-cache entries.
3. Proves fixed cluster-key prefixes and compatible sort suffixes, bounds overlap, preserves stream order, and uses PresortedMerge early stop.
4. Makes row fetch page-aware and reduces row-selection, offset, and bitmap restoration work.
5. Reuses the MySQL runtime, inlines short pipeline workers, makes query-id lookup O(1), and avoids disabled query/profile/access-log work.
6. Adds Decimal zero-comparison and direct MySQL integer/string/decimal encoding paths.

Benchmark

Driver: https://github.com/dbsid/hbx-web3

Environment:

• EC2 i4i.4xlarge, 16 vCPU Intel Xeon Platinum 8375C, single query node, S3 in us-west-2;
• release builds with RUSTFLAGS="-C target-cpu=native";
• enterprise license, 100 GiB on-disk table-data cache, and 16 GiB in-memory table-data cache;
• max_threads=2, max_storage_io_requests=16, distributed pruning off;
• 5 concurrent connections, random parameters, 3,000 measured requests per query, seed 2026071305;
• each query runs round 1 then round 2; round 2 is authoritative.

Binaries:

• baseline 22a762701e: SHA256 590321daac61059e0c90b202e4cddc90e94a6d1d437fa9615c2e4dffff0ec4ec;
• benchmarked tree without the PR-local standalone planner cache a3b66d320c: SHA256 cde801b67c4182c5aee880dbc981727fe7a57f408b9ddca2b40234ffe21f04b4;
• final HEAD 79ca6c8f90: SHA256 1ecabb0220490224250a427b09ce46686d3c22ed705de8bfe0eadccf9a3326b9.

Build	lazy threshold	Query	mean ms	p50 ms	p95 ms	p99 ms	QPS
baseline	1000	Q1	130.43	108.23	199.83	315.82	38.28
baseline	1000	Q2	131.79	118.37	209.00	232.66	37.93
baseline	1000	Flex	136.60	120.94	214.99	280.14	36.58
baseline	0	Q1	33.27	33.04	38.63	41.48	150.23
baseline	0	Q2	46.61	46.07	58.11	63.71	107.22
baseline	0	Flex	47.52	47.30	58.63	63.79	105.14
final PR without standalone cache	1000	Q1	19.54	19.46	22.66	24.34	255.69
final PR without standalone cache	1000	Q2	21.72	21.51	25.57	27.41	230.07
final PR without standalone cache	1000	Flex	22.38	22.17	26.36	28.30	223.33
final PR without standalone cache	0	Q1	19.69	19.58	23.05	24.93	253.84
final PR without standalone cache	0	Q2	21.68	21.51	25.35	27.82	230.48
final PR without standalone cache	0	Flex	22.34	22.16	26.28	28.60	223.62

No-lazy speedups are 1.69x/2.15x/2.13x for Q1/Q2/Flex. Against baseline default lazy, speedups are 6.67x/6.07x/6.10x. Final-head lazy and no-lazy means differ by only 0.2%-0.7%, so the PR removes the baseline's large default-lazy penalty. The 10 ms target is not achieved after removing the standalone planner cache.

The table uses an exact final-HEAD S3 rerun. The attribution and page-ablation tables below use a3b66d320c and adjacent staged binaries so each comparison changes one layer at a time. Final HEAD is 7.0%-10.9% faster than the earlier full run, but that cross-version/run delta is not assigned to a PR layer. The rerun used the i4i instance role after the temporary session files expired.

Attribution

Standalone planner cache on the same old PR binary:

Query	cache off mean ms	cache on mean ms	reduction
Q1	19.13	6.69	12.44 ms / 65.0%
Q2	21.24	8.33	12.91 ms / 60.8%
Flex	21.77	8.71	13.06 ms / 60.0%
SELECT 1	5.17	4.91	0.26 ms / 5.1%

Retained commits with the standalone cache absent:

Layer	Q1 mean ms	Q2 mean ms	Flex mean ms	SELECT 1 mean ms
storage / ordered top-k	28.45	30.68	31.40	5.69
+ short-query execution	19.51	21.94	22.47	5.08
+ MySQL encoding / Decimal	19.16	21.14	21.72	5.10

• Short-query execution contributes 8.94/8.74/8.93 ms, the largest retained-code gain.
• MySQL encoding and Decimal fast paths add 0.35/0.80/0.75 ms.
• Page-aware row fetch contributes about 8 ms for Q2/Flex and is neutral for Q1.
• After excluding row fetch, ordered top-k is mixed from about -2.4% to +2.1% and is not material for these fixed profiles.

Retained optimization	Q1 reduction ms	Q2 reduction ms	Flex reduction ms	Assessment
short-query execution	8.94	8.74	8.93	largest broad gain
page-aware row fetch	-0.23	8.05	7.90	co-primary for wide Q2/Flex
MySQL encoding / Decimal	0.35	0.80	0.76	small additive gain
composite page ranges	0.36	0.70	0.64	small I/O gain

The retained-code ranking is short-query execution first overall, with page-aware row fetch equally important for Q2/Flex. Composite/prefix page zone maps are not the primary optimization.

Page zone-map result

Variant	Q1 mean ms	Q2 mean ms	Flex mean ms
full page-range path	19.10	20.91	21.67
keep PagePruner, discard ranges	19.45	21.61	22.31
disable PagePruner	19.45	21.42	22.32

Page ranges improve latency only 1.8%-3.2%. Full and ablated profiles have identical scan counts:

Query	read rows	scanned block partitions
Q1	4,005	1
Q2	105,184	4
Flex	149,551	4

Page ranges reduce remote bytes (1.13 MiB -> 911.83 KiB for Q2 and 2.37 MiB -> 1.17 MiB for Flex), but do not reduce scanned blocks. Persistent composite/prefix page zone maps are therefore not the main optimization for this workload.

Correctness

• Exact final-HEAD production-data ordered-top-k off/on full-row comparison: Q1 200/200, Q2 200/200, Flex 200/200; zero mismatches and zero errors.
• All 12 final-head benchmark runs contain 3,000 requests at concurrency 5 with zero errors; all 47 retained attribution/ablation round-2 JSON results also have zero errors.
• Final-HEAD (79ca6c8f90) local-storage sqllogic: 105 records passed across ordered top-k lazy read, sort elimination, and data-page options.
• Prefix page-index, page-range selection, mixed table-data-cache, bitmap restoration, composite cluster-stat, prewhere, Decimal, MySQL formatter, and session tests passed.
• cargo check -p databend-query, cargo test -p databend-common-settings --lib, cargo test -p databend-common-sql --tests --no-run, cargo fmt --all -- --check, taplo check, taplo fmt --check, typos, and git diff --check passed.
• Final enterprise binary release build passed.

Tests

• Unit Test
• Logic Test
• Benchmark Test
• No Test - Explain why

Type of change

• Bug Fix (non-breaking change which fixes an issue)
• New Feature (non-breaking change adds functionality)
• Breaking Change (fix or feature that could cause existing functionality not to work as expected)
• Documentation Update
• Refactoring
• Performance Improvement
• Other (please describe):

---

This change is

[github-actions]

🤖 CI Job Analysis (Retry 1)

Workflow: 29544207038

📊 Summary

• Total Jobs: 87
• Failed Jobs: 7
• Retryable: 0
• Code Issues: 7

❌ NO RETRY NEEDED

All failures appear to be code/test issues requiring manual fixes.

🔍 Job Details

• ❌ linux / test_unit: Not retryable (Code/Test)
• ❌ linux / sqllogic / standalone_minio (query, http): Not retryable (Code/Test)
• ❌ linux / sqllogic / standalone_minio (query, hybrid): Not retryable (Code/Test)
• ❌ linux / sqllogic / standalone (query, 4c, hybrid): Not retryable (Code/Test)
• ❌ linux / sqllogic / standalone (query, 4c, http): Not retryable (Code/Test)
• ❌ linux / sqllogic / cluster (query, 4c, http): Not retryable (Code/Test)
• ❌ linux / sqllogic / cluster (query, 4c, hybrid): Not retryable (Code/Test)

---

🤖 About

Automated analysis using job annotations to distinguish infrastructure issues (auto-retried) from code/test issues (manual fixes needed).

[dbsid]

Final head 79ca6c8f90 is mergeable with current main (5876b2de8e). GitHub is waiting for maintainer approval before the fork workflows can start:

• Dev: https://github.com/databendlabs/databend/actions/runs/29481557216
• Typos: https://github.com/databendlabs/databend/actions/runs/29481557516

The prior CI issues are addressed locally: taplo check, taplo fmt --check, typos, and cargo test -p databend-common-sql --tests --no-run all pass.