refactor(query): reduce short-query execution overhead

[dbsid]

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

Extract the short-query execution changes from #20144 into an independently reviewable PR.

This PR reduces fixed per-statement overhead in the MySQL, session, interpreter, logging, privilege-check, and pipeline executor paths. It does not change storage reads, lazy RowFetch, ordered top-k, or the physical sort implementation.

It adds no planner cache or query-result cache. Benchmarks explicitly use the same pre-existing mainline planner cache setting for baseline and PR (enable_planner_cache=1) and disable query-result caching (enable_query_result_cache=0).

Implementation

1. Reuse the shared MySQL query runtime instead of creating and destroying a query-scoped runtime for each statement.
2. Reuse a non-Tokio thread calling QueryPipelineExecutor::execute() as one pipeline worker, avoiding one additional OS thread spawn/join. Both PipelinePullingExecutor and PipelineCompleteExecutor invoke it from their existing dedicated executor threads.
3. Initialize unchanged session settings once per query context.
4. Avoid building query/profile/access log payloads when every applicable sink is off. File logging, history-table sinks, and per-thread dynamic log capture retain their existing behavior.
5. Return immediately for direct user grants before resolving ownership metadata.
6. Keep query-id history order while adding a case-insensitive O(1) lookup index.

The benchmark plans remain Sort(Single) plans. None contains Sort(PresortedMerge).

Benchmark

Driver: https://github.com/dbsid/hbx-web3

Environment:

• EC2 i4i.4xlarge, 16 vCPU Intel Xeon Platinum 8375C, single query node, S3 in us-west-2;
• release binaries built on the EC2 host with RUSTFLAGS="-C target-cpu=native";
• enterprise license, 100 GiB on-disk table-data cache, and 16 GiB in-memory table-data cache;
• max_threads=2, max_storage_io_requests=16, distributed pruning off;
• 5 concurrent connections with random parameters, 3,000 measured requests per query, seed 2026071305;
• round 1 warms the path and round 2 is authoritative;
• independent server restart for lazy_read_threshold=1000 and 0.

Binaries:

• baseline 5876b2de8e: SHA256 ce15f1261baa0103011abeab12c3e06fc4730deef656b12582affc8b0fa7d4ab;
• PR 7524fce781: SHA256 94d1df99896f11444650a5edd49ec73454d0305188e2228d5868fbde74b57407.

Build	lazy threshold	Query	mean ms	p50 ms	p95 ms	p99 ms	QPS
baseline	1000	Q1	128.67	107.17	195.58	312.53	38.79
baseline	1000	Q2	131.41	117.87	211.36	232.26	38.03
baseline	1000	Flex	135.45	119.52	213.17	275.63	36.89
PR	1000	Q1	120.56	99.03	188.01	310.27	41.45
PR	1000	Q2	123.20	109.69	198.67	219.62	40.57
PR	1000	Flex	128.39	112.30	207.86	271.53	38.92
baseline	0	Q1	33.20	33.14	38.63	40.98	150.52
baseline	0	Q2	45.56	45.03	57.12	61.44	109.66
baseline	0	Flex	46.86	46.72	58.06	62.41	106.64
PR	0	Q1	24.29	24.22	28.64	30.70	205.74
PR	0	Q2	37.50	36.99	48.14	53.05	133.24
PR	0	Flex	38.23	37.99	49.17	53.93	130.69

The PR removes 7.05-8.21 ms with default lazy reads and 8.07-8.90 ms with lazy reads disabled. The no-lazy mean improvement is 17.7%-26.8%; the default-lazy mean improvement is 5.2%-6.3%. The similar absolute reduction across both modes is consistent with fixed execution overhead rather than a storage-plan change.

Correctness

• All six authoritative PR runs completed 3,000 requests with zero errors.
• Full ordered-result comparison between lazy_read_threshold=0 and 1000: Q1 200/200, Q2 200/200, Flex 200/200; zero mismatches and zero errors.
• All six captured PR plans use the existing sort path and contain no PresortedMerge.

Validation

• cargo fmt --all -- --check
• git diff --check
• cargo clippy -p databend-query --lib --tests -- -D warnings
• cargo test -p databend-query --lib -- --test-threads=1 (198 passed)
• focused pipeline executor, session-settings, logging-helper, and k-way merge/fuzz regressions (9 passed)
• release databend-query build with RUSTFLAGS="-C target-cpu=native"

Tests

• Unit Test
• Logic Test
• Benchmark Test
• No Test - Explain why

Type of change

• Bug Fix (non-breaking change which fixes an issue)
• New Feature (non-breaking change which adds functionality)
• Breaking Change (fix or feature that could cause existing functionality not to work as expected)
• Documentation Update
• Refactoring
• Performance Improvement
• Other (please describe):

---

This change is 

Native protocol validation

The original benchmark table above used the MySQL driver. The same workload was also tested through Databend's non-MySQL HTTP protocol using lake-go (lake://...:8000/default, HTTP /v1/query). This validates the native protocol path with the same 5 concurrent workers, fixed random seed 2026071305, and 3,000 measured requests per query.

The native HTTP A/B (the native path is unchanged by the follow-up safety commit 40c4cfcaab) produced the following round-2 results:

lazy threshold	Query	baseline mean ms	PR mean ms	reduction
1000	Q1	128.51	121.77	5.2%
1000	Q2	130.15	123.70	5.0%
1000	Flex	134.99	128.39	4.9%
0	Q1	34.32	25.70	25.1%
0	Q2	48.16	39.62	17.7%
0	Flex	48.87	40.46	17.2%

All 36,000 native A/B requests succeeded with zero errors. On the final 40c4cfcaab binary, a fresh no-lazy round also completed 3,000/3,000 requests for each of Q1, Q2, and Flex with means of 26.84, 40.20, and 40.35 ms respectively. Full ordered-result comparison on the final binary returned 100/100 matches for Q1, Q2, and Flex, with zero mismatches and zero errors.

The benchmarked safety-follow-up release binary was built on the EC2 host with RUSTFLAGS="-C target-cpu=native":

• HEAD: 40c4cfcaab47cc907b97a2c5ce374c57df0a6dd5;
• SHA256: a754c99d3c3cc0a775d6c8fc34e29d3801a3ddc45eb49ec6884f0ce56f663be6.

Applicability and large queries

The native HTTP and FlightSQL handlers both use InterpreterFactory -> Interpreter::execute -> PipelinePullingExecutor/PipelineCompleteExecutor -> QueryPipelineExecutor, so the shared executor, session-settings, disabled-log, privilege, and query-id fast paths are protocol-independent. The shared MySQL runtime change in mysql_interactive_worker.rs is MySQL-only.

The benchmark plans remain Sort(Single); no result depends on Sort(PresortedMerge), ordered top-k, lazy RowFetch, or storage block pruning. The pulling path keeps the same total worker count (threads_num - 1 spawned workers plus the caller worker). Complete pipelines retain their dedicated executor thread, but that thread can itself serve as one pipeline worker. Thus the change removes fixed per-statement overhead, but does not reduce scan rows, scanned blocks, or analytical-query parallelism. Empty/no-source pipelines and callers that already run inside a Tokio runtime are not eligible for the inline worker shortcut.

A controlled native HTTP A/B on the final workload used the following broad queries:

• SELECT count(), sum(number), avg(number) FROM numbers(100000000);
• 10M-row group-by and sort with LIMIT 1000;
• a broad aggregate over the 100M-row Q1 table;
• a broad Q1 Top-K over the 100M-row table.

query shape	baseline mean ms	final HEAD mean ms	requests	conclusion
numbers(100M) aggregate	39.61	39.39	30	neutral
10M group-by/sort	101.62	107.19	10	within run variance
100M real-table aggregate	420.87	419.61	5	neutral
100M real-table Top-K	2476.01	2458.57	3	neutral; small sample

All large-query samples completed without errors. These results support applicability to large queries as no-regression/neutral, not as a claim of material scan or execution-speedup.

Runtime safety follow-up

CI exposed that unconditionally running an inline worker can panic when a generic caller invokes QueryPipelineExecutor::execute() inside a Tokio runtime (SyncSenderSink::blocking_send). 40c4cfcaab keeps the inline shortcut only on a non-Tokio thread and falls back to the original all-worker spawn behavior inside a runtime. Regression coverage now exercises both caller modes and verifies that the pipeline finish hook runs exactly once.

Configured logging follow-up

Deep review found that checking only log.query.on, log.profile.on, or the tracing target could suppress payload construction when a history-table sink or per-thread dynamic capture was the actual consumer. eb9c16d925 checks the configured query/profile/access sink explicitly and keeps the disabled-log fast path only when no consumer needs the event. It also adds focused tests for history-only configuration, dynamic capture, executor caller modes, and query-context settings snapshots.

The latest review-fix release binary was built on the EC2 host with RUSTFLAGS="-C target-cpu=native":

• HEAD: eb9c16d92556a6ab09c09e0e35950ba6b9d6e323;
• SHA256: d18d8b82eb5f37e9955951b2f8d1ed1a49a63d70faa5c4e324b956e993bf15c8.

Native correctness follow-up

After the service recovered, the final 40c4cfcaab binary was rechecked through lake-go over native HTTP with the same deterministic parameter generator:

./web3lake -driver lake -dsn "$LAKE_DSN" -mode compare -query both -iterations 500 -seed 2026071305 \
  -compare-off-sql "SET lazy_read_threshold=0; SET max_threads=2; SET max_storage_io_requests=16; SET enable_distributed_pruning=0; SET enable_query_result_cache=0; SET enable_planner_cache=1" \
  -compare-on-sql "SET lazy_read_threshold=1000; SET max_threads=2; SET max_storage_io_requests=16; SET enable_distributed_pruning=0; SET enable_query_result_cache=0; SET enable_planner_cache=1"
./web3lake -driver lake -dsn "$LAKE_DSN" -mode compare -query flex -iterations 500 -seed 2026071305 \
  -compare-off-sql "SET lazy_read_threshold=0; SET max_threads=2; SET max_storage_io_requests=16; SET enable_distributed_pruning=0; SET enable_query_result_cache=0; SET enable_planner_cache=1" \
  -compare-on-sql "SET lazy_read_threshold=1000; SET max_threads=2; SET max_storage_io_requests=16; SET enable_distributed_pruning=0; SET enable_query_result_cache=0; SET enable_planner_cache=1"

Q1, Q2, and Flex each returned 500/500 ordered-result matches, with zero mismatches and zero errors (1,500/1,500 total). The JSON artifacts are final/native-correctness-i500-seed2026071305.json and final/native-correctness-flex-i500-seed2026071305.json under the benchmark result directory. go test ./... for the driver also passed.

Query-shape qualification

The optimization is protocol-independent for correctness, but its measurable benefit is query-shape dependent:

• Result-set SELECT queries that use a pulling pipeline reuse PipelinePullingExecutor's dedicated thread as one worker and receive the shared fixed-overhead reductions.
• INSERT, UPDATE, DELETE, CTAS, and other complete pipelines still use CompleteExecutor's dedicated thread. On a non-Tokio thread it also serves as one pipeline worker, so complete pipelines avoid one additional worker spawn while preserving their dedicated-executor boundary.
• No-source or already-finished statements do not create a pipeline executor, so only the applicable session/logging/privilege fast paths can help.
• A generic caller already running inside a Tokio runtime uses the original all-worker behavior for safety; this avoids blocking_send panics.

Therefore “effective for all queries” means the change is safe on the common public execution paths, not that every query shape gets the same speedup. For large search or analytical queries it is suitable as a no-regression refactoring, but the expected absolute gain is small relative to scan, join, aggregation, and sort time. It should not be presented as a block-pruning or analytical-query acceleration.

[sundy-li]

https://github.com/dbsid/hbx-web3 is not accessable.

[github-actions]

🤖 CI Job Analysis (Retry 1)

Workflow: 29518925692

📊 Summary

• Total Jobs: 87
• Failed Jobs: 2
• Retryable: 0
• Code Issues: 2

❌ NO RETRY NEEDED

All failures appear to be code/test issues requiring manual fixes.

🔍 Job Details

• ❌ linux / test_unit: Not retryable (Code/Test)
• ❌ linux / test_private_tasks: Not retryable (Code/Test)

---

🤖 About

Automated analysis using job annotations to distinguish infrastructure issues (auto-retried) from code/test issues (manual fixes needed).

[dqhl76]

log_enabled! does check dispatch settings, but only asks whether any dispatch accepts the metadata. So if one broad dispatch, such as QueryLogCollector, accepts it, log_enabled! returns true even when the specific history/file dispatch rejects it.

So, Unfortunately, in our system, this check almost always returns true.

You can test this by adding dbg! here, and set the config to

[log.query]
on = false

and will get

[src/query/service/src/interpreters/common/query_log.rs:131:9] log::log_enabled!(target: "databend::log::query", log::Level::Info) = true
[src/query/service/src/interpreters/common/query_log.rs:132:9] log::log_enabled!(target: "databend::log::query", log::Level::Debug) = true
[src/query/service/src/interpreters/common/query_log.rs:133:9] log::log_enabled!(target: "databend::log::query", log::Level::Trace) = true

[dbsid]

Fixed in 0475969. I removed the ineffective target-specific log_enabled! checks and gate query-log serialization/writes with GlobalConfig::instance().log.query.on before constructing the payload. Profile serialization is separately gated by log.profile.on; normal query execution and logging-on behavior remain unchanged.

[dbsid]

Follow-up eb9c16d925 keeps the fast path without relying on log_enabled!: query/profile/access payloads are enabled from the explicit file sink, the matching history-table sink, or an active per-thread capture. This also fixes history-only configurations that 0475969b07 did not cover. Focused history/capture tests, full single-threaded databend-query lib tests (198/198), and clippy with -D warnings pass.