Skip to content

refactor(query): combine short-query and page-aware execution#20176

Draft
dbsid wants to merge 15 commits into
databendlabs:mainfrom
dbsid:codex/web3-combined-p99-10ms-20260720
Draft

refactor(query): combine short-query and page-aware execution#20176
dbsid wants to merge 15 commits into
databendlabs:mainfrom
dbsid:codex/web3-combined-p99-10ms-20260720

Conversation

@dbsid

@dbsid dbsid commented Jul 20, 2026

Copy link
Copy Markdown

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

Draft integration PR for the reviewed changes in #20165 and #20166.

This branch combines:

  • short-query execution overhead reductions across session, logging, privilege, MySQL runtime, and pipeline executor paths;
  • page-aware lazy RowFetch for Fuse Parquet reads, including bounded page-range work, cache isolation, and full-block fallbacks;
  • the focused correctness and CI race fixes already present on both source PRs.

The purpose of this draft is to continue profiling the combined path and reduce the 5-concurrent p99 latency of Q1, Q2, and Flex below 10 ms. The target is not claimed yet: the prior #20166 c5 lazy benchmark reached p99 values of 34.66, 38.68, and 39.57 ms for Q1, Q2, and Flex respectively.

Composition

Benchmark Contract

Driver: https://github.com/dbsid/hbx-web3

  • EC2 i4i.4xlarge, single query node, S3 in us-west-2, native release build;
  • Q1, Q2, and Flex with 5 concurrent clients and lazy_read_threshold=1000;
  • 3,000 measured requests per query after one warm round, second round authoritative;
  • deterministic seed 2026071305, max_threads=2, max_storage_io_requests=16;
  • distributed pruning and query-result cache disabled; planner cache held constant;
  • zero errors and ordered-result comparison against lazy_read_threshold=0 are required for every claimed result.

Profiling Plan

  1. Establish a red-capable c5 p99 loop on the combined binary.
  2. Measure protocol, executor scheduling, session/logging, RowFetch, Parquet page reads, cache, and result serialization boundaries.
  3. Rank falsifiable hypotheses and change one variable at a time.
  4. Retain only optimizations with focused regression coverage and a measured causal improvement.
  5. Repeat the exact benchmark until Q1/Q2/Flex authoritative p99 are all below 10 ms, or document the measured blocker without claiming completion.

Current Validation

  • Both source PR validation suites passed before composition.
  • The combined merge has no conflicts and git diff --check passes.
  • New profiling and optimization results will be added here as they are produced.

Tests

Type of change

  • Bug Fix
  • New Feature
  • Breaking Change
  • Documentation Update
  • Refactoring
  • Performance Improvement
  • Other (please describe):

This change is Reviewable

@dbsid dbsid changed the title perf(query): combine short-query and page-aware execution refactor(query): combine short-query and page-aware execution Jul 20, 2026
@github-actions github-actions Bot added the pr-refactor this PR changes the code base without new features or bugfix label Jul 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-refactor this PR changes the code base without new features or bugfix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant