chore(bench): add TPC-H harness and GPU/MIG placement - #1738
Draft
aocsa wants to merge 7 commits into
Draft
Conversation
… launcher Plan piece 10, the Q1/Q6 subset of D2a/D2b/D2c; future branches stacked/docs-cn-runbooks, stacked/bench-tpch-harness-oracle and stacked/bench-cn-distribution. The one-CN-per-GPU proof needs: cluster8.sh, benchmarks/tpch (bench.sh, q01, q06, README), tools/oracle.py and compare.py from bench/rtxpro6000-2gpu/tools, gen-tpch.sh and scripts/cn-distribution.py. Fixes over SOT: cluster8.sh unsets an inherited CUDA_VISIBLE_DEVICES (it beats --gpu-device) and exports SIRIUS_QUERY_WATCHDOG_SECS; bench.sh takes ORACLE_DIR and compare.py exits non-zero on any mismatch, so a wrong answer fails the sweep; oracle.py spills under $TMPDIR. Drift: README trimmed to what ships; bench.sh's header drops "all 22" and lazy nixl setup (C7 pre-warms sessions); cn-distribution.py hints rm -rf .cn*/telemetry/* since clean-telemetry.sh is absent; gen-tpch.sh keeps SOT's /opt/dlami/nvme/tpch default. Left out: q02-q22, analyze.py, run-comparison.sh, setup-engine-b.sh, QUERY-DEVIATIONS.md, RETARGETING.md (D2b); BUILDING.md, clean-telemetry.sh, cluster8 -cfg/-numa (D2a); the pinned kit (D3). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The tree carried only q01 and q06; the other twenty lived on the source branch, so no arm was reproducible from the repo. q02..q22 and QUERY-DEVIATIONS.md come from perf/profile-sf1000 unchanged, except q11: its HAVING fraction is now `0.0001 / __TPCH_SF__`, substituted with $TPCH_SF (default 1) by bench.sh and oracle.py. With the stock constant the query returns zero rows on both engines at SF1000 (measured, arms P-2cn and D-on), which compare.py then reported as an EMPTY on both sides. compare.py compared only the last non-empty run. In arm P-2cn q15's cold run returned zero rows and its two warm runs one row; the old output hid the flake behind the warm verdict. Every run is compared now, the per-query verdict is the worst of its runs, per-run verdicts are printed when they differ, and the exit status is 0 only when every run matches. Tested: `bash -n bench.sh`; compare.py on arms/P-2cn against the SF1000 oracle reports q15 as `EMPTY (r0 of 3 runs)` with the three per-run lines, the other 21 verdicts unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
At SF1000 q11 returns 936,989 rows and thousands share a value, so two correct engines order the ties differently and the row-order-sensitive compare reported VALUES-DIFFER on tie order alone (arm W1-1cn: 12,790 mismatched cells, all swapped ties). ps_partkey is unique per output row. Recorded in QUERY-DEVIATIONS.md with the SF scaling of the fraction. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…e = 0) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…A_VISIBLE_DEVICES
aocsa
force-pushed
the
codex/tpch-mig-benchmark-harness
branch
from
September 9, 2026 17:22
08820f3 to
a101880
Compare
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add one reproducible StarRocks/Sirius TPC-H workflow:
FILES()-based query texts, including the scale-correct q11 and the CTE-reuse guidance;GPU_DEVICESfor shared-GPU development and futureMIG_DEVICESlauncher plumbing.cluster8.shdocuments the resource contract explicitly: each CN consumesGPU_MEMplus itsSTAGINGarena outside that limit, plus CUDA context overhead. The README now states that an actual multi-CN run still requires #1714's CN bring-up/flags, #1693's stable staging arena, and the follow-on distributed exchange runtime.It also marks UUID-based
MIG_DEVICESunsupported on the current engine because its NVML device-count check fails under UUID-only visibility; use whole-GPU ordinals such asGPU_DEVICES=0,1on that box for now.Why one PR
The scripts, query kit, oracle/comparator, and topology launcher form one reproducible measurement and correctness workflow. Splitting them would leave a reviewer unable to run or validate the workflow end to end. The source commits are preserved as six cherry-picks, followed by a documentation-only prerequisite clarification.
Validation
pixi run bash -nforgen-tpch.sh,cluster8.sh, andbench.shpixi run python -m py_compileforcn-distribution.py,compare.py, andoracle.pyr0) is rejected while a correct warm run (r1) passes; the overall comparator exits non-zero.Known gate limitation, intentionally not changed here:
compare.pycurrently treatsnanversus a finite oracle value as a match because itsd > tolerancecomparison is false for NaN. A focused follow-up should reject non-finite numeric values before using this as a strict correctness gate.Draft status and runtime scope
This remains a Draft because
devdoes not yet contain the distributed runtime needed for real multi-CN execution. The harness and documentation are ready to review independently; runtime measurements should wait for the prerequisite series.