Skip redundant Vitess rebuilds and record per-task timings - #699
Open
arthurschreiber wants to merge 1 commit into
Open
Skip redundant Vitess rebuilds and record per-task timings#699arthurschreiber wants to merge 1 commit into
arthurschreiber wants to merge 1 commit into
Conversation
Rebuilding Vitess from source is the most expensive step of a benchmark run (several minutes). When one commit is benchmarked across several workloads the cluster is torn down and recreated between each workload, rebuilding the exact same binaries every time. On the production host this cost ~5-7.5 min per workload switch (first-of-config runs took 430-566s vs 110-254s for repeats). vitess_build now records the installed ref+Go version in /usr/local/vitess-build.ref and skips the git fetch and make install when it already matches and the binaries are still present. The marker is removed before a build and written only after a fully successful install, so an interrupted build cannot cause the next run to wrongly skip. To let the binaries survive between workloads, the vtctld/vtgate/vttablet teardown no longer removes them from /usr/local/bin (a rebuild overwrites them, so nothing accumulates). Also add ansible/ansible.cfg enabling the profile_tasks callback and point ansible-playbook at it via ANSIBLE_CONFIG (it is not otherwise discovered from the per-execution playbook directory). Every execution's exec-stdout.log now ends with a per-task timing breakdown. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
frouioui
approved these changes
Jul 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
Rebuilding Vitess from source is the single most expensive step of a benchmark run (several minutes). When one commit is benchmarked across the ~7 workloads, the cluster is torn down and recreated between each workload even though the exact same binaries are needed, so
make installruns ~7 times for identical output.Production timing over a 4.3-day sample shows the cost: first-of-config runs took 430–566s vs 110–254s for repeats of the same config — roughly ~25–30 wasted minutes per benchmarked commit.
Changes
Skip the rebuild when nothing changed
vitess_buildnow records the installed ref+Go version in/usr/local/vitess-build.refand skips the git fetch andmake installwhen it already matches and the binaries are still present. The marker is removed before a build and written only after a fully successful install, so an interrupted build can't cause the next run to wrongly skip.To let the binaries survive between workloads of the same commit, the
vtctld/vtgate/vttabletteardown no longer removes them from/usr/local/bin(a rebuild overwrites them, so nothing accumulates).The
when:guard uses| boolto avoid the ansible string-truthiness trap (aset_factresult of"False"is otherwise truthy).Per-task timing (observability)
Adds
ansible/ansible.cfgenabling theprofile_taskscallback, wired viaANSIBLE_CONFIG(ansible-playbookdoes not otherwise discover a config from the per-execution playbook directory). Every execution'sexec-stdout.lognow ends with a per-task timing breakdown — the data needed to evaluate further optimizations (e.g. snapshotting the TPCC dataset).Testing
ansible-lint(v25.12.2, matching CI,productionprofile) — 0 failures ✅go build ./go/...+golangci-lint run ./go/infra/...(v2.12.2) — 0 issues ✅Expected effect
Config switches within one commit drop from ~430–566s to roughly the repeat cost plus cluster start (~2–3 min).
🤖 Generated with Claude Code