Skip to content

Skip redundant Vitess rebuilds and record per-task timings - #699

Open
arthurschreiber wants to merge 1 commit into
mainfrom
skip-redundant-vitess-rebuilds
Open

Skip redundant Vitess rebuilds and record per-task timings#699
arthurschreiber wants to merge 1 commit into
mainfrom
skip-redundant-vitess-rebuilds

Conversation

@arthurschreiber

Copy link
Copy Markdown
Member

Context

Rebuilding Vitess from source is the single most expensive step of a benchmark run (several minutes). When one commit is benchmarked across the ~7 workloads, the cluster is torn down and recreated between each workload even though the exact same binaries are needed, so make install runs ~7 times for identical output.

Production timing over a 4.3-day sample shows the cost: first-of-config runs took 430–566s vs 110–254s for repeats of the same config — roughly ~25–30 wasted minutes per benchmarked commit.

Changes

Skip the rebuild when nothing changed

vitess_build now records the installed ref+Go version in /usr/local/vitess-build.ref and skips the git fetch and make install when it already matches and the binaries are still present. The marker is removed before a build and written only after a fully successful install, so an interrupted build can't cause the next run to wrongly skip.

To let the binaries survive between workloads of the same commit, the vtctld/vtgate/vttablet teardown no longer removes them from /usr/local/bin (a rebuild overwrites them, so nothing accumulates).

The when: guard uses | bool to avoid the ansible string-truthiness trap (a set_fact result of "False" is otherwise truthy).

Per-task timing (observability)

Adds ansible/ansible.cfg enabling the profile_tasks callback, wired via ANSIBLE_CONFIG (ansible-playbook does not otherwise discover a config from the per-execution playbook directory). Every execution's exec-stdout.log now ends with a per-task timing breakdown — the data needed to evaluate further optimizations (e.g. snapshotting the TPCC dataset).

Testing

  • ansible-lint (v25.12.2, matching CI, production profile) — 0 failures ✅
  • go build ./go/... + golangci-lint run ./go/infra/... (v2.12.2) — 0 issues ✅

Expected effect

Config switches within one commit drop from ~430–566s to roughly the repeat cost plus cluster start (~2–3 min).

🤖 Generated with Claude Code

Rebuilding Vitess from source is the most expensive step of a benchmark run
(several minutes). When one commit is benchmarked across several workloads the
cluster is torn down and recreated between each workload, rebuilding the exact
same binaries every time. On the production host this cost ~5-7.5 min per
workload switch (first-of-config runs took 430-566s vs 110-254s for repeats).

vitess_build now records the installed ref+Go version in
/usr/local/vitess-build.ref and skips the git fetch and make install when it
already matches and the binaries are still present. The marker is removed
before a build and written only after a fully successful install, so an
interrupted build cannot cause the next run to wrongly skip. To let the
binaries survive between workloads, the vtctld/vtgate/vttablet teardown no
longer removes them from /usr/local/bin (a rebuild overwrites them, so nothing
accumulates).

Also add ansible/ansible.cfg enabling the profile_tasks callback and point
ansible-playbook at it via ANSIBLE_CONFIG (it is not otherwise discovered from
the per-execution playbook directory). Every execution's exec-stdout.log now
ends with a per-task timing breakdown.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@arthurschreiber
arthurschreiber requested a review from frouioui as a code owner July 7, 2026 09:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants