Bazel CI build is slow: investigate cache miss patterns

[qobilidop]

Problem

The Bazel CI workflow (.github/workflows/bazel.yml) recompiles large transitive dependencies (notably protobuf) on every run despite a two-layer cache: GitHub Actions disk cache via actions/cache@v5 and BuildBuddy remote cache. The recompilation is consistent across runs on main and appears to be much worse on at least some PR runs.

The repository depends on a single-digit number of direct Bazel modules (fuzztest, googletest, rules_cc, rules_shell, z3); the protobuf rebuild comes in transitively, almost certainly through fuzztest. Our own source tree is small, so most of the wall-clock time on every CI run is spent on dependency code that hasn't changed.

Evidence

Main runs are consistently ~2 minutes but show high cache miss rates

Most recent successful Bazel CI runs on main (from gh run list --workflow=bazel.yml --branch main):

Run	Date	Duration
Add exact-match table symbolic execution example (#25)	2026-05-07	1m56s
docs: restate naming rule positively	2026-05-07	1m42s
Update actions/upload-pages-artifact action to v5 (#18)	2026-05-07	1m45s
Update actions/deploy-pages action to v5 (#17)	2026-05-07	1m58s
Update dependency rules_shell to v0.8.0 (#16)	2026-05-07	1m52s

Inspecting the most recent (run id 25479661712) shows the disk cache does restore successfully, but the Bazel summary lines are:

INFO: 2432 processes: 248 disk cache hit, 913 remote cache hit, 1271 internal.
INFO: 31 processes:   510 action cache hit,  18 disk cache hit, 32 remote cache hit.

So of the 2432 build-phase actions, only 1161 (~48%) come from cache. The other 1271 run fresh. Searching that run's log for external/protobuf+/src/google/protobuf/ returns hundreds of lines of compile warnings, confirming protobuf is recompiled despite the partial cache hits.

Our PR sees a much worse cache hit rate

In PR #27, the Bazel CI build job is currently running at ~23 minutes elapsed and counting (started 2026-05-11T00:52:19Z, still pending at time of issue creation). Compared to main's ~2-minute baseline, that's roughly a 10x slowdown.

Our PR does not modify any of the cache key inputs (.bazelversion, MODULE.bazel, MODULE.bazel.lock, .bazelrc):

$ git diff 87ccdc99..HEAD --stat -- .bazelversion MODULE.bazel.lock .bazelrc MODULE.bazel
(empty - no changes)

So the disk cache key (bazel-Linux-${{ hashFiles('.bazelversion', 'MODULE.bazel.lock', '.bazelrc') }}) is identical to main's. The restore-keys: bazel-Linux- fallback should also match. Despite that, the build is dramatically slower than main's, suggesting the cache lookup is missing far more often than on main.

Cache mechanics

Workflow excerpt (.github/workflows/bazel.yml:14-26):

- name: Restore Bazel disk cache
  uses: actions/cache@v5
  with:
    path: .bazel-disk-cache
    key: bazel-${{ runner.os }}-${{ hashFiles('.bazelversion', 'MODULE.bazel.lock', '.bazelrc') }}
    restore-keys: |
      bazel-${{ runner.os }}-

- name: Activate CI-specific Bazel config
  run: |
    echo "build --config=ci" > ci.bazelrc
    echo "build:ci --remote_header=x-buildbuddy-api-key=${{ secrets.BUILDBUDDY_API_KEY }}" >> ci.bazelrc

Two cache layers exist (disk + BuildBuddy remote). Both restored successfully on the May 7 run, yet half the actions ran fresh. That implies action hashes differ run-to-run for ~half the build graph, OR cache content is missing under the right hash for that half.

Proposals

Three angles worth investigating, ordered by leverage:

1. Root-cause the cache miss pattern (highest leverage)

BuildBuddy's UI has a "Compare invocations" view that shows, action-by-action, exactly which inputs differ between two runs and why. The CI is already wired to BuildBuddy (--remote_header=x-buildbuddy-api-key=...), so the data is there. Pick two recent main runs and diff them; the typical culprits are:

• Absolute paths leaking into compile outputs (-MD / .d files, debug info).
• Compiler / sysroot / libc fingerprint drift across runner image revisions.
• Volatile environment variables captured via --action_env.
• Platform/exec_properties differences.

Fix is usually a small .bazelrc change (pin compiler features, scrub action_env, set hermetic toolchain options). Investigation time: probably 30-90 minutes if you have BuildBuddy access. Payoff: potentially fixes the whole graph at once.

2. Stop building fuzz tests by default (cleanest sidestep)

protobuf is transitive through fuzztest, which is only needed by tests/fuzz/* targets. If those targets are tagged manual in tests/fuzz/BUILD.bazel, they drop out of bazel build //... expansion. Bazel only executes actions needed by the requested target set, so protobuf compile actions never run — only the source fetch happens during analysis, which is seconds rather than minutes.

# tests/fuzz/BUILD.bazel
cc_test(
    name = \"sym_bit_vec_cast_test\",
    srcs = [\"sym_bit_vec_cast_test.cc\"],
    tags = [\"manual\"],   # excluded from //...
    deps = [...],
)

To preserve fuzz coverage, add a dedicated CI step (or a separate workflow) that runs bazel test //tests/fuzz/... explicitly — that one job still pays the cost, but only when fuzz code or its deps change, or on a schedule. Implementation cost: ~30 minutes (one BUILD edit per fuzz target, one CI step). Risk: low; the fuzz tests are already isolated under tests/fuzz/.

3. Pre-warm the cache nightly (backstop)

Add a schedule: trigger to the Bazel workflow so a main build runs nightly. The existing cache config already saves with a key that PR runs restore from, so a nightly success leaves a warm cache for the next day's PRs.

on:
  push: { branches: [main] }
  pull_request: { branches: [main] }
  schedule:
    - cron: '0 5 * * *'   # 05:00 UTC daily

Implementation cost: ~15 minutes plus a day to verify. Important caveat: this only helps if the cache misses are about content absence (the right hash was never saved). If misses are about action hash drift (option 1's territory), pre-warming saves under hash A and the PR run looks up hash B and still misses. The 10% disk-cache hit rate today says at least some hashes are stable, so pre-warming would help that fraction — but to know how much, option 1's diagnosis is the prerequisite.

Analyses already done

1. Identified the workflow and cache config at .github/workflows/bazel.yml. Two cache layers (disk + BuildBuddy remote), key derived from hashFiles('.bazelversion', 'MODULE.bazel.lock', '.bazelrc') with a fallback restore-keys: bazel-Linux-.

2. Verified PR Strict comparison operators + math_* family #27 does not touch cache-key inputs. Cache key should match main; restore-keys fallback should match regardless. Yet the PR's Bazel job is currently at ~23 minutes and counting vs. main's ~2 minute baseline.

3. Confirmed protobuf rebuild is pre-existing on main. The May 7 run of main (gh run view 25479661712 --log) emits hundreds of external/protobuf+/src/google/protobuf/... compile warnings. The PR is not introducing the rebuild — it's just much worse at exhibiting the underlying cache miss problem.

4. Confirmed dep path. None of MODULE.bazel's direct deps (fuzztest, googletest, rules_cc, rules_shell, z3) are protobuf. Protobuf is transitive; fuzztest is the most likely path (it uses protobuf for test-input serialization).

5. Cache stats from a representative main run (run 25479661712, build summary):

   • Build phase: 2432 processes — 248 disk cache hit + 913 remote cache hit + 1271 internal = ~48% cached.
   • Test phase: 31 processes — 510 action cache hit + 18 disk cache hit + 32 remote cache hit.
   • Disk cache did restore ("Cache restored from key: bazel-Linux-deb7395ec..."). The cache is working — just not as well as we'd want.

What would close this issue

• A measurable improvement in Bazel CI wall-clock time on main, ideally back under 1 minute for incremental changes.
• Diagnosis of why roughly half of Bazel actions miss cache on main today (and why PRs sometimes miss far more than that).
• A documented fix or a documented tradeoff if the root cause is hard to address.

Useful tools

• BuildBuddy "Compare invocations" view (link to your BuildBuddy org from a recent invocation URL in the CI logs).
• `bazel build //... --execution_log_json_file=exec.json` for local-vs-CI action diff if BuildBuddy isn't sufficient.
• `bazel cquery 'deps(//tests/fuzz/...)' --output=label` to confirm the protobuf dep path before option 2.

🤖 Generated with Claude Code

[qobilidop]

Updated analysis after PRs #27 and #29

Two PRs merged since this issue was filed give cleaner data. The picture has shifted, but not in the direction my first draft of this comment claimed — see the correction at the end.

Two PR data points, very different outcomes

PR	Touches	Bazel CI wall time (PR run)
#27 — strict comparison operators	z3wire/sym_bit_vec.h + tests + examples + docs	~43m48s (run 25644651556)
#29 — packet_view example	examples/packet_view.cc + 3 build/doc files	~1m49s (run 25652067045)

PR #29 was faster than main's baseline despite being a fresh never-before-built branch. PR #27 was ~22x slower. Whatever's going on, the input that triggers it correlates with what content the PR changed, not with PR-vs-main mechanics in general.

Main is fully cached

Cache stats from the two most recent main runs are essentially identical:

# post-#27 (run 25645863174, 2m38s)
INFO: 2448 processes: 245 disk cache hit, 916 remote cache hit, 1287 internal.

# post-#29 (run 25652225593, 1m56s)
INFO: 2454 processes: 245 disk cache hit, 918 remote cache hit, 1291 internal.

All actions are either cache hits (~1160 ≈ 47%) or internal. There are no local or processwrapper-sandbox entries on main runs, which is where fresh compile work would show up. In Bazel summary output, internal means the action ran without spawning a subprocess (symlinks, header scans, action-fs operations) — not a cache miss.

Spot-checking the build log confirms this: every Compiling ... line in main runs is prefixed with 0s disk-cache, remote-cache, indicating the result came from cache. The INFO: From Compiling src/google/protobuf/... warnings I cited in the original issue are Bazel replaying captured stderr from cached actions, not fresh compilation. I misread that as evidence protobuf was being rebuilt on main — it wasn't.

So on main, the ~2-minute wall time is fixed overhead (deps fetch, analysis, action graph construction, cache lookups, linking), not compilation.

But PRs DO rebuild dependencies — correction to my first draft

The PR #27 build summary tells a different story than main's:

INFO: 2448 processes: 245 disk cache hit, 667 remote cache hit, 1287 internal, 249 processwrapper-sandbox.

That 249 processwrapper-sandbox at the end is fresh local execution. Sampling the log for what those actions were:

• Compiling absl/strings/cord.cc
• Compiling absl/container/internal/raw_hash_set.cc
• Compiling src/google/protobuf/compiler/cpp/helpers.cc [for tool]
• Compiling src/google/protobuf/compiler/command_line_interface.cc [for tool]
• Compiling src/google/protobuf/reflection_ops.cc [for tool]
• ...and ~240 more like these

These are external dep actions (abseil, protobuf-for-tool), not Z3Wire's own code. The original issue's intuition that "the slowness comes from rebuilding Z3Wire's dependencies, not Z3Wire itself" was correct, and my first draft of this comment dismissed it wrongly.

The deltas line up: PR #27 ran 249 fresh + 667 remote-cached = 916 compile actions; main post-merge ran 0 fresh + 916 remote-cached = 916. PR #27 wrote 249 entries to BuildBuddy; main post-merge then reused them. So those 249 actions had no BuildBuddy entries at the moment PR #27 ran — even though previous main runs (e.g., May 7, run 25479661712) had 0 fresh actions, meaning they hit cache for the equivalent set.

Open question: why did those 249 entries miss on PR #27?

Things ruled out:

• Runner image drift. Both May 7 main and May 11 PR Strict comparison operators + math_* family #27 used ubuntu-24.04 image 20260413.86.1.
• Cache key mismatch. Both runs restored disk cache from key bazel-Linux-96d48c9ff963f1e056e03e6929639ab1dac12680fb1d5f683c8bf6e17e6222ea.
• BuildBuddy auth failure on PRs. 667 remote cache hits on PR Strict comparison operators + math_* family #27 prove BuildBuddy reads are working.

Plausible remaining hypotheses:

1. BuildBuddy retention / eviction. Free-tier retention may have aged out entries between May 7 (last main run) and May 11 (PR Strict comparison operators + math_* family #27). PR Strict comparison operators + math_* family #27 happened to be the first run to repopulate them; main post-merge then benefited.
2. Non-hermetic input causing action-hash drift. Something in the action key varies between runs in a way git diff doesn't show. The dev container is built per CI run via devcontainers/ci@v0.3 with cacheFrom: ghcr.io/qobilidop/z3wire-dev. If the container layer hashes drift, the toolchain binaries inside it have different content hashes, and Bazel actions that depend on the toolchain (which is most C++ compile actions) get new action hashes.

(1) and (2) have very different fixes. (1) wants pre-warming or a paid BuildBuddy tier. (2) wants a more hermetic toolchain pinning (or sidestepping the dev container for the bazel step).

To distinguish (1) from (2)

BuildBuddy's "Compare invocations" view will tell us. Pick the PR #27 invocation URL (from --bes_results_url) and a recent main invocation, and compare action-by-action:

• If the same logical action shows different action hashes between the two, it's (2): there's a non-hermetic input.
• If the action hashes are identical but PR Strict comparison operators + math_* family #27 says "cache miss" and main says "cache hit", it's (1): BuildBuddy doesn't have the entry.

Without that diff, this is the most likely follow-up investigation — much higher signal than the original "diff two main runs" framing in the issue body.

Implications for the three original proposals

• Proposal 1 (BuildBuddy invocation diff): Still the highest-leverage investigation, but the right comparison is PR-run vs main-run rather than main-vs-main. Apply it to distinguish (1) vs (2) above.
• Proposal 2 (manual fuzz tests): Independently valuable. Removing protobuf from the default action graph would shrink the surface of dep actions that can suffer from (1) or (2). Even if every action hash held perfectly, fewer actions = less stuff to ever recompile.
• Proposal 3 (pre-warm cache nightly): Worth a try if the answer turns out to be (1) eviction. Useless if the answer is (2) hash drift. The right order is: diagnose first, then decide.

Correction to my earlier framing

In an earlier version of this comment I claimed "PR cold-start cost comes from a cascade through sym_bit_vec.h" and suggested splitting that header. That hypothesis would predict the fresh actions on PR #27 are Z3Wire's own .o files. The actual fresh actions are external deps. Header splitting wouldn't help with this problem — it's a cache/hash-stability problem at the dep level, not a Z3Wire-header-cascade problem.

🤖 Generated with Claude Code