Skip to content

fix: purge Burst cache after unclean Cloud Build + CI actions node24/SHA pinning#8966

Draft
aixaCode wants to merge 8 commits into
devfrom
fix/purge-burst-cache-cloud-build
Draft

fix: purge Burst cache after unclean Cloud Build + CI actions node24/SHA pinning#8966
aixaCode wants to merge 8 commits into
devfrom
fix/purge-burst-cache-cloud-build

Conversation

@aixaCode

@aixaCode aixaCode commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

Pull Request Description

What does this PR change?

Windows builds have been failing on and off (half of the runs on Jun 10) with a Burst error:

Error: Failed to find entry-points:
System.IO.EndOfStreamException: Unable to read beyond the end of the stream.

Root cause: when a build gets cancelled while it's uploading its Library cache, the Burst hash-cache files end up truncated, and every following build on that target trips over them. New PR targets can inherit the broken cache too, since it's cloned from the template target. A clean build fixes it, but only for that one target, and it gets poisoned again on the next cancellation. It's not a code issue — the same commit built fine and failed 24s apart on two runs.

The fix: before triggering a build, build.py checks the previous build's status on the target. If it was canceled/unknown (or the target is brand new), it tells PreExport to delete Library/BurstCache so Burst regenerates it. Healthy targets keep their warm cache (Burst AOT is ~4.5 min warm).

While I was in here I also cleaned up around the workflow:

  • moved the remaining third-party actions to node24 and pinned all external actions to commit SHAs (replaces chore: bump third-party actions to node24 runtime #8982)
  • build failures are readable now: the actual Unity error shows up as an annotation and in the step summary instead of being buried in a 20k-line log, which is only dumped on failure
  • cancelled runners still fetch and upload the partial Unity log
  • queue visibility: a notice when the build gets picked up, a warning if it sits queued for >30 min, and Unity-side cancellations now say that another build probably evicted them
  • Build Gate says when a red is just "superseded by a newer run" vs a real failure
  • skip statuses state the real reason (draft PR vs no Explorer/ changes)
  • removing the force-build/clean-build label no longer cancels a build that's already running
  • removed a leftover # Test change comment

Test Instructions

Steps (standard run): N/A — CI/build infrastructure only.

Expected result: N/A

Steps (fresh account): N/A

Expected result: N/A

Automation (if applicable): N/A

Prerequisites

  • force-build label on this PR (it's a draft)

Test Steps

  1. Cancel a build on this PR mid-flight, then re-trigger.
  2. The next build's logs should show purge_burst_cache=True (Prebuild) and [BURST]: purged restored Library/BurstCache (Unity log).
  3. A build after a green one should show purge_burst_cache=False and keep the cache, with bcl.exe time around the usual ~274s.
  4. No more node20 deprecation warnings in run annotations (except game-ci/unity-test-runner and sslcom/esigner-codesign — no node24 release upstream yet).

Additional Testing Notes

  • Comparing bcl.exe exited after N ms between purged and non-purged builds tells us the real cost of a cold Burst cache.
  • Corruption that happens without a cancellation (e.g. infra crash during cache upload) won't trigger the purge — the clean-build label is still the escape hatch for that.

Quality Checklist

  • Changes have been tested locally
  • Documentation has been updated (if required)
  • Performance impact has been considered
  • For SDK features: Test scene is included

@github-actions

github-actions Bot commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

@aixaCode aixaCode added the force-build Used to trigger a build on draft PR label Jun 11, 2026
@m3taphysics

This comment has been minimized.

@m3taphysics

This comment has been minimized.

@aixaCode aixaCode changed the title fix: purge restored Burst cache in Cloud Build PreExport fix: purge Burst cache after unclean Cloud Build + CI actions node24/SHA pinning Jun 12, 2026
@aixaCode aixaCode removed the force-build Used to trigger a build on draft PR label Jun 12, 2026
@aixaCode aixaCode added the force-build Used to trigger a build on draft PR label Jun 12, 2026
@m3taphysics

Copy link
Copy Markdown
Contributor

PR #8966, run #27435436524

Builds: Windows change, Windows baseline, macOS change, macOS baseline

How to read this table
  • Each build is measured 3 times. The values are the median, and (min–max) is the lowest and highest of those runs — a wide range means the metric is noisy and small differences are not trustworthy.
  • Δ is Change minus Baseline (a negative Δ means Change is faster).
  • 🟢 faster / 🔴 slower — a real difference: larger than both 3% and the run-to-run range.
  • ⚪ within noise — the difference is smaller than how much the build varies between its own runs, so it cannot be told apart from random variation. Treat it as no change.
  • Exceptions per run — the average number of exceptions in a run's log; more than the baseline is flagged 🔴 even when frame times look fine. The Exception breakdown under each table groups them by the explorer's report category and exception type (as totals across the runs).
  • A run that logged unusually many exceptions (at least 10 and 5× the median of its build's runs — e.g. a service was down during it) is excluded from all numbers and called out under the table.

Framework 13 i7

Metric Baseline Change Δ Result
Samples 2701 (×3) 2701 (×3)
CPU average 33.3 ms 33.3 ms -0.0 ms ⚪ within noise
CPU 1% worst 33.8 ms (33.7–34.0) 33.6 ms (33.5–33.7) -0.1 ms ⚪ within noise
CPU 0.1% worst 37.1 ms (36.0–37.2) 34.6 ms (33.7–35.5) -2.6 ms 🟢 7% faster
GPU average 7.5 ms (7.5–7.6) 7.7 ms (7.5–7.8) 0.1 ms ⚪ within noise
GPU 1% worst 19.7 ms (19.2–20.0) 19.5 ms (18.8–20.5) -0.2 ms ⚪ within noise
GPU 0.1% worst 24.8 ms (24.1–26.8) 28.5 ms (25.7–29.2) 3.7 ms 🔴 15% slower
Exceptions per run 2 2 0 ⚪ none new
Exception breakdown
Exception Baseline (3 runs) Change (3 runs)
[ENGINE] NullReferenceException 6 6

@charly-bg charly-bg added the clean-build Used to trigger clean build on PR label Jun 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

clean-build Used to trigger clean build on PR force-build Used to trigger a build on draft PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants