feat(runner,api): run v0 snapshot-from-sandbox asynchronously like backups#5004
Draft
mu-hashmi wants to merge 5 commits into
Draft
feat(runner,api): run v0 snapshot-from-sandbox asynchronously like backups#5004mu-hashmi wants to merge 5 commits into
mu-hashmi wants to merge 5 commits into
Conversation
…async flag Signed-off-by: Muhammad Hashmi <mhashmi@berkeley.edu>
…andbox Signed-off-by: Muhammad Hashmi <mhashmi@berkeley.edu>
Signed-off-by: Muhammad Hashmi <mhashmi@berkeley.edu>
|
View your CI Pipeline Execution ↗ for commit 399e05f
💡 Verify your cache is correct by running tasks in a sandbox. Read docs ↗ ☁️ Nx Cloud last updated this comment at |
There was a problem hiding this comment.
2 issues found across 22 files
Reply with feedback, questions, or to request a fix.
Fix all with cubic | Re-trigger cubic
…nner errors Signed-off-by: Muhammad Hashmi <mhashmi@berkeley.edu>
There was a problem hiding this comment.
1 issue found across 2 files (changes from recent commits).
Reply with feedback, questions, or to request a fix.
Fix all with cubic | Re-trigger cubic
Signed-off-by: Muhammad Hashmi <mhashmi@berkeley.edu>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TL;DR: v0 snapshot capture becomes asynchronous (202 + short status polls) behind an opt-in
asyncflag, so capture duration is no longer silently bounded by the API's 15-minute HTTP client timeout - the mechanism behind the production >15-min silent capture deaths.Description
v0 capture was the only long v0 operation holding one HTTP request open through docker commit+push. The API's axios instance timeout (15 min) aborted the request AND the runner's work mid-flight (request-scoped context), then the failure vanished (#4990). Two more production captures died exactly this way on 2026-06-11.
Runner
POST .../snapshot-from-sandboxwithasync: true: validates, records IN_PROGRESS in a newSnapshotFromSandboxInfoCache(BackupInfoCache shape, same retention env), runs the existing capture in a goroutine on a detached context (existingBACKUP_TIMEOUT_MINbudget), returns 202. NewGET .../snapshot-from-sandboxstatus endpoint backed by the cache (NONE/IN_PROGRESS/COMPLETED/FAILED).context.Canceledmaps to FAILED('canceled/superseded') rather than backup's NONE so pollers get a definite terminal state (commented in-code).API
runnerAdapter.v0.createSnapshotFromSandboxsendsasync: true. A 200 with name+hash is the old-runner sync fallback (deploy-window compat; no endpoint versioning). A 202 enters a 5s poll loop budgeted bySANDBOX_SNAPSHOTTING_TIMEOUT_MIN; each poll uses a short per-request timeout clamped to the remaining budget (max(1s, min(30s, remaining))), 4xx fails fast with the runner's error payload (a 404 reason notes a possible runner restart/downgrade), 5xx/network/timeouts retry until budget. Terminal outcome activates or fails the feat(api): create the snapshot record at capture start and mark it error on failure #4999 record with the runner's real ref/size/error.RETRYABLE_NETWORK_ERROR_CODESdeliberately untouched: re-POSTing a full commit+push on ECONNABORTED would storm the runner; async removes the motivation.Known limits (in-code/documented): info cache is in-memory - runner restart mid-capture fails fast with an explicit reason (#4999's cron remains the backstop); a capture succeeding after the API gave up orphans a registry image (pre-existing class); a residual supersede TOCTOU around the unpause claim is documented - same class as the existing backup path, full fix is a per-sandbox mutex, deferred.
Review guide
apps/runner/pkg/docker/snapshot_sandbox.go- THE risk center: goroutine lifecycle, ownership guards, cache state machine.apps/api/src/sandbox/runner-adapter/runnerAdapter.v0.ts- poll loop, clamp, 4xx fail-fast, sync fallback.apps/runner/pkg/api/controllers/sandbox.go- 202/status endpoints.Mechanical: runner swagger +
libs/runner-api-clientregen (isolated commit, additive-only).Commit map
Related PRs
Part of #4990; consumes #4999's record lifecycle. #5000 pairs on the SDK side. This is the cure for the >15-min capture failures; #4999/#5000 are the visibility half. Relief in production requires BOTH the API and the runner fleet deployed - new API + old runner degrades gracefully to the sync fallback, which still dies at 15 min.
Validation
-raceruns.GOOS=linux go build/vet, gofmt, golangci-lint v2.6.2 clean.nx build apigreen.Closes #4993