Skip to content

feat(runner,api): run v0 snapshot-from-sandbox asynchronously like backups#5004

Draft
mu-hashmi wants to merge 5 commits into
feat/snapshot-record-at-capture-startfrom
feat/v0-async-snapshot-capture
Draft

feat(runner,api): run v0 snapshot-from-sandbox asynchronously like backups#5004
mu-hashmi wants to merge 5 commits into
feat/snapshot-record-at-capture-startfrom
feat/v0-async-snapshot-capture

Conversation

@mu-hashmi

@mu-hashmi mu-hashmi commented Jun 11, 2026

Copy link
Copy Markdown
Collaborator

TL;DR: v0 snapshot capture becomes asynchronous (202 + short status polls) behind an opt-in async flag, so capture duration is no longer silently bounded by the API's 15-minute HTTP client timeout - the mechanism behind the production >15-min silent capture deaths.

Stacked PR: base is feat/snapshot-record-at-capture-start (#4999), not main - the diff shows only this PR's commits. Merge #4999 first, then retarget this to main.

Description

v0 capture was the only long v0 operation holding one HTTP request open through docker commit+push. The API's axios instance timeout (15 min) aborted the request AND the runner's work mid-flight (request-scoped context), then the failure vanished (#4990). Two more production captures died exactly this way on 2026-06-11.

Runner

  • POST .../snapshot-from-sandbox with async: true: validates, records IN_PROGRESS in a new SnapshotFromSandboxInfoCache (BackupInfoCache shape, same retention env), runs the existing capture in a goroutine on a detached context (existing BACKUP_TIMEOUT_MIN budget), returns 202. New GET .../snapshot-from-sandbox status endpoint backed by the cache (NONE/IN_PROGRESS/COMPLETED/FAILED).
  • Supersede semantics: starting a capture cancels any in-flight capture for the sandbox. Ownership is identity-guarded end-to-end: a superseded goroutine removes only its own registration, writes nothing to the info cache, and skips its deferred unpause (pause responsibility transfers to the successor that found the container paused). context.Canceled maps to FAILED('canceled/superseded') rather than backup's NONE so pollers get a definite terminal state (commented in-code).

API

  • runnerAdapter.v0.createSnapshotFromSandbox sends async: true. A 200 with name+hash is the old-runner sync fallback (deploy-window compat; no endpoint versioning). A 202 enters a 5s poll loop budgeted by SANDBOX_SNAPSHOTTING_TIMEOUT_MIN; each poll uses a short per-request timeout clamped to the remaining budget (max(1s, min(30s, remaining))), 4xx fails fast with the runner's error payload (a 404 reason notes a possible runner restart/downgrade), 5xx/network/timeouts retry until budget. Terminal outcome activates or fails the feat(api): create the snapshot record at capture start and mark it error on failure #4999 record with the runner's real ref/size/error.
  • RETRYABLE_NETWORK_ERROR_CODES deliberately untouched: re-POSTing a full commit+push on ECONNABORTED would storm the runner; async removes the motivation.

Known limits (in-code/documented): info cache is in-memory - runner restart mid-capture fails fast with an explicit reason (#4999's cron remains the backstop); a capture succeeding after the API gave up orphans a registry image (pre-existing class); a residual supersede TOCTOU around the unpause claim is documented - same class as the existing backup path, full fix is a per-sandbox mutex, deferred.

Review guide

  1. apps/runner/pkg/docker/snapshot_sandbox.go - THE risk center: goroutine lifecycle, ownership guards, cache state machine.
  2. apps/api/src/sandbox/runner-adapter/runnerAdapter.v0.ts - poll loop, clamp, 4xx fail-fast, sync fallback.
  3. apps/runner/pkg/api/controllers/sandbox.go - 202/status endpoints.
    Mechanical: runner swagger + libs/runner-api-client regen (isolated commit, additive-only).

Commit map

Commit What Why it exists
ab7906d runner async capture behind opt-in flag (supersede guards folded pre-publication after internal adversarial review; race-clean x5) implementation (plan)
e0bc29e runner-api-client regen implementation (generated)
a2391bc API poller + record integration implementation (plan)
399e05f short per-poll timeout + 4xx fail-fast cubic review (2 valid)
116e3e3 clamp poll timeout to remaining budget cubic review (valid)

Related PRs

Part of #4990; consumes #4999's record lifecycle. #5000 pairs on the SDK side. This is the cure for the >15-min capture failures; #4999/#5000 are the visibility half. Relief in production requires BOTH the API and the runner fleet deployed - new API + old runner degrades gracefully to the sync fallback, which still dies at 15 min.

Validation

  • Runner: 7 Go tests in a linux container - async lifecycle (IN_PROGRESS write-before-202, COMPLETED with canonical ref/hash/sizeGB, commit-failure error preservation, sync-path regression) and the supersede suite (successor's cache entry survives predecessor termination; unpause-during-commit tripwire); race-clean over 5 consecutive -race runs. GOOS=linux go build/vet, gofmt, golangci-lint v2.6.2 clean.
  • API: 7 jest poller specs (sync fallback without polling, COMPLETED mapping, FAILED surfaces runner error, NONE fail-fast, name-mismatch guard, budget exhaustion, near-deadline clamp); nx build api green.
  • Client regen verified additive-only. All CI checks green on 116e3e3.

Closes #4993

…async flag

Signed-off-by: Muhammad Hashmi <mhashmi@berkeley.edu>
…andbox

Signed-off-by: Muhammad Hashmi <mhashmi@berkeley.edu>
Signed-off-by: Muhammad Hashmi <mhashmi@berkeley.edu>
@nx-cloud

nx-cloud Bot commented Jun 11, 2026

Copy link
Copy Markdown

View your CI Pipeline Execution ↗ for commit 399e05f

Command Status Duration Result
nx run-many --target=docker --configuration=pr-... ✅ Succeeded 4m 3s View ↗

💡 Verify your cache is correct by running tasks in a sandbox. Read docs ↗


☁️ Nx Cloud last updated this comment at 2026-06-11 04:28:58 UTC

@mu-hashmi mu-hashmi marked this pull request as ready for review June 11, 2026 03:32

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2 issues found across 22 files

Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic

Comment thread apps/api/src/sandbox/runner-adapter/runnerAdapter.v0.ts
Comment thread apps/api/src/sandbox/runner-adapter/runnerAdapter.v0.ts Outdated
…nner errors

Signed-off-by: Muhammad Hashmi <mhashmi@berkeley.edu>

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 2 files (changes from recent commits).

Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic

Comment thread apps/api/src/sandbox/runner-adapter/runnerAdapter.v0.ts Outdated
Signed-off-by: Muhammad Hashmi <mhashmi@berkeley.edu>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant