workspace: Recover interrupted container sync - #18
Merged
Conversation
commit: |
Member
|
/bonk review this (edit: i tried) |
Contributor
Author
|
@agent-think review this |
Keep blob metadata and bytes in one transaction so a receiver storage failure leaves the pull cursor pending. Cover retry behavior and repository-shaped generated output through the real wsd harness.
Treat a blob as present only when its payload exists and matches the recorded size. Pull retries now fetch and restage incomplete or corrupt local objects before applying file entries and advancing the cursor.
Return command output and exit status when the post-command pull fails, while marking the structured sync outcome as pending. Bound and redact the reported error before exposing it through results and observer spans.
Let Durable Object hosts persist one pending pull intent per backend and wake Workspace through retryPendingSync. Failed attempts use bounded exponential backoff while exhausted intents remain available for inspection.\n\nCover scheduling, coalescing, cursor resume, convergence, exhaustion, and RPC cleanup with an in-memory scheduler.
aron-cf
force-pushed
the
fix/sync-recovery-stack
branch
from
July 29, 2026 23:42
98b74eb to
e6ed4ee
Compare
stageBlob now repairs corrupt/incomplete rows via ON CONFLICT DO UPDATE SET bytes and clears the blob cache; the normal write path (upsertChunkBlob) still uses DO NOTHING. Update the comment so it no longer claims (hash, bytes) pairs are unconditionally immutable.
retryPendingSync leaves the final intent in storage with a past-due notBefore when a backend exhausts its attempts. The example alarm re-armed on min(notBefore), so an exhausted intent made the alarm fire immediately and repeatedly. Clear exhausted intents in the example.
Construct the WorkspaceExecHandleStub test with a fake ExecHandle so it matches the eagerly-spawned handle implementation after the rebase. This keeps the test focused on forwarding ExecResult.sync through the Workers RPC stub.
aron-cf
force-pushed
the
fix/sync-recovery-stack
branch
from
July 30, 2026 08:18
e6ed4ee to
92c4478
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A post-command Workspace pull can be interrupted while staging file bytes into Durable Object SQLite. The previous implementation wrote blob metadata and bytes separately, treated metadata-only rows as complete, and represented a failed pull as
pulled: 0. A reset could therefore leave an unrecoverable partial blob while callers could not distinguish pending synchronization from a clean no-op.Make blob staging atomic, require matching byte rows for presence checks, and repair incomplete or size-mismatched blobs during retry. Fetch cursors advance only after a batch applies. Command results now include a backwards-compatible structured sync status, preserving exit output while reporting a pending pull with a bounded, redacted error.
Add a host-provided durable retry scheduler and
Workspace.retryPendingSync(). A Durable Object host can persist one retry intent per backend, invoke retries from an alarm, resume from the stored(rev, path)cursor, apply bounded backoff, and expose exhaustion without replaying the command.The full build, typecheck, test, and check suites pass. Coverage includes 433 dofs tests, 62 RPC tests, 728 Workspace tests plus backend/proxy/soak suites, 137 wsd tests, and the real Docker wsd harness.
After merge, publish a Workspace prerelease and matching
workspace-wsd-linux-x64image. The dependent agent-think PR must update its exact package and image pins before deployment.