Deliver build-host images by chunks, and resume after a drop

[chrisdok43]

Closes WDY-2605.

The gap

The build host built the image fine and then delivered it with a plain registry push: every layer whole, and one dropped connection lost the whole transfer. On Spark 3 (US) → ccr1 (Canada) that was four consecutive deploys dying at "exporting + pushing layers" with EOF at 56–162s, each retry starting from zero. #1771 retries the connection and says how far the push got, but as its own description says, a half-sent layer can't be replayed through a proxy — this is the fix it deferred.

Meanwhile wendy run from a laptop has chunk-diffed into the device's content store for months. Only this leg skipped it.

What changes

The build host does what the CLI does, from where it stands:

• Build once, export an OCI tar. Delivery is a separate step per device, fed from that export. A fleet deploy is now one build, not one buildctl pass per device.
• Deliver by chunks. Dial the target's agent over the mesh (same peer dialer and asset-pinned mTLS as the registry hop), QueryLayers to skip layers it already holds, QueryChunks/WriteChunks for only the missing bytes, PrepareImage to register the image under localhost:<port>/<repo> — the name the CLI's CreateContainer already uses. The CLI is unchanged and cannot tell how the image arrived.
• Resume. WriteChunks stages chunks as they land, so a transport drop re-dials and the next QueryChunks reports only what's still missing. Four attempts with backoff; cancellation and device refusals are not retried.
• Fallback. An agent that answers Unimplemented to QueryChunks or PrepareImage (pre 2026-08-12) gets the registry push it always did. Taken for that error only — a genuine failure is reported as one, not retried more slowly and blamed on the wrong leg.
• Progress renders live on the existing "exporting + pushing layers" row; resume and fallback are announced in the stream.

No proto change. BuildImageResult.image_digest (always empty before) now carries the manifest digest. The single-target error contract classifyRemoteBuildError keys on is preserved.

Design: specs/2026-08-26-build-host-chunk-delivery-design.md.

Relationship to #1771

Both touch build_service.go; expect a conflict on whichever lands second. #1771's connection retry and byte accounting still apply to the registry-push path, which this PR keeps as the fallback for old agents.

Test plan

• go test ./internal/agent/services -race — full suite green, 20 new tests: a fake target agent over bufconn (only missing chunks sent; present layers never chunked; image registered under the right name with config intact); resume after an Unavailable mid-stream with every chunk reaching the device exactly once; fallback routing on Unimplemented; a device refusal neither retried nor fallen back; BuildImage end to end with the test binary standing in for buildctl (one pass for two devices; second pass for an old agent; scratch files removed; error contract).
• go build ./..., gofmt -s, go vet clean.
• Hardware: see the verification section below.

Hardware verification (2026-08-26)

Agent side-loaded on Spark 3 (build host, US); ccr1 (Orin Nano, Canada) on release 2026.08.25-111847; all traffic via the cloud relay.

• Cold python-hello: 25.8s end to end. ccr1's agent journal shows the build host dialling localhost:50052, six Wrote layer to content store, Assembled image localhost:5000/…, and Prepared image before container start — the chunk path, not the registry. Spark 3 ran exactly one buildctl pass (--output type=oci,dest=…). No fallback.
• Warm re-run: 3.6s, zero layers written on ccr1.
• 300 MB incompressible layer, CLI killed at ~52%: 2,683 chunks (157 MB) stayed staged on ccr1; the build host removed its export tar and scratch layers and logged the abort as Canceled.
• Re-run after the kill: 100% 150.7MB/150.7MB (28.7s), 34s total — only the missing 48% was sent; the layer was written once, the image assembled, the container started, and staging drained to 0.

Not exercised on hardware: the in-run auto-resume after a transport drop (unit-tested). The interruption here was a client cancel, which exercises the cross-run resume case.

🤖 Generated with Claude Code

[github-actions]

AI Security Review

Note

Automated security review from Claude. Apply, adapt, silence with // SECURITY: <reason>, or dismiss as needed.

Input coverage: 9/9 changed files; 100,688/100,688 bytes reviewed; diff SHA-256 d1c313885c9b31b6ade33d430c8be5de8914788be8eeb84f44b143c88d813b0d; truncation: none.

No security findings.

[thombles]

@EBro912 When you pick this up, there's an earlier, more detailed, issue WDY-2564 that specs some more thorough requirements. It would be worth cross-checking to see if any of the ideas there should be worked into this PR too and then we can close off both.

Edit: Re-add me for review when you're ready!