Skip to content

e2e docker_build_network hangs on BOTH backends (master regression): build stalls before context/egress #490

Description

@AprilNEA

Updated 2026-07-22: originally filed as an HV vsock upload stall — a VZ run and an origin/master control run reproduce the identical hang, so this is NOT HV-specific. The HV-specific docker save truncation evidence stays with #256**.**

Symptom

docker_build_network (8 MiB context upload + RUN sha256sum + RUN wget) times out at 300 s. The other 7 workload checks pass. Last known green: 2026-07-20 (#452, W13/W14).

Reproductions (all 2026-07-21/22, this host)

run backend code result
1 HV branch #489 build timeout; zero egress traffic the whole window (datapath counters frozen, no DNS/SYN)
2 VZ branch #489 identical timeout; splicetcp=debug shows no egress attempt → stalls before RUN wget
3 VZ origin/master (d46dbcd) identical timeout → not caused by #489

Guest agent at teardown: Docker proxy unix→vsock: write error after 327 bytes / vsock→unix: read error after 6047 bytes — single-digit KBs of a multi-MB context moved. The stall sits in the docker-API/buildkit zone (context upload / buildkit session), before any RUN executes.

Leads

  • docker CLI (host, ArcBox.app-bundled) and guest dockerd are both 29.6.1 — no version skew. dockerd jumped 27.5.1 → 29.6.1 at boot-assets 0.6.3; the 07-20 green runs were on 0.6.7/0.6.8 (already 29.6.1), so the bump alone doesn't explain it. Suite-level bisect candidates: the ~16 commits merged since 07-20, and the boot-assets 0.6.8 → 0.6.9 refresh.
  • docker build uses long-lived HTTP-upgraded buildkit /session + /grpc connections through the API proxy — the same upgrade-handling zone as docker run foreground hangs after container exit — HTTP upgrade attach stream close not propagated #268.
  • Noise observed during the window: guest pings gateway every ~4 s, ICMP proxy fails EINVAL (starts pre-build; separate wart). One HV run also had every docker command (load/pull) hang — unreproduced variant.

Repro

ARCBOX_E2E_WORKLOAD_ONLY=docker_build_network cargo test -p arcbox-e2e --test network_workload -- --ignored --nocapture   # VZ
ARCBOX_E2E_BACKEND=hv <same>                                                                                              # HV

Refs #268, #256. Blocks treating the workload suite as green on either backend.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions