qemu: harden hibernate/checkpoint against rootfs corruption

[motatoes]

Summary

Four interlocked guards against a class of failure where savevm captures a qcow2 in an inconsistent state and the rootfs becomes unbootable on next cold-mount (EXT4 inode #2 metadata-checksum failure → kernel panic loop). Each addresses a different ring of the failure chain; together they cover both the trigger (rootfs fills) and the killshot (savevm against unresponsive guest).

The four fixes

1. Hard-fail Hibernate / CreateCheckpoint when guest agent is unresponsive (load-bearing)

prepareAgentForHibernate previously fell back from PrepareHibernate RPC → Exec(sync; …; kill -USR1 1), and silently returned even when both timed out. Callers then proceeded to qmp.SaveVM() against a guest with un-synced page cache and pending EXT4 journal entries — exactly the case that produces an unbootable qcow2.

Now both prepareAgentForHibernate and quiesceAndCloseAgent return error. doHibernate and CreateCheckpoint propagate ErrAgentUnresponsive and refuse to run savevm. The API returns a clean error to the caller; the rootfs stays intact.

internal/qemu/manager.go, internal/qemu/snapshot.go

2. Explicit qmp.Stop() before SaveVM, Cont() after

savevm internally pauses and resumes, but the explicit Stop closes a small race where in-flight virtio-blk writes can land in the qcow2 between the agent's sync and the start of savevm. Standard QEMU quiesce pattern. Cont() is unconditional in CreateCheckpoint (sandbox keeps running); deliberately omitted in doHibernate since it Quits next.

internal/qemu/snapshot.go, internal/qemu/manager.go

3. Bind-mount /var/cache/apt/archives onto workspace

apt commonly stages 1-3 GB of .deb files in /var/cache/apt/archives during installs. On a 4 GiB rootfs, this is a substantial chunk that's easy to redirect to the 20 GiB workspace. Helper setupAptCacheBindMount is invoked from:

• golden-create (inline with the workspace mount block)
• wake (after reinstallProxyCA)
• migration (post-resume hooks)

Idempotent via mountpoint -q short-circuit. Lives in the kernel mount table only; does not modify guest /etc/fstab; re-applied on every resume. Failure is non-fatal (apt cache stays on rootfs, status quo).

internal/qemu/manager.go, internal/qemu/snapshot.go, internal/qemu/migration.go

4. Disk-pressure telemetry + refusal

Agent's Stats RPC now reports statvfs("/") and statvfs("/home/sandbox") in four new wire-compat fields (added with new field numbers per the proto stability contract; older agents return zero, treated as "unknown" → no gate, status quo).

Worker preflights destructive operations:

• >=85% rootfs: log warning so it surfaces in the journal/telemetry.
• >=95% rootfs: refuse with ErrRootfsCritical, including the actual percent and a hint to free space or kill+respawn from a checkpoint.

The check is best-effort: agent unreachable / older agent / Stats RPC failure all fall through to the pre-existing behavior, so this is backward compatible for long-lived sandboxes whose in-guest agent predates this PR.

proto/agent/agent.proto, proto/agent/agent.pb.go, proto/agent/agent_grpc.pb.go, internal/agent/stats.go, internal/qemu/manager.go

How they compose

heavy apt activity
    │
    ├─→ rootfs hits 85% ─────→ #4 logs warning on next destructive op
    ├─→ rootfs hits 95% ─────→ #4 refuses Hibernate/CreateCheckpoint
    ├─→ /var/cache/apt/archives → workspace via #3, the 85/95 lines
    │     are now further away in practice
    └─→ if agent still wedges from disk pressure:
        Hibernate arrives
            └─→ #1 returns ErrAgentUnresponsive BEFORE savevm
                rootfs.qcow2 is untouched, no corruption
                customer sees clear error, kills + respawns

#1 alone is the hard interlock that prevents corruption even if all others fail. #2-#4 reduce how often the trigger fires.

Test plan

• go build ./... passes for all touched packages.
• go vet ./internal/qemu/... ./internal/agent/... clean.
• Existing qemu unit tests pass (one pre-existing test skipped on macOS — needs qemu-img on PATH).
• Verify on dev: simulate a sandbox with disk-full rootfs → confirm oc sandbox hibernate returns ErrRootfsCritical instead of triggering savevm.
• Verify on dev: simulate a sandbox with hung agent (e.g., kill the agent process inside the guest) → confirm oc sandbox hibernate returns ErrAgentUnresponsive, rootfs.qcow2 mtime unchanged.
• Verify on dev: spawn a fresh sandbox, run mountpoint /var/cache/apt/archives in the guest → confirm bind-mount is in place; df -h /var/cache/apt/archives shows the workspace disk.
• Verify on dev: hibernate + wake a sandbox → confirm bind-mount is re-applied on the destination and mountpoint -q short-circuits subsequent runs.
• Run a full apt-heavy build workload (the kind that previously consumed 3+ GB on rootfs) → confirm rootfs use% stays well under 85% with the apt-cache redirect in place.

Notes

• The proto change is wire-compatible per the stability contract in proto/agent/agent.proto: adds new fields with new field numbers; existing field numbers are untouched.
• setupAptCacheBindMount lives in manager.go next to reinstallProxyCA because they share a category (post-resume guest setup hooks).
• I considered baking the bind-mount into the platform default.ext4 /etc/fstab instead. Worker-side injection ships faster, applies to all existing sandboxes on next wake, and is per-sandbox-gateable. Trade-off: doesn't survive an in-guest cold-reboot until next wake (rare in practice).

🤖 Generated with Claude Code