Introduce explicit snapshot types for pause/resume, local forks, and distributed durability

[yingdi-shan]

Summary

AgentENV currently uses "snapshot" for workflows with materially different lifecycle, availability, and durability requirements. Introduce four explicit snapshot types so each workflow pays only the storage and distribution cost it needs:

Type	New immutable OverlayBD layer	Availability	Durability	Primary use case
Temporal	No	Owning sandbox	Ephemeral	Pause/resume
Local	Yes	Current node	Node-local	Fast sandbox fork
Distributed	Yes	Cluster-wide	P2P or external storage	Cluster-wide sandbox creation

Motivation

The current pause path seals the active read/write state and creates new disk and memory layers. Repeated pause/resume cycles can therefore produce long layer chains, increase metadata overhead, and require garbage collection of many short-lived layers.

Pause/resume does not normally require an immutable, reusable snapshot. It only needs enough state to continue the same sandbox. Forking, cluster-wide creation, and durable recovery have progressively stronger requirements and should use explicit snapshot types with matching costs.

Proposed snapshot types

1. Temporal snapshot

A temporal snapshot is an ephemeral continuation point for pausing and resuming the same sandbox.

• Do not seal the current read/write disk or memory state into new immutable OverlayBD layers.
• On resume, continue writing directly into the existing read/write layers.
• Do not grow the immutable layer chain across repeated pause/resume cycles.
• Scope the snapshot to its owning sandbox; it cannot be used for fork or new sandbox creation.
• Treat it as single-use: successful resume consumes or invalidates the snapshot before new writes are accepted.
• Node-local metadata may survive an AgentENV process restart, but the snapshot is not protected against node or disk loss.

The single-use rule is required because resumed writes mutate the same layers represented by the continuation point.

2. Local snapshot

A local snapshot is an immutable snapshot materialized on the current node.

• Seal disk and memory state into reusable OverlayBD layers.
• Allow multiple sandboxes to fork or start from it on the same node.
• Do not require upload to external storage.
• Keep availability limited to the node holding the artifacts.
• Allow promotion to a distributed or distributed persisted snapshot without recapturing the source sandbox.

3. Distributed snapshot

A distributed snapshot is resolvable across the cluster.

• Use immutable disk and memory layers.
• Publish the metadata and artifact availability needed for another node to start a sandbox.
• When P2P is enabled, allow required artifacts to remain P2P-backed without first uploading all of them to external storage.
• Availability may depend on at least one artifact-holding peer remaining online.
• Report a clear resolution failure when neither a peer nor external storage can supply a required artifact.
• Allow to be persisted synchronously or asynchronously.

Lifecycle and promotion

Supported paths:

• Temporal snapshot -> resume the owning sandbox -> consume the snapshot
• Temporal snapshot -> Local Snapshot -> Distributed Snapshot

Promotion should reuse existing immutable artifacts rather than recapturing the running sandbox.

API direction

Make snapshot intent explicit instead of inferring it only from repository configuration. For example:

snapshotType: temporal | local | distributed

Suggested defaults:

• Pause uses a temporal snapshot.
• Same-node fork uses a local snapshot internally.
• User-created snapshots select local or distributed semantics explicitly, subject to compatibility requirements.