Cold rootfs metadata IO dominates first-command latency on freshly-booted sandboxes — expose a warm-up path

[dushulin]

Summary

On a freshly booted sandbox whose overlaybd rootfs is cold on the local node (registryfs backend, block cache empty, any registry-mirror / P2P upstream may or may not be warm), the first metadata-heavy workspace scan run from inside the guest takes ~2 orders of magnitude longer than the same command re-run seconds later in the same sandbox.

By "metadata-heavy workspace scan" we mean anything that has to stat/open a large fraction of the application checkout and its VCS working directory: a full tree walk, a full diff against HEAD, a full clean pass, etc. — the workloads a batch of AI-agent rollouts routinely emit as their very first step per sandbox.

In a rollout-style eval (many short-lived sandboxes, one first-step scan each) this single class of first-time operation is the dominant driver of per-command wall-clock budget overruns, even when no individual command's logical work should exceed that budget on a bare Linux host.

Reproduction sketch

• Overlaybd rootfs image with a non-trivial application checkout on it (a typical "AI agent working on an open-source repo" template).
• Boot a fresh sandbox on a node whose local overlaybd cache is cold for that tag.
• Inside the sandbox, run one metadata-heavy workspace-scan command. Time it.
• Immediately re-run the same command in the same sandbox. Time it.

The ratio, on our NVMe-backed nodes with a warm registry-mirror upstream but an empty local overlaybd block cache, is a two-digit multiplier.

Where the time goes (from instrumenting the cold path)

Two structural properties, not an obviously-fixable hot path:

1. Registryfs_v2 serves reads at a 64 KiB block / 256 KiB refill granularity, over the network, into a cache that starts empty. A metadata scan induces thousands of them.
2. Effective concurrency of those fetches while a single guest command is running is essentially the guest's own outstanding-lookup depth. PrefetchConfig.concurrency exists (default 16) but is only used when a trace-based prefetcher is active.

Once the metadata blocks land in the guest kernel page cache, the second run has ~no IO to do. The problem is that nothing pulls ahead of the guest during the first run.

What already exists in-tree but is not wired

The overlaybd code already ships the trace-prefetch primitive under storage/overlaybd/src/prefetch.rs (PrefetchMode::Record | Replay | Disabled), matching the design containerd/accelerated-container-image uses. On our reading of storage/overlaybd/src/image/image_file.rs::init_image_file (around L322–L340), the prefetcher is only constructed when the image config declares acceleration_layer or when a record_trace_path file already exists on disk. Neither is produced by the sandbox launch path today, so this code effectively never runs in production.

Separately, Firecracker Diff snapshot / resume already preserves the guest kernel page cache across pause / resume — so a deliberate "boot → warm → pause → publish, later resume users into it" pipeline would sidestep the cold path entirely for on-node warm resumes. Nothing in the current template-builder flow does this as a warm-up step.

Ask

We would like guidance on which direction upstream considers the right long-term fix, so we can contribute rather than fork:

1. Wire trace prefetch into the sandbox launch path. Either an explicit "warm-up hook" the template builder runs once to record a trace against the target rootfs, or an auto-record-on-first-launch mode that emits the trace next to the image and replays on subsequent cold starts. The mechanism already exists; there is just no producer for the trace input.
2. Formalize a "pre-warm then snapshot" step in the template builder so an image ships with metadata blocks already in a Diff-snapshot memory image, and cold creates route through it. Complementary to (1): (1) helps ad-hoc images, (2) helps the fleet-wide case where the same image is used many times.
3. Raise on-demand concurrency for registryfs_v2 during cold periods independent of trace prefetch (bounded read-ahead below the guest's request rate). Smallest change, smallest expected effect — only worth it if (1)/(2) are both non-goals.

Happy to run measurements against any candidate patch on the same workload if that is useful, and to prototype (1) if the direction is agreed.

[dushulin]

cc @LSX-s-Software @yingdi-shan Do you have any experience solving, or any recommended approaches for, this kind of metadata access scenario (e.g. find / or rm -rf node cache)? Or do you have any related plans going forward to optimize the performance of system metadata access storms?

[yingdi-shan]

Thanks for the detailed investigation. Our current recommendations are:

1. If many sandboxes will use the same set of workspace, boot the template, run a representative workspace scan, and immediately take a snapshot. The guest page cache is then preserved in the snapshot, so subsequent sandboxes created from it should avoid most of the cold metadata path and be much more efficient. You can also tune the guest DAMON reclaim settings, especially by increasing damon_reclaim.min_age, to keep page-cache pages resident longer. Turn on the memory compression can furthur increase the performance on slow network, at the cost of more CPU cycles.

2. Keep the object storage and AgentENV nodes in the same low-latency network whenever possible. Another option is to host object storage close to the nodes yourself using MinIO.

3. Enabling background download and P2P can improve performance, but the current P2P implementation is not mature yet, so it should be evaluated carefully before relying on it in production.

[dushulin]

Thank you for your reply. @yingdi-shan

If many sandboxes will use the same set of workspace, boot the template, run a representative workspace scan, and immediately take a snapshot

We mainly use it for reinforcement learning (RL) training, so each environment is different. We use the cold-sandbox interface for cold start and do not use a template path.

Keep the object storage and AgentENV nodes in the same low-latency network whenever possible. Another option is to host object storage close to the nodes yourself using MinIO.

Enabling P2P can improve performance, but the current P2P implementation is not mature yet, so it should be evaluated carefully before relying on it in production.

In addition, we have integrated Dragonfly's P2P cluster (the same cluster as agentenv) to accelerate pulling of OBD images.
In our scenario, when the model solves problems it also generates some metadata access requests such as find, and the access paths are not fixed. Is there a way to optimize for this kind of scenario?

[yingdi-shan]

In our use cases, the data/workspace is mounted through the extra-drive interface (we're adding a volume API in PR #214). In this scenario, the cache hit rate can be high since many sandboxes share the same block cache, which makes the problem less severe.

[cheney-xiaobo]

In the AgentENV scenario, why not directly use an existing P2P framework like Dragonfly? What unique design considerations led to building a custom P2P framework instead?

[yingdi-shan]

In the AgentENV scenario, why not directly use an existing P2P framework like Dragonfly? What unique design considerations led to building a custom P2P framework instead?

See issue #12 for more context. The main driver is instant snapshotting: we want snapshots to be usable immediately, without waiting for an upload to object storage to complete. Achieving that requires tightly coupling AENV's garbage collection with the P2P layer itself, something a general-purpose, standalone P2P framework isn't designed to support.

Besides, we're not building a P2P framework entirely from scratch. We use Iroh as the underlying P2P layer rather than reinventing that layer.