Warm Firecracker pool never decays its refill target after a burst

[epicvinny]

Affected component: Firecracker runtime

Problem statement

As an operator running AgentENV with the warm Firecracker process pool enabled,
I need the pool's refill target to decay after the burst that raised it has passed,
because the current behavior pins the excess warm processes — and the memory they
hold — for the entire lifetime of the server process.

Use case

Sandbox hosts serving bursty workloads (e.g. CI/agent sandbox creation in business
hours, idle overnight). A single burst of sandbox creations ratchets the geometric
fill target up; hours later the host still holds that many parked Firecracker
processes. On a shared host this is pure memory waste and also keeps more processes
parked with oom_score_adj=1000 than the steady state needs.

Current behavior and workarounds

WarmPool's maintenance worker computes its fill target from a geometric
high-watermark that only ever ratchets up on acquisitions. There is no time-based
decay: once raised, the target stays for the process lifetime. The only workaround
is restarting the server, which defeats the purpose of a warm pool.

Desired behavior

• Given a configurable idle TTL (seconds),
• When no warm-process acquisition happens within the TTL,
• Then the maintenance worker decays the refill target back to the configured low
  watermark and drains the excess warm processes.
• Any acquisition resets the idle clock (busy pools never decay).
• 0 disables decay and preserves the current ratchet behavior.

Proposed approach

Add PoolConfig::idle_ttl to the warm-pool crate: the maintenance loop tracks
the last-acquisition instant, wakes on the TTL boundary, and when expired reduces
the fill target toward the low watermark and drains the excess entries. Wire it as
[pool.firecracker] idle_ttl_secs (default 600, 0 disables), documented in
config/default.toml and the configuration reference. The network-slot and ublk
pools keep idle_ttl: None (historical ratchet): they hold no guest processes, so
idle decay buys nothing there.

Implementation ready: https://github.com/ResultadosDigitais/AgentENV (branch
upstream/warm-pool-idle-ttl), with unit tests for decay, TTL wake-up, and
clock reset.

Compatibility and operational impact

API/config changes: new optional key [pool.firecracker] idle_ttl_secs (default
600). No breaking change; setting 0 restores exact previous behavior.

Snapshot or storage format changes: none.

New host/runtime requirements: none.

Upgrade and rollback considerations: rolling back simply ignores the new key;
behavior with the key absent on old builds is unchanged.

Alternatives considered

• Time-based decay of parked entries themselves (per-entry TTL): wastes warm
  processes that a steady-state workload still needs; decaying the refill target
  preserves the configured low watermark.
• External memory-pressure-driven scaling: couples pool sizing to host-level
  signals the pool does not currently consume; a TTL is deterministic and
  testable.
• Doing nothing (restart to reset): operationally hostile for long-lived hosts.

Non-goals

• Changing the geometric growth policy itself.
• Per-entry idle TTL for parked processes.
• Applying decay to the network-slot or ublk pools.

Contribution

I can implement it (implementation ready, PR to follow).

Pre-submission checklist

• I searched existing issues and discussions and did not find a duplicate.
• I described a concrete problem and use case, not only a proposed technology.
• I understand that acceptance of the problem does not imply approval of a particular implementation.