Auto-suspend quiescent desired-running agents until demand arrives

[schickling-assistant]

Motivation

A desired-state running Agent Spec currently means that its service task must remain materialized continuously. If the task exits, st2 relaunches it. Desired-state suspended, introduced by #169, means something different: an operator has intentionally declared that the agent must not run until another author changes that desired state.

We need a third behavior for standing agents: the agent remains logically available and desired-running, but st2 may tear down its runtime after a verified quiescent boundary and recreate it when authoritative demand arrives.

This is runtime auto-suspension, not automatic mutation of desired state to suspended.

Scenario

A fleet can contain dozens of standing supervisors, workers, and specialists. Many agents wait for long periods between messages. Keeping every OMP process, PTY daemon, runtime heap, thread pool, and previously used tool worker resident gives instant response latency, but it consumes substantial memory and threads even when no work exists.

Killing an apparently idle PTY is not safe:

• PTY detachment does not mean the agent is idle;
• low CPU does not show whether the agent is waiting for approval, running a child job, or holding unexternalized state; and
• killing a desired-running task outside st2 only causes reconciliation to relaunch it.

The lifecycle owner needs an explicit on-demand residency policy and a reliable quiescence protocol.

Measurements

Controlled pristine OMP launch

On 2026-09-06, an isolated OMP 18.1.7 benchmark launched each variant in an empty workspace, made no model calls, waited three seconds after readiness, and measured the complete PTY-rooted process tree. macOS footprint supplied physical footprint; each row is one sample and should be treated as a shape comparison, not a precise fleet forecast.

Variant	Processes	Threads	Physical footprint	Observable ready time
Plain OMP	2	56	209.5 MiB	734.8 ms
Current-profile equivalent	2	55	230.1 MiB	735.6 ms
Profile + --no-session	2	55	220.2 MiB	732.1 ms
Profile + --no-lsp	2	54	241.8 MiB	736.0 ms
Session, LSP, tools, extensions, skills, rules, internal PTY, and title generation disabled	2	53	213.6 MiB	736.0 ms

Every variant still required one OMP/Bun process and one pty-daemon. The PTY daemon accounted for approximately 29 MiB and 14 threads. The minimal variant saved only two threads and approximately 7% physical footprint relative to the profile-equivalent sample. --no-lsp was larger because LSP is already lazy and had not started.

The result is that launch flags do not materially remove the pristine idle baseline while preserving normal agent capability and resumability. The high-leverage boundary is complete process exit.

Shared Linux fleet snapshot

A separate 2026-09-05 snapshot of a 128 GiB, 32-core shared host found:

• 44 managed st2 cgroups held 29.09 GiB;
• 41 cgroups below 1% CPU held 26.64 GiB, or 92% of managed memory;
• managed OMP channel roots had a median PSS of 561,527 KiB;
• all observed OMP processes, including direct sessions and helper roles, accounted for 39.6 GiB PSS, 9.0 GiB swap, and 5,663 threads.

The 26.64 GiB figure is an upper bound, not an estimate of safely reclaimable memory. A semantic audit of ten detached candidates found that most were still useful, active through descendants, or intentionally blocked; only one was immediately proven stale. This establishes both the potential scale and why CPU/attachment heuristics are insufficient.

The operational impact is tracked in schickling/dotfiles#2375.

Desired model

The Agent Spec should be able to opt into an on-demand residency policy. Exact schema is open. The important distinction is:

desired state: running
residency policy: always | on-demand
observed runtime: cold | starting | active | quiescing

cold, starting, active, and quiescing are observations derived by st2. They must not become a second desired-state database.

Auto-suspend

For an on-demand agent, crossing an idle threshold makes the agent only a candidate. st2 may tear down the runtime only after an authoritative quiescence protocol confirms that:

• no model turn or tool call is running;
• no owned child job still requires the runtime;
• no approval or human response is pending unless that state has an explicit cold-safe representation;
• no client attachment or explicit keep-live lease requires process-local continuity;
• durable working state and session data are flushed; and
• admission of new work is fenced while teardown completes.

st2 then stops the canonical task and its derived/owned tasks together and verifies that the complete runtime is gone. A failed flush or surviving owned process leaves the agent non-cold and produces an observable error.

Wake

Authoritative demand should wake a cold agent through the normal managed launch path. Candidate demand includes:

• a durable inbox/control message;
• an explicit wake or attach request;
• a newly delegated assignment; and
• a declared schedule or event trigger.

Wake must be single-flight and idempotent. Concurrent demand joins the same launch. The agent becomes available only after the normal driver readiness contract succeeds.

Cold residency preserves the Agent Spec identity, workspace, Resource bindings, inbox, externalized working state, and durable provider/auth/MCP state. It does not promise to preserve eval globals, open LSP documents, sockets, TUI state, or in-flight operations. The normal cold-start contract remains authoritative; transcript attachment, where explicitly declared and validated, is an optional optimization rather than a hidden requirement.

Smallest end-to-end slice

1. Add an opt-in on-demand residency policy for one managed service task.
2. Accept an injected, fenced quiescence observation from the harness driver; do not parse provider UI text in st2 core.
3. After a deterministic idle threshold, fence admission, flush/checkpoint, tear down the complete owned runtime, and record a receipt.
4. Treat one durable native message as demand, launch through ordinary reconciliation, and deliver the message after readiness.
5. Expose observed residency, transition reason, Agent Run/generation, timestamps, and failure state in inventory and telemetry.
6. Test with injected time and lifecycle events. Do not use wall-clock sleeps.

Before enabling a fleet policy, run a one-seat proof that measures memory/thread release, cold readiness, first-response latency, transcript/working-state continuity, a real tool call, and whether authentication or MCP state changes. Follow with repeated small/large-session cycles and a canary cohort.

Non-goals

• Reusing desired-state suspended for automatic idling. That state remains explicit author intent.
• Inferring semantic idleness from PTY detachment, CPU percentage, or terminal text.
• SIGSTOP/process freezing. It retains mappings and threads and can worsen swap pressure.
• Presenting cgroup memory caps as efficiency. Resource envelopes such as Design host resource envelopes and supervisor-adjustable leases for managed agents #208 provide containment, not resident-footprint elimination.
• Disabling tools, sessions, or LSP globally.
• Fresh-context rotation. Continuously exercise agent restartability with policy-driven fresh-context rotation #171 immediately replaces one run with another; auto-suspension intentionally leaves an interval with no runtime and waits for demand.

Questions

• Which generic driver signal proves quiescence without putting harness-specific vocabulary in st2 core?
• Should residency policy live on the agent, the service task, or a compiled driver task group?
• Which demand sources belong in the first slice?
• How should pending approval/human-wait states and keep-live leases be represented?
• Does a cold agent retain dead PTY metadata for diagnostics, or retire the session record after a receipt is durable?
• How should auto-suspend and wake budgets compose with crash-loop restart accounting?

Related

• Add reversible Agent Spec desired state with suspension rationale #169: explicit, author-controlled desired-state suspension.
• Notify a supervisor when a managed child remains idle past a configured threshold #173: notify a supervisor about sustained child idleness; it does not reclaim the runtime.
• Continuously exercise agent restartability with policy-driven fresh-context rotation #171: policy-driven fresh-context rotation; it starts a replacement immediately rather than waiting cold.
• Design host resource envelopes and supervisor-adjustable leases for managed agents #208: aggregate resource envelopes and leases; containment is complementary.
• Add optional turn-boundary + fresh-idle DING without harness parsing in core #111: turn-boundary and fresh-idle DING delivery.
• Define renderer-owned harness sessions and the generic st2 lifecycle boundary #124 and Define Axe harness drivers and extract all harness-specific behavior from st2 core #162: generic st2 lifecycle versus harness-driver responsibilities.

Posted on behalf of @schickling

field	value
agent_identity	mbp2025.direct.omp.bn5hrz39
session	mbp2025.bn5hrz39
agent_persona	generalist
agent_supervisor	unavailable
agent_tool	OMP
agent_tool_version	18.1.7
agent_runtime	OMP 18.1.7
tooling_profile	dotfiles@4816a7a