diff --git a/docs/vrs/02-agent-spec/requirements.md b/docs/vrs/02-agent-spec/requirements.md index 433bc3ed..4f51ebcc 100644 --- a/docs/vrs/02-agent-spec/requirements.md +++ b/docs/vrs/02-agent-spec/requirements.md @@ -13,8 +13,8 @@ the Agent Spec. General field-change behavior that the canonical specification does not yet define is proposed until a matching evals specification and proof change adopts it. -This work supports root [R01, R06, R11, R13 through R19, and R24 through -R26](../requirements.md). +This work supports root [R01, R06, R11, R13 through R19, R24 through R26, and +R31](../requirements.md). It applies to every st2 tool that runs an agent or test. A valid local catalog and host-local runtime state are sufficient. It requires no compare-and-swap (CAS), lock service, cross-host call, or external registry. diff --git a/docs/vrs/02-agent-spec/spec.md b/docs/vrs/02-agent-spec/spec.md index 10ebb0c4..5ef2aedb 100644 --- a/docs/vrs/02-agent-spec/spec.md +++ b/docs/vrs/02-agent-spec/spec.md @@ -318,7 +318,7 @@ Authoring: [pinned tasks][evals-tasks] and [environment][evals-environment]. The Current st2 source: [`AgentSpec` and `Task`](../../../crates/agent-spec/src/spec.rs). Evidence: [spawn construction](../../../src/run.rs). -

F12 Future policy

+

F12 Future policy (R31)

Agent or task `keep`, restart `attempts`, `interval`, `delay`, and `mode`, and task `lifecycle` are future policy. Adopt healthy work. `adopt-only` holds absent @@ -327,11 +327,24 @@ the canonical agent's effective eligibility: `adopt-only` holds it, and exhausting a fail-mode restart policy stops or suppresses it. Invalid policy refuses changes to the related agent and tasks. +`delay` is the minimum spacing between launches in either restart mode. In +`mode = delay`, `attempts` is a rate limit over the sliding `interval` window; +an exhausted limit defers launch until the window clears and never parks the +task. In `mode = fail`, `attempts` is a terminal launch budget counted since +the task was observed alive on every completed accounting pass for a full +`interval`. The budget remains reachable independent of reconcile cadence: a +task that repeatedly launches and dies before recovery is parked after its +declared successful-launch budget is exhausted. A completed accounting pass +that does not observe the task alive breaks accrued recovery uptime rather than +forgiving failures through silence. + Authoring: [pinned complete declaration][evals-fields]. The [pinned explicit-task list][evals-task-fields] and st2 `9887b28` predate task `lifecycle`. Current st2 source: -[`Restart` and `TaskLifecycle`](../../../crates/agent-spec/src/spec.rs). -Evidence: [policy planning](../../../src/reconcile.rs). +[`Restart` and `TaskLifecycle`](../../../crates/agent-spec/src/spec.rs), +[`FlappingCap`](../../../src/flapping.rs), and +[`execute`](../../../src/run.rs). Evidence: +[policy planning](../../../src/reconcile.rs).

F13 retired #true

diff --git a/docs/vrs/requirements.md b/docs/vrs/requirements.md index b436fe56..ea5bbe98 100644 --- a/docs/vrs/requirements.md +++ b/docs/vrs/requirements.md @@ -52,6 +52,22 @@ accepted. deterministic st2 reconciler keeps declared local processes converged; the root observes host-local runtime health, diagnoses failures, performs bounded recovery, and escalates what it cannot resolve. +- **R31 Reachable restart bounds:** Within one supervisor run, restart + accounting is per task and only successful launches spend the declared + budget. `delay` is the minimum launch spacing in either mode. In delay mode, + `attempts` limits launches over the sliding `interval` window without parking + the task. In fail mode, `attempts` is reachable independent of reconcile + cadence: a task that repeatedly launches and dies without being observed + alive on every completed accounting pass for a full `interval` is parked for + the remainder of the supervisor run after its successful-launch budget is + exhausted. A completed pass that does not observe the task alive breaks + accrued recovery uptime rather than forgiving failures through silence. +- **R32 Bounded helper teardown:** After st2 spawns a bounded non-interactive + helper, input delivery failure or deadline expiry targets the helper's entire + process group, not only its direct child, so teardown includes descendants + that outlive that child. st2 either reaps the direct child before returning or + transfers wait ownership to a background reaper; the failure remains bounded + and reports its originating input error or timeout. - **R22 Quiet coordination after events:** A network with minimal or default personas stays quiet while useful work continues. Agents coordinate only after an inbox DING, a durable failure, a real blocker, a completion or decision diff --git a/docs/vrs/spec.md b/docs/vrs/spec.md index 85eae627..4fab28c2 100644 --- a/docs/vrs/spec.md +++ b/docs/vrs/spec.md @@ -463,7 +463,7 @@ validate ──► materialize ──► host-local st2 scheduler/reconciler claims fail every conflicting owner before the first workspace write. Targeted reconciliation checks the selected owner against the full fleet, so selection cannot bypass this ownership boundary. -- **R04:** Each machine schedules and reconciles only its pinned work. The st2 +- **R04, R31:** Each machine schedules and reconciles only its pinned work. The st2 loop is deterministic; exactly one declared root agent provides intelligent host-local supervision, bounded recovery, and escalation. Filesystem reads never wake reconciliation; only create, modify, rename, or remove events may @@ -473,6 +473,27 @@ validate ──► materialize ──► host-local st2 scheduler/reconciler failing to restart the agent, or terminally parking it, suppresses companion launch and stops an exact generated companion proved live; explicitly authored sibling tasks remain independent. + Restart accounting is per task and persists across reconcile passes. Only a + successful launch spends its declared budget. Each completed pass supplies + the exact task IDs it proved alive; uninterrupted observed liveness may + forgive a fail-mode budget according to the + [restart field contract](./02-agent-spec/spec.md#f12), while an unobserved task + loses accrued recovery uptime. A pass that exits before execution neither + supplies a liveness observation nor closes the accounting pass. + [PR #191](https://github.com/compoundingtech/st2/pull/191) provides cadence, + recovery, and unobserved-pass evidence for this accounting. +- **R32:** Bounded non-interactive helpers such as `pty list --json` and + `pty metadata patch` start in a fresh session whose leader PID is also its + process-group ID. Standard output and error use regular temporary files, so a + descendant inheriting those descriptors cannot hold a capture pipe open. + After spawn, an input setup or write failure or a deadline expiry sends + `SIGKILL` to the process group and explicitly terminates the direct child. + st2 waits for that child until the cleanup deadline; if it cannot finish the + wait synchronously, a background waiter takes ownership before the failure + returns. The process-group signal reaches a descendant that outlives the + direct child; terminating the direct child alone does not. [PR + #202](https://github.com/compoundingtech/st2/pull/202) provides + descendant-lifetime and direct-child-reap evidence for this contract. - **R06:** st2 passes the complete effective task definition to the underlying launcher so manual and supervised restarts are equivalent. Harness readiness that depends on a dynamically selected account belongs to that declared diff --git a/src/flapping.rs b/src/flapping.rs index c1eaafbf..ad4b0cb8 100644 --- a/src/flapping.rs +++ b/src/flapping.rs @@ -1,18 +1,18 @@ //! The restart cap (M1b) — st2's crash-loop guard, driven by the job's declared `restart{}` policy -//! (spec.md §4 / R16), so every conformant runner behaves identically on a flapping task. +//! (`docs/vrs/requirements.md` R31), so every conformant runner behaves identically on a flapping +//! task. //! //! st2 owns liveness (it respawns a dead task on the next reconcile), so it also owns the restart //! decision. Per task it enforces `delay` spacing, then applies `mode`: `delay` **rate-limits** using //! a sliding `interval` window (keep restarting once the window clears), while `fail` **parks** the //! task (give up + surface it) once its failure budget is spent. //! -//! Those two need different counters, because a rate window cannot express a budget. The supervisor -//! reconciles every 30s by default, so it can start a task at most twice a minute — below the -//! 3-per-minute that the default `attempts 3 / interval 60s` budgets. Counted in a window, `fail`'s -//! budget could never accrue and a permanently-broken task relaunched forever. So `fail` counts -//! launches since the task last *stayed up* for a full `interval`, which is uptime-scoped rather than -//! wall-clock-scoped and cannot be outrun by a coarse cadence. That makes [`FlappingCap::end_pass`] -//! load-bearing: it is how the cap learns a task survived. +//! Those two need different counters, because a rate window cannot express a budget. When a task +//! stays up longer than roughly `interval / attempts` before dying again, earlier launches can age +//! out before `attempts` coexist in the window, so a window-scoped `fail` budget never exhausts. +//! The `fail` counter instead spans launches until the task *stays up* for a full `interval`; +//! reconcile cadence cannot outrun it. That makes [`FlappingCap::end_pass`] load-bearing: it is how +//! the cap learns a task survived. //! //! The clock is injected (`now: Instant`) so every branch is unit-testable without sleeping.