Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/vrs/02-agent-spec/requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,8 @@ the Agent Spec. General field-change behavior that the canonical specification
does not yet define is proposed until a matching evals specification and proof
change adopts it.

This work supports root [R01, R06, R11, R13 through R19, and R24 through
R26](../requirements.md).
This work supports root [R01, R06, R11, R13 through R19, R24 through R26, and
R31](../requirements.md).
It applies to every st2 tool that runs an agent or test. A valid local catalog
and host-local runtime state are sufficient. It requires no compare-and-swap
(CAS), lock service, cross-host call, or external registry.
Expand Down
19 changes: 16 additions & 3 deletions docs/vrs/02-agent-spec/spec.md
Original file line number Diff line number Diff line change
Expand Up @@ -318,7 +318,7 @@ Authoring: [pinned tasks][evals-tasks] and [environment][evals-environment]. The
Current st2 source: [`AgentSpec` and `Task`](../../../crates/agent-spec/src/spec.rs).
Evidence: [spawn construction](../../../src/run.rs).

<h3 id="f12">F12 Future policy</h3>
<h3 id="f12">F12 Future policy (R31)</h3>

Agent or task `keep`, restart `attempts`, `interval`, `delay`, and `mode`, and
task `lifecycle` are future policy. Adopt healthy work. `adopt-only` holds absent
Expand All @@ -327,11 +327,24 @@ the canonical agent's effective eligibility: `adopt-only` holds it, and
exhausting a fail-mode restart policy stops or suppresses it. Invalid policy
refuses changes to the related agent and tasks.

`delay` is the minimum spacing between launches in either restart mode. In
`mode = delay`, `attempts` is a rate limit over the sliding `interval` window;
an exhausted limit defers launch until the window clears and never parks the
task. In `mode = fail`, `attempts` is a terminal launch budget counted since
the task was observed alive on every completed accounting pass for a full
`interval`. The budget remains reachable independent of reconcile cadence: a
task that repeatedly launches and dies before recovery is parked after its
declared successful-launch budget is exhausted. A completed accounting pass
that does not observe the task alive breaks accrued recovery uptime rather than
forgiving failures through silence.

Authoring: [pinned complete declaration][evals-fields]. The
[pinned explicit-task list][evals-task-fields] and st2 `9887b28` predate task
`lifecycle`. Current st2 source:
[`Restart` and `TaskLifecycle`](../../../crates/agent-spec/src/spec.rs).
Evidence: [policy planning](../../../src/reconcile.rs).
[`Restart` and `TaskLifecycle`](../../../crates/agent-spec/src/spec.rs),
[`FlappingCap`](../../../src/flapping.rs), and
[`execute`](../../../src/run.rs). Evidence:
[policy planning](../../../src/reconcile.rs).

<h3 id="f13">F13 <code>retired #true</code></h3>

Expand Down
16 changes: 16 additions & 0 deletions docs/vrs/requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,22 @@ accepted.
deterministic st2 reconciler keeps declared local processes converged; the
root observes host-local runtime health, diagnoses failures, performs bounded
recovery, and escalates what it cannot resolve.
- **R31 Reachable restart bounds:** Within one supervisor run, restart
accounting is per task and only successful launches spend the declared
budget. `delay` is the minimum launch spacing in either mode. In delay mode,
`attempts` limits launches over the sliding `interval` window without parking
the task. In fail mode, `attempts` is reachable independent of reconcile
cadence: a task that repeatedly launches and dies without being observed
alive on every completed accounting pass for a full `interval` is parked for
the remainder of the supervisor run after its successful-launch budget is
exhausted. A completed pass that does not observe the task alive breaks
accrued recovery uptime rather than forgiving failures through silence.
- **R32 Bounded helper teardown:** After st2 spawns a bounded non-interactive
helper, input delivery failure or deadline expiry targets the helper's entire
process group, not only its direct child, so teardown includes descendants
that outlive that child. st2 either reaps the direct child before returning or
transfers wait ownership to a background reaper; the failure remains bounded
and reports its originating input error or timeout.
- **R22 Quiet coordination after events:** A network with minimal or default
personas stays quiet while useful work continues. Agents coordinate only after
an inbox DING, a durable failure, a real blocker, a completion or decision
Expand Down
23 changes: 22 additions & 1 deletion docs/vrs/spec.md
Original file line number Diff line number Diff line change
Expand Up @@ -463,7 +463,7 @@ validate ──► materialize ──► host-local st2 scheduler/reconciler
claims fail every conflicting owner before the first workspace write.
Targeted reconciliation checks the selected owner against the full fleet, so
selection cannot bypass this ownership boundary.
- **R04:** Each machine schedules and reconciles only its pinned work. The st2
- **R04, R31:** Each machine schedules and reconciles only its pinned work. The st2
loop is deterministic; exactly one declared root agent provides intelligent
host-local supervision, bounded recovery, and escalation. Filesystem reads
never wake reconciliation; only create, modify, rename, or remove events may
Expand All @@ -473,6 +473,27 @@ validate ──► materialize ──► host-local st2 scheduler/reconciler
failing to restart the agent, or terminally parking it, suppresses companion
launch and stops an exact generated companion proved live; explicitly
authored sibling tasks remain independent.
Restart accounting is per task and persists across reconcile passes. Only a
successful launch spends its declared budget. Each completed pass supplies
the exact task IDs it proved alive; uninterrupted observed liveness may
forgive a fail-mode budget according to the
[restart field contract](./02-agent-spec/spec.md#f12), while an unobserved task
loses accrued recovery uptime. A pass that exits before execution neither
supplies a liveness observation nor closes the accounting pass.
[PR #191](https://github.com/compoundingtech/st2/pull/191) provides cadence,
recovery, and unobserved-pass evidence for this accounting.
- **R32:** Bounded non-interactive helpers such as `pty list --json` and
`pty metadata patch` start in a fresh session whose leader PID is also its
process-group ID. Standard output and error use regular temporary files, so a
descendant inheriting those descriptors cannot hold a capture pipe open.
After spawn, an input setup or write failure or a deadline expiry sends
`SIGKILL` to the process group and explicitly terminates the direct child.
st2 waits for that child until the cleanup deadline; if it cannot finish the
wait synchronously, a background waiter takes ownership before the failure
returns. The process-group signal reaches a descendant that outlives the
direct child; terminating the direct child alone does not. [PR
#202](https://github.com/compoundingtech/st2/pull/202) provides
descendant-lifetime and direct-child-reap evidence for this contract.
- **R06:** st2 passes the complete effective task definition to the underlying
launcher so manual and supervised restarts are equivalent. Harness readiness
that depends on a dynamically selected account belongs to that declared
Expand Down
16 changes: 8 additions & 8 deletions src/flapping.rs
Original file line number Diff line number Diff line change
@@ -1,18 +1,18 @@
//! The restart cap (M1b) — st2's crash-loop guard, driven by the job's declared `restart{}` policy
//! (spec.md §4 / R16), so every conformant runner behaves identically on a flapping task.
//! (`docs/vrs/requirements.md` R31), so every conformant runner behaves identically on a flapping
//! task.
//!
//! st2 owns liveness (it respawns a dead task on the next reconcile), so it also owns the restart
//! decision. Per task it enforces `delay` spacing, then applies `mode`: `delay` **rate-limits** using
//! a sliding `interval` window (keep restarting once the window clears), while `fail` **parks** the
//! task (give up + surface it) once its failure budget is spent.
//!
//! Those two need different counters, because a rate window cannot express a budget. The supervisor
//! reconciles every 30s by default, so it can start a task at most twice a minute — below the
//! 3-per-minute that the default `attempts 3 / interval 60s` budgets. Counted in a window, `fail`'s
//! budget could never accrue and a permanently-broken task relaunched forever. So `fail` counts
//! launches since the task last *stayed up* for a full `interval`, which is uptime-scoped rather than
//! wall-clock-scoped and cannot be outrun by a coarse cadence. That makes [`FlappingCap::end_pass`]
//! load-bearing: it is how the cap learns a task survived.
//! Those two need different counters, because a rate window cannot express a budget. When a task
//! stays up longer than roughly `interval / attempts` before dying again, earlier launches can age
//! out before `attempts` coexist in the window, so a window-scoped `fail` budget never exhausts.
//! The `fail` counter instead spans launches until the task *stays up* for a full `interval`;
//! reconcile cadence cannot outrun it. That makes [`FlappingCap::end_pass`] load-bearing: it is how
//! the cap learns a task survived.
//!
//! The clock is injected (`now: Instant`) so every branch is unit-testable without sleeping.

Expand Down
Loading