Skip to content

Commit 171b669

Browse files
docs(vrs): define reachable restart budgets
agent-tool: Codex agent-tool-version: 0.145.0 agent-runtime: Codex 0.145.0 agent-session-lookup: unavailable tooling-profile: dotfiles@unknown-dirty
1 parent f78ad95 commit 171b669

5 files changed

Lines changed: 46 additions & 14 deletions

File tree

docs/vrs/02-agent-spec/requirements.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -13,8 +13,8 @@ the Agent Spec. General field-change behavior that the canonical specification
1313
does not yet define is proposed until a matching evals specification and proof
1414
change adopts it.
1515

16-
This work supports root [R01, R06, R11, R13 through R19, and R24 through
17-
R26](../requirements.md).
16+
This work supports root [R01, R06, R11, R13 through R19, R24 through R26, and
17+
R31](../requirements.md).
1818
It applies to every st2 tool that runs an agent or test. A valid local catalog
1919
and host-local runtime state are sufficient. It requires no compare-and-swap
2020
(CAS), lock service, cross-host call, or external registry.

docs/vrs/02-agent-spec/spec.md

Lines changed: 16 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -318,7 +318,7 @@ Authoring: [pinned tasks][evals-tasks] and [environment][evals-environment]. The
318318
Current st2 source: [`AgentSpec` and `Task`](../../../crates/agent-spec/src/spec.rs).
319319
Evidence: [spawn construction](../../../src/run.rs).
320320

321-
<h3 id="f12">F12 Future policy</h3>
321+
<h3 id="f12">F12 Future policy (R31)</h3>
322322

323323
Agent or task `keep`, restart `attempts`, `interval`, `delay`, and `mode`, and
324324
task `lifecycle` are future policy. Adopt healthy work. `adopt-only` holds absent
@@ -327,11 +327,24 @@ the canonical agent's effective eligibility: `adopt-only` holds it, and
327327
exhausting a fail-mode restart policy stops or suppresses it. Invalid policy
328328
refuses changes to the related agent and tasks.
329329

330+
`delay` is the minimum spacing between launches in either restart mode. In
331+
`mode = delay`, `attempts` is a rate limit over the sliding `interval` window;
332+
an exhausted limit defers launch until the window clears and never parks the
333+
task. In `mode = fail`, `attempts` is a terminal launch budget counted since
334+
the task was observed alive on every completed accounting pass for a full
335+
`interval`. The budget remains reachable independent of reconcile cadence: a
336+
task that repeatedly launches and dies before recovery is parked after its
337+
declared successful-launch budget is exhausted. A completed accounting pass
338+
that does not observe the task alive breaks accrued recovery uptime rather than
339+
forgiving failures through silence.
340+
330341
Authoring: [pinned complete declaration][evals-fields]. The
331342
[pinned explicit-task list][evals-task-fields] and st2 `9887b28` predate task
332343
`lifecycle`. Current st2 source:
333-
[`Restart` and `TaskLifecycle`](../../../crates/agent-spec/src/spec.rs).
334-
Evidence: [policy planning](../../../src/reconcile.rs).
344+
[`Restart` and `TaskLifecycle`](../../../crates/agent-spec/src/spec.rs),
345+
[`FlappingCap`](../../../src/flapping.rs), and
346+
[`execute`](../../../src/run.rs). Evidence:
347+
[policy planning](../../../src/reconcile.rs).
335348

336349
<h3 id="f13">F13 <code>retired #true</code></h3>
337350

docs/vrs/requirements.md

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,16 @@ accepted.
5252
deterministic st2 reconciler keeps declared local processes converged; the
5353
root observes host-local runtime health, diagnoses failures, performs bounded
5454
recovery, and escalates what it cannot resolve.
55+
- **R31 Reachable restart bounds:** Within one supervisor run, restart
56+
accounting is per task and only successful launches spend the declared
57+
budget. `delay` is the minimum launch spacing in either mode. In delay mode,
58+
`attempts` limits launches over the sliding `interval` window without parking
59+
the task. In fail mode, `attempts` is reachable independent of reconcile
60+
cadence: a task that repeatedly launches and dies without being observed
61+
alive on every completed accounting pass for a full `interval` is parked for
62+
the remainder of the supervisor run after its successful-launch budget is
63+
exhausted. A completed pass that does not observe the task alive breaks
64+
accrued recovery uptime rather than forgiving failures through silence.
5565
- **R32 Bounded helper teardown:** After st2 spawns a bounded non-interactive
5666
helper, input delivery failure or deadline expiry targets the helper's entire
5767
process group, not only its direct child, so teardown includes descendants

docs/vrs/spec.md

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -463,7 +463,7 @@ validate ──► materialize ──► host-local st2 scheduler/reconciler
463463
claims fail every conflicting owner before the first workspace write.
464464
Targeted reconciliation checks the selected owner against the full fleet, so
465465
selection cannot bypass this ownership boundary.
466-
- **R04:** Each machine schedules and reconciles only its pinned work. The st2
466+
- **R04, R31:** Each machine schedules and reconciles only its pinned work. The st2
467467
loop is deterministic; exactly one declared root agent provides intelligent
468468
host-local supervision, bounded recovery, and escalation. Filesystem reads
469469
never wake reconciliation; only create, modify, rename, or remove events may
@@ -473,6 +473,15 @@ validate ──► materialize ──► host-local st2 scheduler/reconciler
473473
failing to restart the agent, or terminally parking it, suppresses companion
474474
launch and stops an exact generated companion proved live; explicitly
475475
authored sibling tasks remain independent.
476+
Restart accounting is per task and persists across reconcile passes. Only a
477+
successful launch spends its declared budget. Each completed pass supplies
478+
the exact task IDs it proved alive; uninterrupted observed liveness may
479+
forgive a fail-mode budget according to the
480+
[restart field contract](./02-agent-spec/spec.md#f12), while an unobserved task
481+
loses accrued recovery uptime. A pass that exits before execution neither
482+
supplies a liveness observation nor closes the accounting pass.
483+
[PR #191](https://github.com/compoundingtech/st2/pull/191) provides cadence,
484+
recovery, and unobserved-pass evidence for this accounting.
476485
- **R32:** Bounded non-interactive helpers such as `pty list --json` and
477486
`pty metadata patch` start in a fresh session whose leader PID is also its
478487
process-group ID. Standard output and error use regular temporary files, so a

src/flapping.rs

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,18 +1,18 @@
11
//! The restart cap (M1b) — st2's crash-loop guard, driven by the job's declared `restart{}` policy
2-
//! (spec.md §4 / R16), so every conformant runner behaves identically on a flapping task.
2+
//! (`docs/vrs/requirements.md` R31), so every conformant runner behaves identically on a flapping
3+
//! task.
34
//!
45
//! st2 owns liveness (it respawns a dead task on the next reconcile), so it also owns the restart
56
//! decision. Per task it enforces `delay` spacing, then applies `mode`: `delay` **rate-limits** using
67
//! a sliding `interval` window (keep restarting once the window clears), while `fail` **parks** the
78
//! task (give up + surface it) once its failure budget is spent.
89
//!
9-
//! Those two need different counters, because a rate window cannot express a budget. The supervisor
10-
//! reconciles every 30s by default, so it can start a task at most twice a minute — below the
11-
//! 3-per-minute that the default `attempts 3 / interval 60s` budgets. Counted in a window, `fail`'s
12-
//! budget could never accrue and a permanently-broken task relaunched forever. So `fail` counts
13-
//! launches since the task last *stayed up* for a full `interval`, which is uptime-scoped rather than
14-
//! wall-clock-scoped and cannot be outrun by a coarse cadence. That makes [`FlappingCap::end_pass`]
15-
//! load-bearing: it is how the cap learns a task survived.
10+
//! Those two need different counters, because a rate window cannot express a budget. When a task
11+
//! stays up longer than roughly `interval / attempts` before dying again, earlier launches can age
12+
//! out before `attempts` coexist in the window, so a window-scoped `fail` budget never exhausts.
13+
//! The `fail` counter instead spans launches until the task *stays up* for a full `interval`;
14+
//! reconcile cadence cannot outrun it. That makes [`FlappingCap::end_pass`] load-bearing: it is how
15+
//! the cap learns a task survived.
1616
//!
1717
//! The clock is injected (`now: Instant`) so every branch is unit-testable without sleeping.
1818

0 commit comments

Comments
 (0)