Skip to content

Commit f272019

Browse files
committed
test: prove R11 control-plane replacement safety
1 parent 3e2e819 commit f272019

6 files changed

Lines changed: 287 additions & 46 deletions

File tree

INVARIANTS.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ materialization, messaging, DING, or presence must preserve them.
55

66
| Invariant | Guarantee | Proof |
77
|---|---|---|
8-
| **Supervisor-decoupled lifecycle** | SIGTERM or SIGKILL of st2 never kills a task. A new runner adopts survivors; only explicit teardown kills. This covers PTY and exec tasks. | `tests/nomad_survival.rs` |
8+
| **R11 control-plane replacement safety** | Normal stop or SIGKILL of `st2 up`, atomic binary replacement, and control-plane restart leave live PTY and exec agents usable with the same PID and creation identity. The replacement adopts each survivor exactly once, launches only genuinely missing work, and never duplicates a survivor; only explicit teardown kills an agent. | `tests/nomad_survival.rs::normal_stop_and_binary_replacement_adopt_exec_unchanged_without_duplicate`; `tests/nomad_survival.rs::forced_kill_and_binary_replacement_adopt_exec_unchanged_without_duplicate`; `tests/nomad_survival.rs::normal_stop_and_binary_replacement_adopt_pty_unchanged_without_duplicate`; `tests/nomad_survival.rs::forced_kill_and_binary_replacement_adopt_pty_unchanged_without_duplicate`; `tests/nomad_survival.rs::explicit_teardown_kills_exec_but_plain_stop_does_not`; `tests/nomad_survival.rs::explicit_teardown_kills_pty_but_plain_stop_does_not` |
99
| **Transport-decoupled lifecycle** | Each task is isolated from a supervisor/transport process-group or cgroup cascade. | `tests/transport_isolation.rs`; `tests/transport_isolation_macos.rs` |
1010
| **Clean exec teardown** | Killing an exec task reaps its whole process group. | `tests/exec_backend.rs::exec_kill_reaps_the_whole_process_group_not_just_the_leader` |
1111
| **Bounded restart diagnostics** | Relaunching an exec task preserves the just-finished log as one prior generation while bounding retained diagnostics to current plus prior. Final retirement removes the PID and both logs. | `tests/exec_backend.rs::exec_restart_reap_keeps_bounded_diagnostics_and_final_remove_cleans_them`; `tests/run.rs::up_once_finally_removes_dead_retired_tasks_without_restarting_them` |

README.md

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -172,6 +172,30 @@ For a foreground supervisor on any host:
172172
st2 up --catalog "$CATALOG" --host <host>
173173
```
174174

175+
### Staged control-plane replacement gate
176+
177+
`st2 up` is a replaceable control plane, not the lifetime owner of an agent. Stopping it normally
178+
or killing it must leave every agent running. `st2 down` and declaration retirement are separate,
179+
explicit lifecycle actions and must never be used merely to replace the control plane.
180+
181+
Every staged recovery or binary cutover must retain a pre-stop receipt of each agent's stable task
182+
identity, PID, and process-creation identity. Install the verified replacement binary atomically
183+
while those tasks continue running, restart the control plane, and do not accept the host until its
184+
first reconcile proves all of the following:
185+
186+
- every pre-existing agent is still usable with the same PID and creation identity;
187+
- the replacement reports those agents as adopted and does not launch or duplicate them;
188+
- only genuinely missing declared work is launched;
189+
- explicit teardown remains the only path that stops an agent.
190+
191+
The executable gate drives the real st2 binary and both PTY and exec backends through normal stop,
192+
forced kill, atomic binary replacement, adoption, a live-task heartbeat, and a duplicate-boot
193+
receipt:
194+
195+
```sh
196+
cargo test --test nomad_survival --all-features
197+
```
198+
175199
## Messages, DING, status, and context
176200

177201
Inside a managed task, `CATALOG`, `ST_ROOT`, and `ST_AGENT` are already set:

docs/vrs/requirements.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -56,6 +56,12 @@ accepted.
5656
launch fields.
5757
- **R07 Verified hooks:** Required hook content is installed explicitly and
5858
verified before a rendered agent depends on it.
59+
- **R11 Control-plane replacement safety:** Stopping or killing `st2 up` must
60+
not stop, restart, or replace any agent it launched. st2 can be reinstalled
61+
and restarted while running agents continue unchanged; the replacement
62+
control plane adopts those existing processes by stable identity and starts
63+
only genuinely missing work. Stopping an agent is a separate, explicit
64+
lifecycle action.
5965

6066
### Must externalize agent state and scope
6167

docs/vrs/spec.md

Lines changed: 14 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -11,8 +11,9 @@ replacement for the README, CLI help, KDL examples, or tests.
1111
## Scope
1212

1313
st2 validates a declared agent fleet, materializes agent workspaces, launches
14-
host-local work, supervises restartable tasks, and delivers messages. The
15-
agent grammar and harness-facing contract remain canonical in
14+
host-local work, adopts and supervises independently surviving tasks, and
15+
delivers messages. The agent grammar and harness-facing contract remain
16+
canonical in
1617
[`compoundingtech/evals/AGENT-SPEC.md`](https://github.com/compoundingtech/evals/blob/main/AGENT-SPEC.md).
1718

1819
## Host-local scheduling and supervision
@@ -46,6 +47,17 @@ validate ──► materialize ──► host-local st2 scheduler/reconciler
4647
launcher so manual and supervised restarts are equivalent.
4748
- **R07:** Hook bundles are explicit, content-addressed, installed separately,
4849
and verified before materialization references them.
50+
- **R11:** `st2 up` is a replaceable control plane, not the lifetime owner of
51+
its agents. Normal exit, forced termination, binary replacement, and restart
52+
leave every running agent PID and creation identity unchanged. The new
53+
control plane adopts those processes and starts only missing work; it does
54+
not duplicate them. Agent stop or retirement requires a separate explicit
55+
lifecycle action.
56+
57+
Executable acceptance starts an agent, terminates `st2 up` normally and with
58+
a forced kill, verifies the agent remains alive and usable, replaces the st2
59+
binary, starts the control plane again, and proves adoption with the same
60+
agent PID/creation identity and no duplicate process.
4961

5062
## Message lifecycle
5163

docs/vrs/vision.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,9 @@
3636
path, with unsupported behavior stated plainly.
3737
2. A canonical agent declaration can be validated, materialized, launched, and
3838
restarted without losing its effective launch definition or durable state.
39+
The st2 control plane itself can be stopped, killed, reinstalled, and
40+
restarted without stopping or replacing any running agent; the replacement
41+
runtime adopts the existing agent processes without duplication.
3942
3. Every machine has exactly one root agent that observes local health,
4043
resolves bounded runtime failures, and escalates what it cannot resolve.
4144
4. A human, CoS, root, or supervisor can inspect catalog-backed state to

0 commit comments

Comments
 (0)