Skip to content

Commit 68f7f6d

Browse files
schickling-assistantEvals Owner
andauthored
eval: guard adopt-only migration lifecycle (#53)
* eval: guard adopt-only migration lifecycle agent-session-id: a078daee-6f98-4916-91a8-d21291407789 agent-tool: Codex CLI agent-tool-version: 0.145.0 agent-model: unknown agent-runtime-profile: /nix/store/mnx8agbdq3wiyb6vz63lhgscgazkrn98-coding-agent-runtime-profile/share/coding-agents/profile.json agent-skills-manifest: /nix/store/5r69m9k2llmri3na81518zx0a7y0d3cn-agent-skills-corpus/share/agent-skills/manifest.json tooling-profile: dotfiles@0fb7e03 * Restack adopt-only migration eval --------- Co-authored-by: Evals Owner <evals-owner@invalid>
1 parent 0070ba4 commit 68f7f6d

11 files changed

Lines changed: 287 additions & 2 deletions

File tree

CATALOG.md

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
<!-- Generated by bin/generate-catalog.sh from cell KDL and evidence/*.tsv. Do not edit. -->
22
# Current eval catalog and evidence
33

4-
This is the sole current corpus surface: **46 included cells** (15 model-free,
4+
This is the sole current corpus surface: **47 included cells** (16 model-free,
55
31 model-backed) and **4 retired exclusions**. Inventory fields are derived from the active
66
KDL and executable judge scripts. Latest accepted PASS evidence remains distinct from the append-only run
77
history: a failed last run is visible without being advertised as accepted evidence, and cells with no
@@ -16,6 +16,7 @@ harness-native loader plus canonical hook file.
1616

1717
| Cell | Harness | Model(s) / effort | Model seats | Cost | Timeout | Held-out judges | Latest accepted PASS | Last recorded run |
1818
|---|---|---|---:|---|---|---:|---|---|
19+
| `adopt-only-migration` | model-free || 0 | none | `90s` | 6 || **NO STRUCTURED RUN** |
1920
| `context-resource-continuity` | model-free || 0 | none | `90s` | 4 || **NO STRUCTURED RUN** |
2021
| `crash-ding` | mixed | `claude-sonnet-5+gpt-5.6-sol` / medium | 4 | high | `180s` | 5 || **NO STRUCTURED RUN** |
2122
| `ding-mode` | Claude | `claude-sonnet-5` / medium | 2 | medium | `1200s` | 4 || **NO STRUCTURED RUN** |
@@ -82,6 +83,7 @@ while the last-run column makes a recorded failure distinct from a cell with no
8283

8384
## Harness hook exclusions
8485

86+
- `adopt-only-migration` / `cell` — deterministic native adoption/replacement lifecycle probe; no harness seat
8587
- `context-resource-continuity` / `cell` — deterministic native context/resource restart probe; no harness seat
8688
- `docs` / `judge:cold-reader` — one-shot offline Claude print grader; no bus identity, DING, or hook surface
8789
- `exec-lifecycle-logging` / `cell` — deterministic native exec lifecycle probe; no harness seat
@@ -99,7 +101,7 @@ while the last-run column makes a recorded failure distinct from a cell with no
99101
- `targeted-reconcile-isolation` / `cell` — deterministic selected reconcile isolation probe; no bus-connected agent
100102
- `two-networks-coexist` / `cell` — deterministic catalog-isolation probe; no harness seat
101103

102-
These **16** rows are generated from structured exclusions. A cell-level row is required for
104+
These **17** rows are generated from structured exclusions. A cell-level row is required for
103105
every derived model-free cell. The one-shot docs cold reader is a model-backed grader, but not a bus-connected
104106
agent: it has no st2 identity, DING sidecar, or lifecycle hooks.
105107

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
# adopt-only-migration
2+
3+
**Type:** st2 / live-migration lifecycle · **Runtime dependency:** implemented
4+
by merged [compoundingtech/st2#99](https://github.com/compoundingtech/st2/pull/99),
5+
which closed [compoundingtech/st2#98](https://github.com/compoundingtech/st2/issues/98),
6+
and tracked in evals by
7+
[compoundingtech/evals#52](https://github.com/compoundingtech/evals/issues/52).
8+
9+
**Capabilities required:** `st2,pty,jq`. No model and no bus. Every declaration,
10+
process, PTY record, state file, log, and receipt is synthetic and rooted below
11+
the eval-owned catalog.
12+
13+
**Discriminates:** can an operator publish a declaration that adopts one
14+
already-live process generation without granting authority to create, collect,
15+
or replace it? An absent task and a later-dead adopted task must both remain
16+
held until the declaration explicitly transitions to ordinary service
17+
lifecycle.
18+
19+
## What it proves
20+
21+
- **Live adoption:** a pre-existing PTY is adopted with the same daemon and
22+
child process generation.
23+
- **Absent hold:** an absent adopt-only task never executes its declared launch
24+
command.
25+
- **Exited hold:** once the adopted generation exits, reconciliation retains
26+
its backend record and does not cold-launch a successor.
27+
- **Explicit replacement:** removing the adopt-only lifecycle changes the
28+
desired contract; ordinary reconciliation may then collect and replace the
29+
exited generation.
30+
- **Mutation-valid control:** an ordinary absent service task launches on the
31+
first pass, proving the launch substrate is real.
32+
- **Isolation and cleanup:** the cell uses only its temporary catalog and PTY
33+
root and leaves no live synthetic processes.
34+
35+
## Run it
36+
37+
```sh
38+
st2 eval ./cells/adopt-only-migration/
39+
```
40+
41+
The current cell passes 7/7 against merged st2
42+
`c6846f6239329f0803142afc06c15a07b93937c1`. Historically, it was intentionally
43+
RED before #98/#99 because the runtime treated `lifecycle` as inert metadata;
44+
that pre-implementation result is no longer the current shipping state.
45+
46+
Immutable pre-reap generation receipts remain the distinct design and
47+
acceptance surface in
48+
[compoundingtech/st2#40](https://github.com/compoundingtech/st2/issues/40).
49+
This cell does not freeze a receipt path or schema ahead of that design.
Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
// Adoption-only migration boundary over synthetic tasks in an eval-owned PTY root.
2+
eval {
3+
copy "./fixture"
4+
max-timeout "90s"
5+
6+
run "matrix" {
7+
command "bash ./matrix.sh"
8+
}
9+
10+
judges {
11+
judge "LIVE ADOPTION - an existing task keeps its original process generation" {
12+
exec "grep -Fqx LIVE-ADOPTED-UNCHANGED-GREEN-a091 $RUNS_DIR/matrix.out"
13+
}
14+
judge "ABSENT HOLD - an absent adopt-only task is never cold-launched" {
15+
exec "grep -Fqx ABSENT-ADOPT-ONLY-HELD-GREEN-a091 $RUNS_DIR/matrix.out"
16+
}
17+
judge "EXIT HOLD - an adopted task that exits is retained without replacement" {
18+
exec "grep -Fqx EXITED-ADOPTED-GENERATION-HELD-GREEN-a091 $RUNS_DIR/matrix.out"
19+
}
20+
judge "EXPLICIT REPLACEMENT - ordinary lifecycle transition permits a new generation" {
21+
exec "grep -Fqx EXPLICIT-ORDINARY-REPLACEMENT-GREEN-a091 $RUNS_DIR/matrix.out"
22+
}
23+
judge "ORDINARY CONTROL - an absent ordinary service task cold-launches" {
24+
exec "grep -Fqx ORDINARY-MISSING-LAUNCH-CONTROL-GREEN-a091 $RUNS_DIR/matrix.out"
25+
}
26+
judge "CLEANUP - no synthetic task remains live in the eval root" {
27+
exec "grep -Fqx SYNTHETIC-MIGRATION-ROOT-CLEAN-GREEN-a091 $RUNS_DIR/matrix.out"
28+
}
29+
}
30+
}
Lines changed: 157 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,157 @@
1+
#!/usr/bin/env bash
2+
set -euo pipefail
3+
4+
root="${CATALOG:?CATALOG must be set}"
5+
net="$root/net"
6+
export PTY_ROOT="$net/pty"
7+
export XDG_STATE_HOME="$root/state"
8+
9+
live_id="migrate.live.agent"
10+
absent_id="migrate.absent.agent"
11+
control_id="migrate.control.agent"
12+
live_spec="$net/agents/migrate/live/agent.kdl"
13+
14+
pty_at() {
15+
env -u PTY_SESSION PTY_ROOT="$PTY_ROOT" pty "$@"
16+
}
17+
18+
session() {
19+
pty_at list --json | jq -c --arg id "$1" '.[] | select(.name == $id)'
20+
}
21+
22+
status_of() {
23+
session "$1" | jq -r '.status // empty'
24+
}
25+
26+
pid_of() {
27+
session "$1" | jq -r '.pid // empty'
28+
}
29+
30+
run_count() {
31+
file="$net/$1-count"
32+
test -f "$file" || {
33+
printf '0\n'
34+
return
35+
}
36+
cat "$file"
37+
}
38+
39+
wait_status() {
40+
id="$1"
41+
expected="$2"
42+
for _ in $(seq 1 100); do
43+
status="$(status_of "$id")"
44+
test "$status" = "$expected" && return 0
45+
sleep 0.05
46+
done
47+
printf '%s did not reach status %s (last status: %s)\n' \
48+
"$id" "$expected" "${status:-missing}" >&2
49+
return 1
50+
}
51+
52+
wait_count() {
53+
label="$1"
54+
expected="$2"
55+
for _ in $(seq 1 100); do
56+
test "$(run_count "$label")" -eq "$expected" && return 0
57+
sleep 0.05
58+
done
59+
printf '%s did not reach generation %s\n' "$label" "$expected" >&2
60+
return 1
61+
}
62+
63+
retire() {
64+
spec="$1"
65+
grep -Fq 'retired #true' "$spec" ||
66+
sed -i '/role "worker"/a\\ retired #true' "$spec"
67+
}
68+
69+
cleanup() {
70+
for identity in live absent control; do
71+
retire "$net/agents/migrate/$identity/agent.kdl" 2>/dev/null || true
72+
done
73+
st2 up --once --catalog "$net" --host migrate >/dev/null 2>&1 || true
74+
sleep 0.1
75+
st2 up --once --catalog "$net" --host migrate >/dev/null 2>&1 || true
76+
for id in "$live_id" "$absent_id" "$control_id"; do
77+
pty_at kill "$id" >/dev/null 2>&1 || true
78+
pty_at rm "$id" >/dev/null 2>&1 || true
79+
done
80+
}
81+
trap cleanup EXIT
82+
83+
# Seed exactly one pre-existing process generation before st2 sees the
84+
# declaration. The declared launch command is therefore observable but must not
85+
# execute during adoption.
86+
mkdir -p "$PTY_ROOT"
87+
CATALOG="$net" pty_at run -d --id "$live_id" --tag keep=true \
88+
--cwd "$net/workspace" -- bash "$net/task.sh" live >/dev/null
89+
wait_status "$live_id" running
90+
wait_count live 1
91+
live_pid_before="$(pid_of "$live_id")"
92+
test -n "$live_pid_before"
93+
94+
st2 up --once --catalog "$net" --host migrate >"$root/first-pass.out"
95+
96+
if
97+
test "$(status_of "$live_id")" = running &&
98+
test "$(pid_of "$live_id")" = "$live_pid_before" &&
99+
test "$(run_count live)" -eq 1 &&
100+
grep -Fq 'adopted (1): live' "$root/first-pass.out"
101+
then
102+
echo "LIVE-ADOPTED-UNCHANGED-GREEN-a091"
103+
fi
104+
105+
if
106+
test "$(run_count absent)" -eq 0 &&
107+
test -z "$(session "$absent_id")" &&
108+
grep -Fq 'held (1): migrate.absent.agent' "$root/first-pass.out"
109+
then
110+
echo "ABSENT-ADOPT-ONLY-HELD-GREEN-a091"
111+
fi
112+
113+
if
114+
wait_count control 1 &&
115+
test "$(status_of "$control_id")" = running &&
116+
grep -Fq "$control_id" "$root/first-pass.out"
117+
then
118+
echo "ORDINARY-MISSING-LAUNCH-CONTROL-GREEN-a091"
119+
fi
120+
121+
# The adopted process exits. Adopt-only must retain the dead record and refuse
122+
# both collection and replacement.
123+
pty_at kill "$live_id" >/dev/null
124+
wait_status "$live_id" exited
125+
live_record_before="$(session "$live_id")"
126+
test "$(run_count live)" -eq 1
127+
128+
st2 up --once --catalog "$net" --host migrate >"$root/held-pass.out"
129+
130+
if
131+
test "$(run_count live)" -eq 1 &&
132+
test "$(status_of "$live_id")" = exited &&
133+
test "$(session "$live_id")" = "$live_record_before" &&
134+
grep -Fq 'held (2): migrate.absent.agent, migrate.live.agent' "$root/held-pass.out"
135+
then
136+
echo "EXITED-ADOPTED-GENERATION-HELD-GREEN-a091"
137+
fi
138+
139+
# This is the explicit authorization edge. The declaration becomes an ordinary
140+
# service task; only now may st2 reap the exited record and cold-launch.
141+
sed -i 's/lifecycle "adopt-only"/lifecycle "service"/' "$live_spec"
142+
st2 up --once --catalog "$net" --host migrate >"$root/replacement-pass.out"
143+
144+
if
145+
wait_count live 2 &&
146+
test "$(status_of "$live_id")" = running &&
147+
test "$(pid_of "$live_id")" != "$live_pid_before" &&
148+
grep -Fq "$live_id" "$root/replacement-pass.out" &&
149+
test "$(run_count absent)" -eq 0
150+
then
151+
echo "EXPLICIT-ORDINARY-REPLACEMENT-GREEN-a091"
152+
fi
153+
154+
cleanup
155+
trap - EXIT
156+
test "$(pty_at list --json | jq 'length')" -eq 0
157+
echo "SYNTHETIC-MIGRATION-ROOT-CLEAN-GREEN-a091"
Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
agent "absent" {
2+
host "migrate"
3+
role "worker"
4+
workspace "$CATALOG/workspace"
5+
pty "agent" {
6+
id "migrate.absent.agent"
7+
command "exec bash \"$CATALOG/task.sh\" absent"
8+
lifecycle "adopt-only"
9+
}
10+
}
Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
agent "control" {
2+
host "migrate"
3+
role "worker"
4+
workspace "$CATALOG/workspace"
5+
pty "agent" {
6+
id "migrate.control.agent"
7+
command "exec bash \"$CATALOG/task.sh\" control"
8+
}
9+
}
Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
agent "live" {
2+
host "migrate"
3+
role "worker"
4+
workspace "$CATALOG/workspace"
5+
pty "agent" {
6+
id "migrate.live.agent"
7+
command "exec bash \"$CATALOG/task.sh\" live"
8+
lifecycle "adopt-only"
9+
}
10+
}
Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
#!/usr/bin/env bash
2+
set -euo pipefail
3+
4+
label="${1:?label is required}"
5+
count_file="${CATALOG:?CATALOG must be set}/$label-count"
6+
generation=1
7+
if test -f "$count_file"; then
8+
generation=$(( $(cat "$count_file") + 1 ))
9+
fi
10+
printf '%s\n' "$generation" >"$count_file"
11+
printf '%s-GENERATION-%s-a091\n' "$label" "$generation"
12+
sleep 100000
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
fixture

docs/vrs/spec.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,10 @@ tracked cell
3939
- An LLM judge is not a defect. Its provider, model, prompt, inputs, and failure
4040
behavior must be explicit, and mutation checks must demonstrate useful
4141
discrimination.
42+
- The model-free `adopt-only-migration` cell separates process-generation
43+
adoption from replacement authority: live generations are adopted, dead or
44+
absent migration tasks are held without mutation, and only an explicit
45+
lifecycle transition permits ordinary replacement.
4246

4347
## Current execution
4448

0 commit comments

Comments
 (0)