fix(eval): close canonical causal admission

agent-session-id: dev3.dotfiles-cos-misc-agent-runtime-simplification
agent-tool: Codex
agent-tool-version: 0.145.0
agent-model: gpt-5.6-sol
agent-runtime-profile: /home/schickling/.config/coding-agents/profile.json
agent-skills-manifest: /nix/store/nk9iml2841l1yjjg0f6f0d3y60zkg1nn-agent-skills-corpus/share/agent-skills/manifest.json
tooling-profile: dotfiles@4a0515f

Author: schickling-assistant

README.md

@@ -409,13 +409,14 @@ eval {
 `canonical-agents` then discovers and materializes declarations at
 `agents/<host>/<identity>/agent.kdl`. It is mutually exclusive with compact `team` / `agent` seats:
 the discovered vector is the sole authority for launch, kickoff routing, supervision, logs, and
-teardown. Strict catalog validation, main-PTY admission, and warning-free materialization all finish
-before a seat starts; backend launch errors are fatal. The native inbox/archive paths are frozen from
-that admitted vector, so later catalog mutation cannot redirect eval traffic. A multi-seat team
-completes only after the existing worker-report ordering; a singleton completes on its first
-interviewer-to-requester confirmation that post-dates the exact kickoff receipt. Canonical completion
-gates the verdict. Without the directive, Agent Spec-shaped files inside a fixture remain inert and
-compact evals retain their flat bus and completion semantics.
+teardown. Strict catalog validation, main-PTY admission, fleet-unique nonempty task runtime IDs, and
+warning-free materialization all finish before a seat starts; backend launch errors are fatal. The
+native inbox/archive paths are frozen from that admitted vector, so later catalog mutation cannot
+redirect eval traffic. A multi-seat team completes only after the existing worker-report ordering.
+For a singleton, the requester inbox is snapshotted before kickoff; only a newly appearing
+interviewer reply at-or-after the exact kickoff receipt completes it. Canonical completion gates the
+verdict. Without the directive, Agent Spec-shaped files inside a fixture remain inert and compact
+evals retain their flat bus and completion semantics.
 
 `st2 compile-agent` remains experimental. Hand-authored KDL is the canonical st2 authoring
 interface, and generated output must be reviewed before materialization.

docs/vrs/spec.md

@@ -29,9 +29,10 @@ and teardown.
 Admission applies `validate_for_host` strictly, then fails before spawn when
 discovery is empty, malformed, warning-bearing, duplicate, retired, nonlocal,
 noncanonical, root-overriding, unrunnable, or does not expose exactly one
-independently launchable service main PTY per declaration. Main runtime IDs
-must be nonempty and fleet-unique. Materialization warnings and backend launch
-errors are fatal. The kickoff target must resolve to exactly one member of the
+independently launchable service main PTY per declaration. Every resolved task
+runtime ID must be nonempty and fleet-unique, including collisions between a
+main PTY and another declaration's sidecar. Materialization warnings and
+backend launch errors are fatal. The kickoff target must resolve to exactly one member of the
 discovered fleet. The eval owns one native `CATALOG` / `ST_ROOT` and its
 `<catalog>/pty` registry; declarations cannot override those roots. Workspace
 renders are materialized before any seat starts.
@@ -40,9 +41,12 @@ Native inbox/archive paths are derived once from the admitted Agent Spec paths
 and carried as frozen data; routing never re-discovers the mutable catalog.
 The requester alone is an explicit eval-owned flat mailbox. Multi-seat
 completion retains the worker-report-before-supervisor-confirmation ordering.
-A singleton canonical team completes when its interviewer-to-requester
-confirmation post-dates the exact kickoff receipt. Canonical completion is a
-gating judge, so a timeout cannot pass on unrelated final-state checks alone.
+For a singleton, the eval snapshots the requester inbox before kickoff and
+completes only for a newly appearing interviewer reply whose timestamp is
+at-or-after the exact kickoff receipt. The filename snapshot rejects
+future-dated pre-seeded messages while `>=` accepts a causally new same-ms
+reply. Canonical completion is a gating judge, so a timeout cannot pass on
+unrelated final-state checks alone.
 
 Without `canonical-agents`, fixture declarations are not discovered or launched
 and compact evals retain their catalog-less flat bus. This explicit opt-in keeps

src/eval_run.rs

@@ -128,6 +128,12 @@ fn admitted_route<'a>(
         .unwrap_or_else(|| panic!("strict canonical admission did not freeze route for `{id}`"))
 }
 
+fn task_runtime_id(spec: &AgentSpec, task: &Task, host: &str) -> String {
+    task.id
+        .clone()
+        .unwrap_or_else(|| format!("{}.{}", spec.bus_id(host), task.name))
+}
+
 fn main_pty_id(spec: &AgentSpec, host: &str) -> Result<String> {
     let bus_id = spec.bus_id(host);
     let main = spec
@@ -151,10 +157,7 @@ fn main_pty_id(spec: &AgentSpec, host: &str) -> Result<String> {
             "Agent Spec `{bus_id}` main PTY named `agent` must be an independently launchable service"
         );
     }
-    let id = main
-        .id
-        .clone()
-        .unwrap_or_else(|| format!("{bus_id}.agent"));
+    let id = task_runtime_id(spec, main, host);
     if id.trim().is_empty() {
         anyhow::bail!("Agent Spec `{bus_id}` main PTY id must be nonempty");
     }
@@ -211,7 +214,7 @@ fn load_canonical_eval_team(catalog: &Path, host: &str) -> Result<CanonicalEvalT
 
     let mut paths = BTreeMap::<PathBuf, usize>::new();
     let mut bus_ids = HashSet::new();
-    let mut main_ids = HashSet::new();
+    let mut runtime_ids = BTreeMap::<String, (String, bool)>::new();
     let mut seat_ids = Vec::new();
     let mut routes = BTreeMap::new();
     for spec in &found.specs {
@@ -254,8 +257,27 @@ fn load_canonical_eval_team(catalog: &Path, host: &str) -> Result<CanonicalEvalT
             }
         }
         let main_id = main_pty_id(spec, host)?;
-        if !main_ids.insert(main_id.clone()) {
-            anyhow::bail!("canonical-agents found duplicate main PTY id `{main_id}`");
+        for task in &spec.tasks {
+            let runtime_id = task_runtime_id(spec, task, host);
+            if runtime_id.trim().is_empty() {
+                anyhow::bail!(
+                    "canonical-agents Agent Spec `{bus_id}` task `{}` runtime task id must be nonempty",
+                    task.name
+                );
+            }
+            let is_main = task.kind == TaskKind::Pty && task.name == "agent";
+            if let Some((previous, previous_is_main)) =
+                runtime_ids.insert(runtime_id.clone(), (bus_id.clone(), is_main))
+            {
+                let kind = if is_main && previous_is_main {
+                    "duplicate main PTY runtime task id"
+                } else {
+                    "duplicate runtime task id"
+                };
+                anyhow::bail!(
+                    "canonical-agents found {kind} `{runtime_id}` in `{previous}` and `{bus_id}`"
+                );
+            }
         }
         seat_ids.push(main_id);
         let agent_dir = spec
@@ -519,15 +541,17 @@ fn from_is(from: Option<&str>, id: &str) -> bool {
 
 /// Wait for the DONE signal, message-driven (not grade-poll). Multi-seat teams require a
 /// `sup → requester` confirmation whose timestamp follows a `worker → sup` report. A canonical
-/// singleton instead requires its confirmation to post-date the exact kickoff receipt. Compact
-/// singleton semantics remain unchanged. Bounded by `timeout`. Returns whether done fired.
+/// singleton instead requires a causally new requester-inbox entry at-or-after the exact kickoff
+/// receipt. Compact singleton semantics remain unchanged. Bounded by `timeout`. Returns whether done
+/// fired.
 fn wait_done(
     bus: &Path,
     canonical_routes: Option<&BTreeMap<String, CanonicalRoute>>,
     sup: &str,
     requester: &str,
     workers: &[String],
     kickoff_ts: Option<u64>,
+    requester_before_kickoff: Option<&HashSet<String>>,
     timeout: Duration,
     on_tick: &mut dyn FnMut(),
 ) -> bool {
@@ -559,11 +583,16 @@ fn wait_done(
         let sup_archived = crate::message::list_dir(&sup_archive).unwrap_or_default();
         if workers.is_empty()
             && let Some(kickoff_ts) = kickoff_ts
+            && let Some(before) = requester_before_kickoff
         {
             let confirmed = crate::message::list_dir(&req_inbox)
                 .unwrap_or_default()
                 .iter()
-                .any(|m| from_is(m.from.as_deref(), sup) && m.ts_ms > kickoff_ts);
+                .any(|m| {
+                    !before.contains(&m.filename)
+                        && from_is(m.from.as_deref(), sup)
+                        && m.ts_ms >= kickoff_ts
+                });
             if confirmed {
                 return true;
             }
@@ -1074,6 +1103,13 @@ fn run_eval_inner(spec: &Spec, eval: &Eval, spec_dir: &Path, catalog: &Path, hos
             Some(routes) => admitted_route(routes, &sup).inbox.clone(),
             None => bus.join(&sup).join("inbox"),
         };
+        let requester_before_kickoff = eval.canonical_agents.then(|| {
+            crate::message::list_dir(&bus.join(&msg.from).join("inbox"))
+                .unwrap_or_default()
+                .into_iter()
+                .map(|message| message.filename)
+                .collect::<HashSet<_>>()
+        });
         let kickoff_receipt =
             crate::message::send_to_inbox(&to_inbox, &msg.from, None, None, &[], &body)
             .with_context(|| format!("seeding kickoff into {}", to_inbox.display()))?;
@@ -1185,6 +1221,7 @@ fn run_eval_inner(spec: &Spec, eval: &Eval, spec_dir: &Path, catalog: &Path, hos
                 &msg.from,
                 &workers,
                 kickoff_ts,
+                requester_before_kickoff.as_ref(),
                 eval.max_timeout,
                 &mut tick,
             )
@@ -1559,6 +1596,28 @@ agent "worker" { identity "worker"; host "evalhost"; argv "true" }
   identity "two"
   host "evalhost"
   pty "agent" { id "shared"; command "sleep 60" }
+}"#,
+                    ),
+                ],
+            ),
+            (
+                "duplicate runtime task id",
+                vec![
+                    (
+                        "agents/evalhost/one/agent.kdl",
+                        r#"agent "one" {
+  identity "one"
+  host "evalhost"
+  pty "agent" { id "shared"; command "sleep 60" }
+}"#,
+                    ),
+                    (
+                        "agents/evalhost/two/agent.kdl",
+                        r#"agent "two" {
+  identity "two"
+  host "evalhost"
+  pty "agent" { id "two-main"; command "sleep 60" }
+  exec "poison" { id "shared"; command "true" }
 }"#,
                     ),
                 ],
@@ -1777,6 +1836,7 @@ agent "worker" { identity "worker"; host "evalhost"; argv "true" }
             req,
             &workers,
             None,
+            None,
             Duration::from_millis(100),
             noop,
         ));
@@ -1791,6 +1851,7 @@ agent "worker" { identity "worker"; host "evalhost"; argv "true" }
             req,
             &workers,
             None,
+            None,
             Duration::from_millis(100),
             noop,
         ));
@@ -1803,6 +1864,7 @@ agent "worker" { identity "worker"; host "evalhost"; argv "true" }
             req,
             &workers,
             None,
+            None,
             Duration::from_millis(2000),
             noop,
         ));
@@ -1830,6 +1892,7 @@ agent "worker" { identity "worker"; host "evalhost"; argv "true" }
                 req,
                 &workers,
                 None,
+                None,
                 Duration::from_millis(2000),
                 noop,
             ),
@@ -1852,6 +1915,7 @@ agent "worker" { identity "worker"; host "evalhost"; argv "true" }
             "req",
             &["w".to_string()],
             None,
+            None,
             Duration::from_millis(400),
             &mut tick,
         );
@@ -1860,7 +1924,7 @@ agent "worker" { identity "worker"; host "evalhost"; argv "true" }
     }
 
     #[test]
-    fn canonical_singleton_done_requires_a_confirmation_after_the_exact_kickoff() {
+    fn canonical_singleton_done_is_new_after_kickoff_and_allows_the_same_millisecond() {
         let root = tempfile::tempdir().unwrap();
         let agent_dir = root.path().join("agents/h/interviewer");
         let routes = BTreeMap::from([(
@@ -1871,7 +1935,15 @@ agent "worker" { identity "worker"; host "evalhost"; argv "true" }
             },
         )]);
         let requester = root.path().join("requester/inbox");
-        seed_msg(&requester, 1_700_000_001_000, "aaaaaa", "h.interviewer");
+        let kickoff_ts = 1_700_000_002_000;
+        // An already-present message may even claim a FUTURE timestamp. Causality comes from the
+        // pre-kickoff filename snapshot, not wall-clock trust.
+        seed_msg(&requester, kickoff_ts + 1_000, "aaaaaa", "h.interviewer");
+        let before = crate::message::list_dir(&requester)
+            .unwrap()
+            .into_iter()
+            .map(|message| message.filename)
+            .collect::<HashSet<_>>();
         let noop = &mut (|| {}) as &mut dyn FnMut();
         assert!(
             !wait_done(
@@ -1880,25 +1952,28 @@ agent "worker" { identity "worker"; host "evalhost"; argv "true" }
                 "h.interviewer",
                 "requester",
                 &[],
-                Some(1_700_000_002_000),
+                Some(kickoff_ts),
+                Some(&before),
                 Duration::from_millis(100),
                 noop,
             ),
-            "a pre-kickoff acknowledgement must not complete a singleton"
+            "a future-dated pre-kickoff acknowledgement must not complete a singleton"
         );
-        seed_msg(&requester, 1_700_000_003_000, "bbbbbb", "h.interviewer");
+        // Filename novelty establishes after-kickoff causality; `>=` keeps a valid same-ms reply.
+        seed_msg(&requester, kickoff_ts, "bbbbbb", "h.interviewer");
         assert!(
             wait_done(
                 root.path(),
                 Some(&routes),
                 "h.interviewer",
                 "requester",
                 &[],
-                Some(1_700_000_002_000),
+                Some(kickoff_ts),
+                Some(&before),
                 Duration::from_secs(1),
                 noop,
             ),
-            "a post-kickoff singleton confirmation should complete promptly"
+            "a newly appearing same-ms singleton confirmation should complete promptly"
         );
     }
 

tests/eval_run_e2e.rs

@@ -236,7 +236,7 @@ eval {
       exec "test -f $CATALOG/sup/roots-ok && test -f $CATALOG/worker/roots-ok"
     }
     judge "render materialized before launch" {
-      exec "test \"$(cat $CATALOG/sup/materialized.txt)\" = rendered"
+      exec "test -f $CATALOG/sup/render-seen-at-process-start"
     }
     judge "custom main id survived supervision" {
       exec "test -f $CATALOG/worker/restarted-once"
@@ -306,6 +306,8 @@ exec sleep 60
     std::fs::write(
         fixture.join("scripts/sup.sh"),
         r#"#!/bin/sh
+test "$(cat "$CATALOG/sup/materialized.txt")" = rendered || exit 41
+: > "$CATALOG/sup/render-seen-at-process-start"
 echo "canonical supervisor main"
 test "$CATALOG" = "$ST_ROOT" &&
   test "$PTY_ROOT" = "$CATALOG/pty" &&
@@ -726,6 +728,20 @@ fn canonical_agents_fail_closed_matrix_is_pre_spawn_and_non_vacuous() {
                 ),
             ],
         ),
+        (
+            "duplicate runtime task id",
+            "evalhost.one",
+            vec![
+                (
+                    "one",
+                    r#"agent "one" { identity "one"; host "evalhost"; pty "agent" { id "shared"; command "touch \"$CATALOG/SPAWNED\"; sleep 60" } }"#,
+                ),
+                (
+                    "two",
+                    r#"agent "two" { identity "two"; host "evalhost"; pty "agent" { id "two-main"; command "sleep 60" }; exec "poison" { id "shared"; command "touch \"$CATALOG/SPAWNED\"; sleep 60" } }"#,
+                ),
+            ],
+        ),
         (
             "retired Agent Spec",
             "evalhost.worker",