Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 10 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -499,13 +499,22 @@ Use the typed inventory instead of parsing `doctor` prose:
st2 tasks --catalog "$CATALOG" --host <host> --json
```

The `st2.task-inventory.v1` envelope joins the selected host's desired PTY and exec tasks to
The `st2.task-inventory.v2` envelope joins the selected host's desired PTY and exec tasks to
read-only runtime evidence. A complete observation exits zero. Catalog parse errors, declaration
drift during observation, duplicate runtime IDs, timeouts, malformed output, PID reuse, and
otherwise unprovable generations emit `complete: false` and exit non-zero. Missing runtime rows
become `absent` only when the corresponding backend observation is complete; uncertainty remains
`indeterminate`.

Every runtime has a tagged `resourceTarget`. A live Linux process reports
`{"type":"linuxCgroupV2","path":"/..."}` from its exact unified
`/proc/<pid>/cgroup` membership; a live Darwin process reports
`{"type":"darwinProcessTree","rootPid":...}` as a best-effort tree root. All
other cases report `{"type":"unavailable","reason":"..."}` from a bounded
reason set. These are per-observation locators, not identity: consumers keep
using `runtimeId` as the stable task key and rediscover the target on each
sample.

A PTY root positively absent at admission is not passed to `pty` and remains absent; an absent exec
state root likewise remains absent. If an admitted PTY root is concurrently removed, the result is
incomplete because its filesystem identity changed, but the external `pty list` implementation may
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
# Task resource targets are strict observations

Status: accepted

Accepted on 2026-09-05 for R23 fail-closed task inventory.

## Context

A downstream sampler needs to attribute host CPU and memory observations to a
running st2 task every 15 seconds. st2 already has the authoritative read-only
join between desired tasks and observed PTY or exec generations, but its
version-1 inventory exposes only process metadata. Linux sampling is truthful
at the task boundary only when it uses the live process's unified cgroup-v2
membership. Systemd unit names, scope names, and task naming conventions are
not observations of that membership. Darwin has no equivalent cgroup locator;
a process-tree walk rooted at the observed PID is necessarily best effort.

The locator can disappear or change while it is read. PID alone cannot reject
reuse, and retaining scope membership in st2 would add a second state plane
whose lifecycle could diverge from the backend and kernel.

## Options

| Option | Tradeoffs |
| --- | --- |
| Add a required strict tagged target to task-inventory v2 — selected | Makes every row self-describing, represents degradation explicitly, and lets the sampler remain downstream. |
| Derive Linux targets from systemd unit names | Rejected because direct/degraded launches and cgroup migration make the name an untruthful proxy for current membership. |
| Add optional or nullable locator fields | Rejected because absence cannot distinguish non-running, indeterminate, unsupported, raced, and degraded observations. |
| Persist a runtime-ID-to-scope registry in st2 | Rejected because it creates stale ownership and reconciliation obligations without improving kernel evidence. |
| Sample and export resource metrics from st2 | Rejected because cadence, retention, and metric transport belong to the downstream observer, not the task control plane. |

## Evidence and Argument

The focused experiment separated strict cgroup parsing from the
process-generation fence. One exact unified `0::` entry preserved its
slash-prefixed path, including spaces and colons, while absent, duplicate,
relative, repeated-separator, trailing-separator, and traversal-capable shapes
were rejected. Stable start-token observations admitted the candidate target.
A deterministic PTY regression replaced the PID generation between the
registry snapshot and start-token capture; the socket-backed stats row's
changed creation generation rejected that token. A two-live-task fixture
proved one unscoped stats snapshot serves the whole inventory. A token mismatch
before target read, a token change after it, and process exit each emitted only
a bounded unavailable result and never the candidate locator.

A live Darwin exec observation joined the current PID as
`darwinProcessTree.rootPid` without rewriting its retained generation record,
and a real PTY replacement changed generation identity while retaining the
same resource-target wire contract. The complete task-inventory unit and CLI
integration surfaces preserved absence, indeterminate, park, timeout,
declaration-drift, and read-only behavior.

This evidence favors a required tagged union over nullable fields: each sample
states both whether a locator exists and why it does not. Reading kernel
membership is also strictly stronger than reconstructing it from unit naming,
while keeping sampling downstream avoids adding lifecycle and retention state
to st2.

## Decision

`st2 tasks --json` advances to `st2.task-inventory.v2`. Every runtime object has
one `resourceTarget` internally tagged by `type`:

- `{"type":"linuxCgroupV2","path":"/<exact-unified-path>"}`;
- `{"type":"darwinProcessTree","rootPid":<pid>}`; or
- `{"type":"unavailable","reason":<bounded-reason>}`.

Unavailable reasons are exactly `notRunning`, `runtimeIndeterminate`,
`processUnavailable`, `generationChanged`, `cgroupV2Unavailable`, and
`unsupportedPlatform`.

On Linux, st2 reads exactly one unified `0::<path>` entry from the observed
process's `/proc/<pid>/cgroup`. Exec compares the target fence to the token in
its generation record. PTY captures each candidate daemon token after the
registry snapshot, then takes one unscoped, socket-backed `pty stats --json`
snapshot for the entire inventory. A candidate is admitted only when its named
live stats row reports the same daemon PID and creation generation and a second
token read is unchanged; that admitted token then fences the target read. A
missing, exited, or recycled process never publishes the candidate locator.
Darwin fences the current process in the same way before exposing its PID as a
best-effort tree root. All targets are rediscovered on each inventory command.

Runtime ID remains stable task identity. PID, creation time, generation ID,
cgroup path, root PID, unit, and incarnation remain observation locators only.

## Consequences

- A downstream 15-second sampler can consume one versioned inventory without
parsing scope names or asking st2 to sample metrics.
- Resource degradation is bounded and machine-readable while the inventory's
existing fail-closed catalog/runtime semantics remain intact.
- The Linux path is slash-prefixed and relative to the cgroup-v2 mount; `/`
denotes the mount root.
- st2 adds no persistent locator registry, metrics server, sampling cadence, or
resource history.
- Version-1 strict consumers must explicitly adopt schema version 2.
- Evidence is recorded in the [task resource target experiment](../.experiments/2026-09-05-task-resource-target-observation.md).
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
# Task resource target observation

Date: 2026-09-05
Fixture: deterministic process-generation seams plus a live Darwin process

## Question

Can task inventory expose a useful Linux cgroup-v2 or Darwin process-tree
locator without treating PID, systemd scope naming, or retained st2 state as
stable task identity, and without associating a locator after process exit or
PID-generation replacement?

## Method

The inventory resource-target boundary was split into two narrow operations:
strict parsing of the Linux `/proc/<pid>/cgroup` payload and a process-start
fence around target acquisition. Deterministic unit fixtures supplied stable,
changed, and disappearing start tokens around a candidate target. A PTY fixture
then reused the same numeric daemon PID with a different creation generation
between the registry snapshot and token capture; the live socket-backed stats
row tested that exact pre-fence window. A second fixture gave one inventory two
live candidates, recorded the single unscoped stats invocation, and changed
target proof from unavailable to available without changing either PTY
generation ID. Parser
fixtures covered a unified-only record, a hybrid record, root membership,
spaces and colons, duplicate unified entries, missing v2 entries, malformed
fields, carriage returns, repeated/trailing separators, and traversal
components.

A live-process fixture read the current process start token through st2's
platform process observer and asked the production target observer to associate
that exact generation. On Darwin it asserted the strict
`darwinProcessTree/rootPid` wire shape. CLI integration reused a live legacy
exec record to prove the target is joined to the correct runtime row without
rewriting exec state. Existing complete, missing-root, malformed, timeout,
park, declaration-drift, and generation-replacement inventory cases remained
on the same command surface.

Focused commands:

```sh
nix develop -c cargo test --lib task_inventory::tests
nix develop -c cargo test --lib run::tests::pty_stats_rejects_pid_reuse_between_registry_snapshot_and_start_token_capture
nix develop -c cargo test --lib run::tests::pty_resource_observation_uses_one_stats_snapshot_and_preserves_generation_id
nix develop -c cargo test --test task_inventory_cli
```

## Result

| Probe | Observation |
| --- | --- |
| Unified cgroup parser | Preserved one exact slash-prefixed `0::` path, including spaces and colons; rejected ambiguous, absent, relative, or traversal-capable shapes. |
| Stable start token | Published the candidate resource target. |
| Changed start token | Published only `unavailable/generationChanged`; the candidate locator was discarded. |
| PTY reuse before token capture | The socket-backed stats row observed the changed creation generation and returned `unavailable/generationChanged`; the replacement token was never admitted. |
| PTY stats fan-out and identity | Two live candidates used one `pty stats --json` invocation with no session argument; transient target-proof failure then success preserved each PTY generation ID. |
| Process disappeared during read | Published only `unavailable/processUnavailable`; the candidate locator was discarded. |
| Stable target read failed | Published the bounded degraded reason `unavailable/cgroupV2Unavailable`. |
| Live Darwin process | Published `{"type":"darwinProcessTree","rootPid":<current-pid>}` for the proven current generation. |
| Wire shape | Schema v2 requires one of three tagged variants; non-running and indeterminate runtime states also carry explicit unavailable reasons. |
| Existing inventory behavior | The same fail-closed envelope and non-zero behavior remains for unprovable catalog or runtime evidence; resource-target degradation alone remains a complete bounded observation. |

## Conclusion

A resource target can remain a pure observation. PTY captures the daemon token,
then uses one socket-backed stats snapshot to prove every candidate's live
daemon PID and creation generation before rereading and admitting the token;
the token subsequently fences the target read. This prevents publication
across the measured pre-fence reuse, exit, and generation-change races. Strict
cgroup parsing supplies the exact kernel locator without scope-name
parsing. Darwin exposes only the weaker process-tree root the platform can
support. Runtime ID remains the task identity and no persistent registry or
sampling loop is required. This supports [R23 fail-closed task inventory](../requirements.md)
and [decision 0017](../.decisions/0017-task-resource-targets-are-strict-observations.md).

## VRS Impact

- `requirements.md` requires a fenced, explicit resource target while keeping
inventory fail-closed and read-only.
- `ontology.md` distinguishes runtime resource targets and observation locators
from task identity.
- `spec.md` fixes schema version 2, exact tagged shapes, parser rules, bounded
unavailable reasons, and platform behavior.
- Decision 0017 records the strict-shape and no-registry/no-sampler boundary.
29 changes: 29 additions & 0 deletions docs/vrs/ontology.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,35 @@ A terminal-backed or terminal-free unit declared for an agent.
Authority: [Agent Spec task contract](https://github.com/compoundingtech/evals/blob/main/AGENT-SPEC.md#compact-and-explicit-tasks);
[`Task`](../../crates/agent-spec/src/spec.rs#L121-L148)

### runtime resource target

A short-lived locator from one task inventory observation that tells an
external sampler where the observed task's process resources can be read.
Linux uses the exact unified cgroup-v2 path from the live process's
`/proc/<pid>/cgroup`; Darwin uses a best-effort process-tree root PID. An
unavailable target carries one bounded reason instead of a guessed or nullable
locator.

This target is not task identity, ownership, a declaration, or a retained
registry entry. Runtime ID identifies the task; PID, process generation,
cgroup path, and process-tree root locate only the observation that produced
them.

Authority: [R23 fail-closed task inventory](requirements.md);
[`task_inventory`](../../src/task_inventory.rs)

### observation locator

Ephemeral evidence used to find one observed runtime generation, never to
identify the task across generations. PID, creation time, generation ID,
runtime resource target, cgroup path, unit name, and incarnation can all be
locators in their owning contexts; only a contract that explicitly exposes one
may be used by a consumer. In the task inventory, systemd unit and scope names
are deliberately not exposed as resource locators.

Authority: [R23 fail-closed task inventory](requirements.md);
[decision 0017](.decisions/0017-task-resource-targets-are-strict-observations.md)

### launch argv

The ordered, opaque OS-string sequence comprising a task program and each of
Expand Down
56 changes: 37 additions & 19 deletions docs/vrs/requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -283,29 +283,47 @@ accepted.
- **R23 Fail-closed task inventory:** One read-only machine command exposes
every desired local PTY and exec task by immutable agent ID, nullable current
bus address, task name, runtime id, kind, lifecycle, retirement, desired
state, runtime state, PID, creation time, and opaque runtime-generation id.
state, runtime state, PID, creation time, opaque runtime-generation id, and a
required resource-observation target. Runtime ID remains task identity. PID,
creation time, generation ID, cgroup path, and process-tree root are
observation locators only and may change between samples.
The resource target is a closed tagged union. A proved live Linux generation
exposes its exact unified cgroup-v2 path read from that process's
`/proc/<pid>/cgroup`; unit and scope names are not locators. A proved live
Darwin generation exposes its PID as a best-effort process-tree root. For
PTY, st2 captures the candidate daemon's kernel start token, takes one
socket-backed `pty stats --json` snapshot for all candidates, and admits the
token only when the named live session reports the same daemon PID and
creation generation and a second token read is unchanged. That token fences
target observation, as the recorded token does for exec, but its transient
availability never changes the PTY generation ID. A changed, recycled, or
exited process is never associated with the target.
Non-running, indeterminate, degraded, and unsupported observations expose a
bounded explicit unavailable reason rather than a nullable or guessed locator.
Resource-target unavailability is itself a complete observation and does not
turn a trustworthy task inventory into an incomplete one.
A proved non-routable retired subject has a null bus address without making
coverage incomplete. Unknown, duplicate, malformed, unreadable, timed-out,
PID-reused, or otherwise unprovable evidence is indeterminate and makes the
versioned envelope incomplete and the command unsuccessful; it is never
reported as absence. The command projects a
supervisor's known park fault and per-task recovery action alongside the
unmodified runtime observation. The action is structured executable argv
carrying the exact canonical catalog folder and selected host, so invoking it
never falls back to ambient catalog or host defaults. A believable park
remains complete, while an unbelievable park marker fails closed like other
unprovable evidence.
PID-reused, or otherwise unprovable runtime evidence is indeterminate and
makes the versioned envelope incomplete and the command unsuccessful; it is
never reported as absence. The command projects a supervisor's known park
fault and per-task recovery action alongside the unmodified runtime
observation. The action is structured executable argv carrying the exact
canonical catalog folder and selected host, so invoking it never falls back
to ambient catalog or host defaults. A believable park remains complete,
while an unbelievable park marker fails closed like other unprovable
evidence.
Observation detects semantic declaration drift across its runtime probe,
does not invoke a backend for a root positively absent at admission, and
performs no reconciliation, cleanup, lifecycle change, or state rewrite. An
admitted PTY root that changes
filesystem identity during the backend probe makes the observation
incomplete; the external backend may already have recreated a concurrently
removed registry. This diagnostic boundary is not transactionally serialized
with catalog or runtime writers and is not control-plane cutover authority.
It samples the durable catalog generation and incomplete marker around
discovery and runtime observation; any marker, malformed fence, or generation
change makes the envelope incomplete.
performs no reconciliation, cleanup, lifecycle change, state rewrite,
resource sampling, or persistent locator registration. An admitted PTY root
that changes filesystem identity during the backend probe makes the
observation incomplete; the external backend may already have recreated a
concurrently removed registry. This diagnostic boundary is not
transactionally serialized with catalog or runtime writers and is not
control-plane cutover authority. It samples the durable catalog generation
and incomplete marker around discovery and runtime observation; any marker,
malformed fence, or catalog-generation change makes the envelope incomplete.
- **R24 Immutable agent ID and mutable agent address:** Each logical agent
subject has one explicit, catalog-global, immutable agent ID. IDs are unique
across the live catalog and structural archive, across hosts and desired
Expand Down
Loading
Loading