Add optional turn-boundary + fresh-idle DING without harness parsing in core

[myobie]

Goal

Preserve today's aggressive plain DING as the unconfigured default, while an optional generic delivery policy uses two independent paths: lifecycle hooks inject unread durable-inbox work into the next harness context at a turn boundary without PTY input, and a fresh generic idle lease authorizes ordinary PTY DING input. Core st2 must remain harness-neutral; Codex, Claude, terminal-renderer, and provider vocabulary belongs in provider adapters or compile-agent output.

Semantic lock

• With no rich adapter configured, initial DING keeps the current aggressive combined terminal delivery. This issue does not silently change the fleet default.
• Once ding {} selects the rich generic policy, core may write to the PTY only after an explicit fresh idle lease for the same live runtime incarnation and launch generation.
• active, child-command-running, unknown, stale, unreachable, malformed, timeout, adapter error, incarnation mismatch, generation change, compaction, clear, and crash all hold the message and send no PTY input.
• The durable inbox is authority. Held work remains durable/FIFO and retries only after a new qualifying idle lease.
• Harness lifecycle hooks are a first-class turn-boundary path: after a turn they check unread inbox work and inject it into the next harness context without any PTY input. They are not a special mid-turn interruption protocol.
• A supervisor may still use explicit PTY input for an exceptional intervention; that is outside ordinary automatic DING delivery.
• A fresh idle lease is authorization to attempt PTY delivery, not a harness receipt. Existing staged ownership and positive post-submit receipt semantics remain fail-closed until they are replaced by an equally explicit provider receipt.

Current harness/terminal-text inventory on c6846f6

Every branch below must leave harness-neutral DING core or be recast as a generic PTY/event contract:

1. src/ding/composer.rs parses ANSI/OSC/CSI terminal text, assumes the current Codex/Claude 80-column soft-wrap geometry, recognizes a › 1. choice-menu shape, and selects the lowest renderer-shaped composer on a screen.
2. src/ding/harness/mod.rs hard-codes the Codex/Claude registry, carries raw-versus-ANSI-stripped screen representations, recognizes submitted-prompt glyphs, and matches the Messages to be submitted after next tool call queue heading.
3. src/ding/harness/codex.rs owns exact ANSI composer markers, ›, the gpt-* · /... footer, 80/2/2 wrap and padding constants, submitted/queued receipt pixels, and active/modal literals. It also deliberately carries a Claude-shaped inherited progress-line heuristic.
4. src/ding/harness/claude.rs owns full-width separator parsing, ❯/nonbreaking-space prompt forms, ⏵⏵ ... permissions on footer proof, Try "..." placeholders, gerund-plus-ellipsis progress lines, submitted-prompt pixels, and active/modal literals.
5. src/ding/mod.rs invokes pty peek, routes pre/post-submit screen classification, adopts exact staged composer text, permits a bare Return after two text observations, and uses fixed bracketed-paste/delay/Return terminal transactions. The PTY transport can remain generic; screen grammar and harness receipt parsing cannot.
6. docs/vrs/01-ding/{requirements,spec.md,01-claude,02-codex} normatively defines rendered-screen heuristics and per-harness grammars. The new contract must relocate those claims to provider adapters and keep only generic event/delivery guarantees in core VRS.
7. src/hooks.rs embeds named Codex and Claude hook files and identifies Codex by command/argv basename. src/compile_agent.rs branches on claude|codex, emits provider commands/settings, and installs different hook event sets. These are provider surfaces; their emitted event protocol may be used, but DING core must not branch on those names.
8. The shipped hooks themselves encode provider APIs: Codex SessionStart/PreCompact/Stop JSON and Claude SessionStart/PreCompact/StopFailure exit/stderr semantics. Codex Stop currently supplies the post-turn inbox path; equal Claude behavior must be delivered through the Claude adapter rather than a Claude string branch in core.
9. Current catalog evidence is deliberately non-normative but defines the migration seam: all nine active declared Codex agents render-installed SessionStart, PreCompact, and Stop hooks; retired comparison agents do not; no active Claude agent currently proves equivalent turn-boundary coverage. Rich policy configuration alone must not pretend those hooks are automatic.

Authoritative design boundary

Do not add a special mid-turn interruption protocol. The two rich paths have separate owners:

1. Turn boundary: provider lifecycle hooks read the durable inbox and inject unread work into the next model context without PTY input.
2. Idle PTY: a future generic PTY API supplies a fresh idle lease; core DING may then use its ordinary PTY transport.

The earlier alternatives resolve as follows:

• A built-in provider-named KDL enum is rejected because it recreates a Codex/Claude registry and invites command parsing in validation or supervision.
• The accepted opt-in seam is an opaque external adapter command. Provider adapters may parse provider state; core receives only the generic protocol below.
• Turn-boundary injection and PTY idleness remain distinct event kinds. A provider hook owns next-context translation; the future PTY/activity source owns the idle lease.
• compile-agent may generate both the adapter binding and explicit provider hook render entries. Hand-authored KDL may declare both. st2 up invents neither.

Minimal KDL/interface proposal

Provisional Codex example; Claude selects its own opaque adapter executable. Exact syntax remains open pending PTY-owner assessment:

agent "worker" {
  ding {
    delivery "turn-boundary+fresh-idle"
    adapter {
      command "$ST_HOOKS/codex-ding-adapter"
    }
  }
}

The $ST_HOOKS text above is literal portable KDL. No machine-specific absolute path belongs in declarations, docs, code, or tests.

• ding.delivery is a generic policy token, not a provider selector.
• ding.adapter.command is an opaque executable contract. Core never infers a provider from its spelling and never appends provider-specific arguments.
• The declaration author or generator separately renders provider-native SessionStart/resume, PreCompact, and turn-boundary hooks. The adapter package owns the provider JSON, exit-code, stderr, and screen/state parsing needed by those entries.
• The same normalized adapter command is part of the DING sidecar launch fingerprint. A command or policy edit reports drift and does not restart an adopted sidecar automatically; an explicit replacement/relaunch is required.

Generic adapter protocol v1

Core consumes newline-delimited JSON events only. Every event carries:

{"v":1,"kind":"activity","session":"host.identity","incarnation":"opaque","generation":"sha256:...","sequence":7,"state":"idle","validForMs":250,"reason":"provider-opaque"}

Common rules:

• v, kind, session, incarnation, generation, and strictly increasing sequence are required.
• activity.state is one of idle, active, child, or unknown. Only exact idle for the current session/incarnation/generation creates a lease.
• The provider adapter also returns a narrow composer guard (empty, nonempty, or unknown). Screen parsing may prove only empty; it is never idle authority. Only idle + empty can proceed to the PTY fence.
• Core anchors validForMs at event receipt, caps it to a configured maximum, and invalidates it on any newer event, adapter EOF/error, timeout, incarnation/generation change, compaction, clear, or crash.
• Unknown versions, kinds, fields required by that kind, malformed JSON, stale/out-of-order sequences, and identity mismatch are fail-closed holds. Provider diagnostics stay opaque.

A provider turn-boundary hook that actually injected unread work into the next context emits a generic receipt through its adapter/event channel:

{"v":1,"kind":"turn-boundary","session":"host.identity","incarnation":"opaque","generation":"sha256:...","sequence":8,"messages":["<exact-inbox-filename>.md"]}

• messages contains exact currently unread filenames included in that next context; timestamps or counts are insufficient.
• Core marks those notices hook-owned and sends no PTY input for them while they remain unread. Archive receipts clear ownership. Startup/resume hooks may surface the same durable backlog after a restart.
• The hook event does not trigger a model call and does not authorize Return. It records that the provider-native hook used an already-occurring next context.
• An adapter may obtain activity from a future generic PTY idle API or provider state, but core sees only this protocol. Provider-specific input/output translation stays outside core.

Compare-and-send safety option

Do not ask PTY to infer provider semantics such as submit versus newline. The preferred safe split is:

1. The provider adapter supplies a fresh exact-idle lease plus an empty-composer safety guard.
2. PTY supplies the exact session incarnation, launch generation, and a monotonic snapshot/I/O revision.
3. DING requests one conditional send containing the existing bounded transport and the exact expected tuple.
4. PTY atomically rechecks that tuple immediately before writing. A match writes the transaction once; a mismatch writes zero bytes and returns conflict.

Any key, paste, terminal output, resize/snapshot mutation, restart, clear, or other I/O revision change invalidates the compare. Adapter timeout/error, nonempty/unknown composer, or any tuple mismatch keeps inbox ownership and sends nothing. The conditional API needs only revision equality and byte transport; it contains no Codex/Claude strings and no submit/newline inference.

A candidate generic request/receipt shape is:

{"session":"host.identity","incarnation":"opaque","generation":"sha256:...","expectedRevision":42,"bytes":"<opaque bounded transport>"}
{"result":"sent","revision":43}
{"result":"conflict","actualRevision":44}

The wire syntax and exact revision domain remain provisional until the PTY owner assesses whether one monotonic snapshot/I/O revision can cover every input, output, restart, and resize path atomically.

Compatibility and installation boundary

• Omitted ding {} keeps the current aggressive delivery behavior exactly.
• Configured turn-boundary+fresh-idle requires adapter.command and never falls back to aggressive PTY input when the adapter, hook receipt, or lease evidence is missing. Unknown policy tokens and a missing/empty command fail validation generically.
• compile-agent may emit the opaque adapter binding plus provider-specific hooks for future generated agents. Hand-authored declarations and external adapters own equivalent explicit configuration.
• Policy configuration does not by itself prove hook installation. Provider fixture tests and external acceptance own that evidence; core validation does not infer it from a command basename.
• Existing agents migrate only when their declaration/renderer explicitly opts in. Retired agents receive no new installation work.

Documentation and source boundaries

• Document the currently shipped hook reality now in docs/vrs/01-ding/spec.md as a non-normative deployment note: active Codex coverage is render-installed and repeated per declaration, not an automatic default; current active Claude parity is unproven.
• Keep the normative generic policy, durable-inbox authority, fail-closed lease states, and compatibility behavior in docs/vrs/01-ding/{requirements.md,spec.md}.
• Move provider hook setup and any renderer receipt grammar to provider adapter/compile-agent documentation and tests; remove Codex/Claude screen grammar from core DING VRS when the replacement is implemented.
• Core source owns generic KDL lowering, adapter process supervision, JSONL validation, durable FIFO/hook ownership, the typed idle-lease consumer, PTY transport, and generic staged ownership. Provider source owns hook JSON/exit semantics, state parsing, and provider receipt translation.
• No PR docs(vrs): define Agent Spec field-change reconciliation #102 edit is part of this work.

Test boundaries

• Core unit tests use fixed generic session JSONL/lease fixtures for fresh idle, active, child, unknown, stale, unreachable, malformed, timeout, adapter error, incarnation mismatch, generation change, compaction, clear, and crash, plus empty/nonempty/unknown composer guards and exact/conflicting PTY revisions.
• Every configured non-idle, nonempty/unknown composer, or revision-conflict case proves zero PTY bytes and durable FIFO retention.
• Provider tests prove Codex and Claude lifecycle-hook installation plus next-context unread injection using provider-owned fixtures; core fixtures contain neither provider names nor renderer glyphs.
• External real-pane behavior remains owned by the evaluation harness.

A/B done condition

No benefit claim is made by this issue or an implementation PR until evals #57 compares the optional policy against no-adapter aggressive DING on equivalent Codex and Claude workloads and proves:

• equal end-to-end message delivery;
• strictly fewer PTY collisions/corrupted drafts or submissions; and
• no additional model calls.

Startup/resume backlog, during-turn arrival through the next turn-boundary hook, exact-idle PTY delivery, adapter loss, compaction, clear, crash, and explicit supervisor intervention are separate matrix rows. The optional policy remains opt-in until that evidence is accepted.

Model-free acceptance

• Fixed ASCII screen fixtures prove that plain unconfigured DING retains current aggressive behavior without any adapter process.
• Fixed session JSONL/event fixtures cover fresh idle, active, child, unknown, stale, unreachable, malformed, timeout, adapter error, incarnation mismatch, generation change, compaction, clear, and crash.
• Every non-idle configured case proves zero pty send calls and durable FIFO retention.
• Codex and Claude provider fixture suites prove their native lifecycle hooks inject unread work into the next context; core fixtures contain neither provider names nor renderer glyphs.
• Turn-boundary hook tests prove unread messages remain model-visible without PTY input even when rich live delivery held them.
• External behavior acceptance covers real Codex and Claude panes and is owned outside core st2 by the evaluation harness.

Related work

• PR docs(vrs): require quiet evented coordination #96 defines quiet evented coordination at the root VRS level.
• ding delivery mode has no spec-level lever, and a permanently-guarded (undeliverable) agent is invisible #7/DING never delivers to claude-code 2.1.220: idle_footer requires a hint the harness composes at runtime #26 document earlier provider-selection and renderer-heuristic failures; both are closed and do not own this event protocol.
• st2 ding reports delivery when Codex has not submitted or queued the notice #58 is superseded in mechanism by merged st2 ding reports delivery without confirming that Codex accepted the notice #92/PR fix(ding): require positive harness receipts #100's positive receipt work, but this issue preserves that fail-closed receipt boundary.
• PR docs(vrs): define Agent Spec field-change reconciliation #102 remains frozen and is unrelated to provider event mechanics.

[schickling]

Ownership clarification: #162 now owns all harness-specific behavior across st2, including DING screen grammars, provider activity classification, provider hook/event translation, composer and submission receipts, hook installation, and provider fixtures.

This issue remains open for the provider-neutral DING contract: durable FIFO ownership, generic event-envelope validation, fresh incarnation/generation fences, fail-closed holds, turn-boundary ownership, and exact PTY compare-and-send. It should consume driver evidence rather than define Codex/Claude behavior itself.

#111 is linked as a sub-issue of #162 so provider extraction and generic DING delivery can be implemented and accepted independently without duplicating ownership.