Skip to content

Commit 3c778bb

Browse files
feat(resource): add atomic proposal publication core
agent-identity: dev3.direct.omp.2cshu64q agent-persona: generalist agent-supervisor: unavailable agent-tool: OMP agent-tool-version: 18.0.9 agent-runtime: OMP 18.0.9 tooling-profile: dotfiles@b607597
1 parent d072a03 commit 3c778bb

10 files changed

Lines changed: 1050 additions & 502 deletions

INVARIANTS.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,4 +35,5 @@ materialization, messaging, DING, or presence must preserve them.
3535
| **Tracked workspaces fail closed** | Materialization simulates content operations before writing and refuses a real change to any Git-tracked target. Byte-identical tracked, untracked, and non-Git targets retain useful behavior. | `tests/materialize.rs::every_content_directive_refuses_to_change_a_tracked_target_before_any_write`; `tests/materialize.rs::byte_identical_tracked_target_is_allowed_without_modification`; `tests/materialize.rs::untracked_and_non_git_targets_remain_materializable` |
3636
| **Native flat root** | Without an authored override, catalog tasks, eval messaging, shell helpers, and DING all use the catalog itself as `ST_ROOT`; no nested bus directory is synthesized. | `src/eval_run.rs::bus_root_expands_st_root_else_defaults`; `tests/eval_run_e2e.rs::st2_eval_runs_a_benign_folder_to_a_pass_verdict`; `tests/pty.rs` |
3737
| **Resource observation is state-first, atomic, and fenced** | ABI-3 periodic publication and demanded `Published` results reuse one bounded `Publication` payload and one host acceptance, digest, relevance, typed-fact, and catch-up core; the host never trusts a runtime digest or observation timestamp. Demand reaches only a resident runtime that explicitly declares `capability "demand"`. Every `Observe` carries a positive watermark and the exact owner, binding, and registration, and exactly one matching `Unchanged`, `Failed`, or `Published` atomic result closes it. One outstanding dispatch plus one latest trailing watermark coalesces bursts without losing in-flight arrivals. Backpressure retains queued demand, replacement fences stale output, restart and provider failure settle honestly, and client disconnect or wait expiry never cancels accepted work. | `tests/resource_profile_supervisor_e2e.rs::demand_observation_settlement_matrix_is_atomic_and_preserves_facts`; `tests/resource_profile_supervisor_e2e.rs::demand_observation_coalesces_and_fences_watermarks`; `tests/resource_profile_supervisor_e2e.rs::demand_observation_survives_restart_disconnect_and_denies_missing_capability`; `tests/resource_profile_supervisor_e2e.rs::observable_publication_reaches_builtin_resync_with_filter_catch_up_and_scope_isolation`; `tests/agent_resource.rs::refresh_cli_reports_exact_receipts_and_wait_expiry_keeps_the_request`; `src/resource_observe.rs::tests::receipt_evidence_shape_matches_atomic_results` |
38+
| **Atomic resource proposal publication** | Every changed resource publication is one host-owned compare-and-swap fenced by binding generation, state revision, and prior carrier digest. A persistent cross-process lock admits at most one proposal from the same prior. The content-derived proposal ID binds the accepted carrier digest and semantic outbox envelope; the durable intent becomes eligible only with the exact canonical carrier, then folds into one authoritative catch-up state. A pre-carrier crash exposes old state, a post-carrier crash catches up on restart, and retry after a lost acknowledgement returns the durable receipt without another transition. Ordinary reconciliation fails closed on out-of-band divergence after a committed intent; only an explicit generation-advance recovery may re-adopt the canonical carrier or its absence while invalidating the old intent and fence. | `src/resource_profile.rs::atomic_publication_fences_races_and_survives_crash_restarts`; `src/resource_profile.rs::generation_advance_explicitly_recovers_diverged_or_missing_carrier` |
3839
| **Proof references resolve** | Every qualified test named in this table exists in its named source file, so stale invariant claims fail the suite instead of silently surviving a refactor. | `tests/invariants.rs::qualified_proof_references_resolve` |

crates/st2-resource-protocol/src/lib.rs

Lines changed: 151 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -543,6 +543,130 @@ impl<'de> Deserialize<'de> for SnapshotDigest {
543543
}
544544
}
545545

546+
/// Content-derived identity of one host-validated publication proposal.
547+
///
548+
/// The host derives this after validating the publication. It is deliberately distinct from the
549+
/// snapshot digest: the same bytes proposed against a different binding, generation, revision,
550+
/// prior digest, selected-topic set, or ordered fact envelope are a different proposal.
551+
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash, Serialize, Deserialize)]
552+
#[serde(transparent)]
553+
pub struct ProposalId(SnapshotDigest);
554+
555+
impl ProposalId {
556+
pub fn of(bytes: &[u8]) -> Self {
557+
Self(SnapshotDigest::of(bytes))
558+
}
559+
560+
pub fn as_bytes(&self) -> &[u8; 32] {
561+
self.0.as_bytes()
562+
}
563+
}
564+
565+
impl fmt::Display for ProposalId {
566+
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
567+
fmt::Display::fmt(&self.0, formatter)
568+
}
569+
}
570+
571+
/// Compare-and-swap fence captured before a provider computes a publication.
572+
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
573+
#[serde(rename_all = "camelCase", deny_unknown_fields)]
574+
pub struct ProposalFence {
575+
generation: u64,
576+
revision: u64,
577+
#[serde(skip_serializing_if = "Option::is_none")]
578+
prior_digest: Option<SnapshotDigest>,
579+
}
580+
581+
impl ProposalFence {
582+
pub fn new(
583+
generation: u64,
584+
revision: u64,
585+
prior_digest: Option<SnapshotDigest>,
586+
) -> Self {
587+
Self {
588+
generation,
589+
revision,
590+
prior_digest,
591+
}
592+
}
593+
594+
pub fn generation(&self) -> u64 {
595+
self.generation
596+
}
597+
598+
pub fn revision(&self) -> u64 {
599+
self.revision
600+
}
601+
602+
pub fn prior_digest(&self) -> Option<SnapshotDigest> {
603+
self.prior_digest
604+
}
605+
}
606+
607+
/// Durable receipt for one changed publication.
608+
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
609+
#[serde(rename_all = "camelCase", deny_unknown_fields)]
610+
pub struct PublicationCommit {
611+
proposal_id: ProposalId,
612+
generation: u64,
613+
revision: u64,
614+
digest: SnapshotDigest,
615+
}
616+
617+
impl PublicationCommit {
618+
pub fn new(
619+
proposal_id: ProposalId,
620+
generation: u64,
621+
revision: u64,
622+
digest: SnapshotDigest,
623+
) -> Self {
624+
Self {
625+
proposal_id,
626+
generation,
627+
revision,
628+
digest,
629+
}
630+
}
631+
632+
pub fn proposal_id(&self) -> ProposalId {
633+
self.proposal_id
634+
}
635+
636+
pub fn generation(&self) -> u64 {
637+
self.generation
638+
}
639+
640+
pub fn revision(&self) -> u64 {
641+
self.revision
642+
}
643+
644+
pub fn digest(&self) -> SnapshotDigest {
645+
self.digest
646+
}
647+
}
648+
649+
/// Result of the host's one authoritative compare-and-swap transition.
650+
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
651+
pub enum ProposalCommit {
652+
Committed(PublicationCommit),
653+
AlreadyCommitted(PublicationCommit),
654+
Unchanged {
655+
generation: u64,
656+
revision: u64,
657+
digest: SnapshotDigest,
658+
},
659+
StaleGeneration {
660+
actual_generation: u64,
661+
actual_revision: u64,
662+
},
663+
StalePrior {
664+
actual_generation: u64,
665+
actual_revision: u64,
666+
actual_digest: Option<SnapshotDigest>,
667+
},
668+
}
669+
546670
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
547671
#[serde(
548672
tag = "type",
@@ -909,6 +1033,33 @@ mod tests {
9091033
}
9101034
}
9111035

1036+
#[test]
1037+
fn proposal_fence_and_commit_receipt_are_domain_typed() {
1038+
let prior = SnapshotDigest::of(b"prior");
1039+
let fence = ProposalFence::new(7, 11, Some(prior));
1040+
assert_eq!(
1041+
serde_json::to_value(fence).unwrap(),
1042+
json!({
1043+
"generation": 7,
1044+
"revision": 11,
1045+
"priorDigest": prior.to_string(),
1046+
})
1047+
);
1048+
assert_eq!(
1049+
serde_json::from_value::<ProposalFence>(serde_json::to_value(fence).unwrap()).unwrap(),
1050+
fence
1051+
);
1052+
1053+
let proposal_id = ProposalId::of(b"proposal identity");
1054+
let digest = SnapshotDigest::of(b"carrier");
1055+
let commit = PublicationCommit::new(proposal_id, 7, 12, digest);
1056+
assert_eq!(commit.proposal_id(), proposal_id);
1057+
assert_eq!(commit.generation(), 7);
1058+
assert_eq!(commit.revision(), 12);
1059+
assert_eq!(commit.digest(), digest);
1060+
assert_ne!(proposal_id.as_bytes(), digest.as_bytes());
1061+
}
1062+
9121063
#[test]
9131064
fn host_frames_have_exact_json_shape_and_newline() {
9141065
let register = HostMessage::Register {

docs/vrs/.decisions/0009-resource-profiles-use-a-feature-gated-wasm-boundary.md

Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -126,3 +126,96 @@ unprojected inputs, and load a relative module from the applied live catalog.
126126
missing newly introduced module.
127127
- Resolver observability and same-path module-cache invalidation remain explicit
128128
design questions; neither weakens the containment and feature-gating contract.
129+
130+
## Amendment — 2026-09-01 (Q39)
131+
132+
Johannes approved Q39 to make WASIp2 Component Model components the universal
133+
execution envelope for observable Resource providers. This amendment preserves
134+
the original decision's closed core-wasm resolver and replaces only the
135+
observable host-process mechanism described later by
136+
[decision 0014](./0014-resource-profiles-are-state-first-read-and-observe-capabilities.md).
137+
Decision 0014's state-first authority, demand semantics, typed `Publication`,
138+
semantic filtering, and catch-up model remain in force.
139+
140+
### Context
141+
142+
The closed resolver has one pure job: map an opaque Resource URI to a contained
143+
carrier path. It needs neither provider I/O nor the Component Model. Observable
144+
providers have a different job: call a remote or local provider, normalize its
145+
domain state, and propose a canonical publication. A catalog-trusted native
146+
process can perform that job, but it carries ambient host authority and creates
147+
a second long-lived lifecycle and JSON protocol beside Wasmtime.
148+
149+
Five disposable prototypes tested a narrower boundary with Wasmtime 48.0.1.
150+
Typed GitHub and PTY observations proved real domain I/O without exposing raw
151+
HTTP, caller-selected executable/arguments, environment, filesystem, or socket
152+
access. Fresh-Store cancellation tests ended with zero active tasks or
153+
capabilities. Verified compiled-code reuse kept a small provider's AOT disk-hit
154+
p50 at 0.482 ms and fresh Store plus instance observation p50 at 47.751 µs. An
155+
independent multiprocess oracle passed three 30-process runs covering
156+
one-winner compare-and-swap, stale generation, process-crash boundaries,
157+
acknowledgement loss, deterministic outbox identity, and restart catch-up.
158+
159+
### Decision
160+
161+
1. The core-wasm resolver ABI, its no-import sandbox, fresh resolution Store,
162+
path containment, registry behavior, feature gate, and transactional module
163+
ownership remain the only resolution mechanism.
164+
2. Every observable provider executes through one versioned WASIp2 Component
165+
Model envelope. st2 does not maintain a parallel native or host-process
166+
provider framework.
167+
3. A provider component may import only explicit provider-domain capabilities
168+
linked by the host. The host owns credentials, allowlists, limits,
169+
deadlines, cancellation, and redacted typed failures. No ambient WASI
170+
command, environment, clock, random, process, raw HTTP, filesystem, or
171+
socket authority is linked.
172+
4. Every descriptor call and observation receives a fresh Store and component
173+
instance. Engine, Linker, compiled Component, and compatible host-produced
174+
AOT bytes may be reused; Store and instance state may not be pooled or reset.
175+
5. The component returns `Unchanged`, a typed failure, or the existing
176+
`Publication` payload. It never writes the carrier, state record, receipt, or
177+
outbox. The host pairs a publication with
178+
`ProposalFence { generation, revision, prior_digest }`, validates it, and
179+
owns the only atomic commit.
180+
6. One durable transition makes the carrier, resulting digest and revision,
181+
freshness and catch-up state, and deterministic `PublicationIntent` visible
182+
together. A crash before publication leaves the old state. A crash after
183+
publication but before acknowledgement leaves the intent retryable;
184+
delivery remains separate and idempotent.
185+
7. WASIp3 production execution, Store pooling, generic exec/raw
186+
HTTP/filesystem/socket authority, a parallel native provider framework, and
187+
runtime WAC graphs are explicit non-goals. Each requires new evidence and a
188+
further accepted decision rather than an alternate dormant path.
189+
190+
### Evidence and argument
191+
192+
The durable record is the
193+
[WASIp2 component and atomic publication experiment](../07-resource-profile/.experiments/2026-09-01-wasip2-component-and-atomic-publication-prototypes.md).
194+
The disposable harnesses prove the boundary; their local source, result, cache,
195+
and state paths are not production interfaces or scaffolding.
196+
197+
The Component Model is selected for observable providers because its typed
198+
imports make authority reviewable at link time and its typed export preserves
199+
one observation result. Fresh Stores remove the need for an incomplete reset
200+
protocol across guest memory, resources, host state, traps, and cancellation.
201+
Host-only commit prevents a network- or command-capable guest from bypassing
202+
validation or racing publication against settlement. The deterministic durable
203+
outbox makes publication and delivery intent one crash-consistent fact without
204+
claiming exactly-once effects across an external sink.
205+
206+
### Consequences
207+
208+
- A profile may carry two wasm artifacts with deliberately disjoint jobs: a
209+
closed core module for resolution and, only when observable, a WASIp2
210+
component for provider observation.
211+
- Provider support requires a reviewed typed host capability; adding a generic
212+
authority under a domain-flavored name is non-conforming.
213+
- Domain acquisition state such as ETags, cursors, rate limits, and webhook
214+
repair cannot rely on guest Store lifetime. It belongs in bounded host-owned
215+
capability state or explicit durable provider state.
216+
- Compiled-code caching is an optimization, not an authority transfer. Any AOT
217+
deserialize path must authenticate exact host-produced bytes and the complete
218+
engine-compatibility key before crossing Wasmtime's unsafe boundary.
219+
- Decision 0014's host-process topology and newline-delimited JSON mechanism
220+
are superseded. Its publication, demand, delivery, and state-first semantics
221+
apply to direct component invocations.

docs/vrs/07-resource-profile/.experiments/2026-08-29-selector-and-runtime-protocol-prototype.md

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -65,3 +65,13 @@ Two complexity reductions survive the prototypes:
6565
- Remove `reconcile` and `shutdown` from the normative runtime protocol.
6666
- Specify EOF as runtime termination and supervisor lifecycle as the only shutdown authority.
6767
- Keep restart/backoff policy in existing task lifecycle machinery rather than the profile protocol.
68+
69+
## Subsequent evidence
70+
71+
Q39 retains the selector round-trip and directional-fencing findings as
72+
evidence, but supersedes the executable JSON-line runtime, including its
73+
process-EOF lifecycle, as the observable-provider mechanism. The
74+
[WASIp2 component and atomic publication prototypes](./2026-09-01-wasip2-component-and-atomic-publication-prototypes.md)
75+
proved a narrower universal envelope: one fresh Store and component invocation,
76+
domain-typed host capabilities, and a generation/revision/prior-digest proposal
77+
fence. No production path or protocol depends on the disposable runtime driver.

docs/vrs/07-resource-profile/.experiments/2026-08-29-smart-resource-lifecycle-prototype.md

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -61,3 +61,12 @@ The same lifecycle reducer is independent of shared versus per-binding runtime t
6161
- Reuse existing event supersession and DING transport for thin invalidations.
6262
- Keep provider cursors, webhook delivery identifiers, polling intervals, and observation repair inside the profile implementation.
6363
- Specify shared and per-binding runtimes behind one normalized host protocol; do not duplicate delivery state machines.
64+
65+
## Subsequent evidence
66+
67+
Q39 preserves the topology-independent catch-up reducer but supersedes the
68+
shared/per-binding host-process mechanism in the original VRS impact. All
69+
observable providers now use one fresh-Store WASIp2 component invocation.
70+
Provider cursors and repair state remain outside the guest Store in host-owned
71+
domain capability state or explicit durable provider state. The delivery
72+
reducer remains independent of how those typed capabilities acquire input.

0 commit comments

Comments
 (0)