OT-RFC-38 LU-7/8/9/10 — catchup + verify + attestation + devnet harness

CHANGELOG.md

@@ -4,6 +4,24 @@ All notable changes to the DKG V9 node are documented here. The format is based
 
 ## [Unreleased]
 
+**OT-RFC-38 Phase A — Edge curator can publish curated CGs to Verified Memory; members verify post-decrypt; outsiders verify via attestation tokens.** Builds on the SPEC_CG_MEMORY_MODEL surface below by closing the curated-CG publish path end-to-end. Edge curators (no on-chain `identityId`) now create curated CGs, share private SWM with named members, publish to VM with `attributionId=0` (no-attribution mode the V10 contract already supports), and ship every member-attested verification artifact a downstream verifier needs. RFC: [`docs/specs/SPEC_CG_HOSTING_MEMBERSHIP.md`](./docs/specs/SPEC_CG_HOSTING_MEMBERSHIP.md) §1.1 and §7.1.1.
+
+### Added — OT-RFC-38 Phase A
+
+- **LU-5: curated payload AEAD wrap + identity-less publish + UI honesty** (`packages/core/src/proto/publish-intent.ts`, `packages/core/src/crypto/v10-publish-payload.ts`, `packages/publisher/src/storage-ack-handler.ts`, `packages/publisher/src/ack-collector.ts`, `packages/publisher/src/dkg-publisher.ts`, `packages/agent/src/dkg-agent.ts`, `packages/cli/src/publisher-runner.ts`, `packages/node-ui/src/ui/views/MemoryLayerView.tsx`, `packages/node-ui/src/ui/views/project/components.tsx`): adds `isEncryptedPayload` (field 14) to `PublishIntent` so cores can ACK curated payloads they cannot decrypt — they verify `chunkDigest` + `byteSize` against the persisted ciphertext, never N-Quad-parse. New `v10-publish-payload.ts` is an AES-256-GCM AEAD helper keyed off the CG chain key (HKDF). Publisher's `storage-ack-handler` branches on `isEncryptedPayload` to persist opaque ciphertext under `(contextGraphId, batchId)` and signs the existing V10 digest verbatim (no contract change). `DKGAgent._resolveEncryptInlinePayload` wires the encryption hook on the agent side for curated CGs. **Critical fix**: dropped the `publisherNodeIdentityId > 0n` gate from `DKGPublisher` — edge agents now reach the on-chain publish path in no-attribution mode (`attributionId=0n`), where the V10 contract accepts the publish as an authority delegation against the curator's wallet signature. UI surfaces the result honestly: green "Published to Verified Memory" with full `txHash` + block number only when the daemon reports `status === 'confirmed'` AND a `txHash` is present, red "NOT published to Verified Memory" with the on-chain failure reason otherwise. This closes the §1.1 bug surfaced in the RFC.
+- **LU-7: SWMCatchupRequest endpoint** (`packages/cli/src/daemon/routes/memory.ts` `POST /api/shared-memory/catchup`): the daemon route lifts the existing per-peer sync substrate (`PROTOCOL_SYNC`) into a caller-initiated catchup primitive. Body `{ contextGraphId, peerId?, includeDurable?, perPeerBudgetMs? }`. Public CGs (`accessPolicy=0`) accept anonymous catchup; curated CGs (`accessPolicy=1`) run the responder's existing `authorizePrivateSyncRequest` against the requester's signed envelope (member → allowed; outsider → denied). When `peerId` is omitted the route fans out to every connected peer in parallel, bounded by `perPeerBudgetMs`. Returns per-peer triple-insertion counts so callers can tell who served the request.
+- **LU-8: member post-decrypt root recompute + BatchRejected gossip** (`packages/agent/src/swm/verify-batch.ts`, `packages/cli/src/daemon/routes/memory.ts` `POST /api/shared-memory/{verify-batch,report-batch-rejection}`, `packages/agent/test/verify-batch.test.ts`): pure recompute helper hashes the member's decrypted plaintext quads using V10's `computeFlatKCRootV10` + `computeFlatKCMerkleLeafCountV10` and compares against the on-chain anchor. Mismatch returns `{ ok: false, reason: 'root-mismatch', actualRoot, expectedRoot, leafCount }`. The `verify-batch` daemon route lets callers POST `{ contextGraphId, expectedMerkleRoot, quads?, privateRoots? }`; when `quads` is omitted the route reconstructs from the local SWM or post-publish CG data graph. The `report-batch-rejection` route ships a structured `BatchRejection` record through SWM gossip via `agent.share()` so other members can sanity-check and refetch from a different host.
+- **LU-9: member-attestation token mint + outsider verification** (`packages/agent/src/swm/member-attestation.ts`, `packages/cli/src/daemon/routes/memory.ts` `POST /api/attestation/{mint,verify}`, `packages/agent/test/member-attestation.test.ts`): a member signs an envelope binding `(chainId, kavAddress, contextGraphId, batchId, merkleRoot, plaintextLeafHash, attesterAddress, attestedAt)` with `keccak256(abi.encodePacked(...))` + EIP-191 secp256k1 (matches V10 chain-side signature layout — outsiders can hand-verify against `ContextGraphStorage`). The daemon `mint` route signs via the node's chain adapter; the `verify` route runs four checks: signature recovery, signer-matches-attester, optional `candidateLeaf` rehash against `plaintextLeafHash`, optional async `membershipResolver` chain hook for "was this attester a CG member at `attestedAt`". Returns structured `{ ok, recoveredSigner, signerMatchesAttester, leafCheck, membership, reason? }` so consumers can decide based on which checks passed.
+- **`enumerate-cg-hosts` helper** (`packages/agent/src/swm/enumerate-cg-hosts.ts`, `packages/agent/test/enumerate-cg-hosts.test.ts`): library helper distinct from `enumerate-cg-members`. Returns the dialable peer set the LU-7 catchup primitive will try in turn (Phase A: all connected peers minus self; Phase B will refine to a sharding-table-eligible subset once shard count > 1).
+- **Devnet integration tests** (`scripts/devnet-test-rfc38-*.sh`): 11 standalone end-to-end scenarios, all driven through the daemon HTTP API. `devnet-test-rfc38-all.sh` runs the full suite. Covers LU-5 (curated + public), LU-7, LU-8, LU-9, LU-10 (public-CG regression sweep), `e2e` (LU-5→LU-7→LU-8→LU-9 composed), `cross-cg` isolation (member of CG-A cannot decrypt CG-B), `multi-member` (3 distinct member wallets cross-verify the same batch + cross-verify each other's attestations), `scale` (50 triples / 25 KAs single batch), `late-joiner` (member-from-curator + member-from-member-with-curator-offline + the documented LU-6 cores-only gap as a passing fail-soft assertion). Re-runnable; every CG id is timestamp-suffixed.
+- **`./scripts/devnet.sh restart-node N`** + UI proxy `DEVNET_UI_NODE` env (`scripts/devnet.sh`, `packages/node-ui/vite.config.ts`): operator surfaces for restarting one node without wiping state, and pointing the Vite dev-server proxy at any devnet node (not hardcoded to node 1). Lets the user test the edge-publish path from `http://localhost:5173/ui/` against node 5 instead of needing custom curl.
+
+### Deferred (Phase A sub-task, tracked for follow-up)
+
+- **LU-6 substrate hosting on cores**: cores do not yet subscribe to the curated-CG SWM gossip topic via the sharding-table assignment (RFC §5.1 + §5.1.1 pre-registration staging). Today's catchup model works when the curator OR any other current member is online; if every member is offline, a late joiner's catchup against cores returns 0 triples cleanly (no crash). `devnet-test-rfc38-late-joiner.sh` SCENARIO C asserts this fail-soft shape. Full LU-6 lands the encrypted SWM substrate (the `SwmSenderKey` two-layer Sender Keys construction already in `packages/core/src/crypto/swm-sender-key.ts` but not yet wired to the workspace-gossip topic) plus the TTL + byte-cap staging policies in §5.1.1. Path forward documented in `docs/specs/SPEC_CG_HOSTING_MEMBERSHIP.md` §7.1.1.
+
+---
+
 **Context Graph memory model — edge agents can create curated CGs**: the on-chain Context Graph surface no longer accepts per-CG hosting committees or per-CG ACK quorums; hosting and ACK quorum are network-level concerns (sharding table + `parametersStorage.minimumRequiredSignatures()`). This unblocks edge-node agents who have no on-chain `identityId` from registering invite-only / curators-only CGs — previously the agent SDK threw at register time when `ensureIdentity()` returned `0n`. RFC: [`docs/specs/SPEC_CG_MEMORY_MODEL.md`](./docs/specs/SPEC_CG_MEMORY_MODEL.md). Wire-format break end-to-end (contracts + ABIs + SDK + daemon + CLI + MCP + UI all rev together); no compatibility shim — every package upgrades in lockstep.
 
 ### Changed — Context Graph memory model

docs/specs/SPEC_CG_HOSTING_MEMBERSHIP.md

@@ -825,6 +825,27 @@ Cores hosting may face abuse vectors:
 3. Member detects a malicious-publisher batch: publisher commits a root that doesn't match the SWM ciphertext members can decrypt. Member rejects the batch, alerts via SWM gossip, and the rejection propagates.
 4. Outsider with `(assertion, attestation)` from a member runs the attestation-verification flow against a target batch; verification succeeds for a real attestation, fails for a tampered one or one signed by a wallet that wasn't a CG member at the attested epoch.
 
+#### 7.1.1 Phase A — implementation status (as-shipped)
+
+The Phase A milestones above are mostly landed. Devnet validation lives in `scripts/devnet-test-rfc38-*.sh`; run `scripts/devnet-test-rfc38-all.sh` against a fresh 6-node devnet (4 cores + 2 edges) to exercise the full suite end-to-end. Current scope on this branch:
+
+| Sub-task | Source surface | Devnet test | Status |
+|---|---|---|---|
+| LU-5: edge curator → curated CG → VM publish (the §1.1 unblocker — `isEncryptedPayload` PublishIntent, AEAD wrap, no-attribution V10 publish) | `packages/core/src/crypto/v10-publish-payload.ts`, `packages/publisher/src/{storage-ack-handler,dkg-publisher}.ts`, `packages/agent/src/dkg-agent.ts` (`_resolveEncryptInlinePayload`) | `devnet-test-rfc38-lu5.sh` + `lu5-public.sh` | ✅ landed |
+| LU-7: `SWMCatchupRequest` catchup endpoint (anon for public CGs, member-attested for curated; outsider denial) | `packages/cli/src/daemon/routes/memory.ts` (`POST /api/shared-memory/catchup`) + the existing `PROTOCOL_SYNC` substrate | `devnet-test-rfc38-lu7.sh` | ✅ landed |
+| LU-8: member post-decrypt root recompute + `BatchRejected` SWM gossip | `packages/agent/src/swm/verify-batch.ts` + `POST /api/shared-memory/{verify-batch,report-batch-rejection}` | `devnet-test-rfc38-lu8.sh` | ✅ landed |
+| LU-9: member-attestation token mint + outsider verification (with optional `membershipResolver` chain hook) | `packages/agent/src/swm/member-attestation.ts` + `POST /api/attestation/{mint,verify}` | `devnet-test-rfc38-lu9.sh` | ✅ landed |
+| LU-10: public-CG regression sweep (publish + anonymous catchup + verify-batch + attestation, all on a public CG) | reuses LU-5/7/8/9 surfaces with `accessPolicy: 0` | `devnet-test-rfc38-lu10.sh` | ✅ landed |
+| Cross-CG isolation, multi-member (3-way), scale (50 triples / 25 KAs), late-joiner (member-from-member with curator offline) | scenario coverage on top of the landed surfaces | `devnet-test-rfc38-{cross-cg,multi-member,scale,late-joiner}.sh` | ✅ landed |
+| LU-6: sharding-table-driven SWM substrate subscription on cores + pre-registration staging (TTL, byte caps, ciphertext fanout to cores) so cores can serve catchup when the curator AND all live members are offline | (deferred) | `devnet-test-rfc38-late-joiner.sh` SCENARIO C documents the gap with a passing fail-soft assertion (cores-only catchup returns 0 triples cleanly, no crash) | ⚠️ deferred (see below) |
+
+**What "deferred LU-6" means in practice on this branch:**
+
+- A new member joining when the curator OR any other current member is online → catches up the full SWM history via `POST /api/shared-memory/catchup` against that peer. ✅ works.
+- A new member joining when the curator AND all current members are offline → catchup against cores returns 0 triples. The endpoint shape is correct (`peersAttempted > 0`, `totalInsertedTriples == 0`, no crash); the data simply isn't there because today's cores don't subscribe to curated CG SWM gossip topics outside the member allowlist. ⚠️ gap.
+
+This gap is acceptable for the Phase A user-visible surface (the §1.1 bug was about *publishing*, not about a specific late-joiner pattern), but is the next thing to land for the full "scenarios 1–4 of §2.4" promise to be honest. The substrate-subscription work itself is non-trivial: it touches the `SharedMemoryHandler` apply path (currently signature-checks the publisher and applies plaintext quads; needs a parallel "store opaque ciphertext under sharding-table assignment" path) and the SWM gossip wire format (Phase B in §7.2 will move it to AEAD per §5.2; Phase A could ship a transitional "cores subscribe but only persist for members" mode if needed sooner).
+
 ### 7.2 Phase B — Explicit key lifecycle + monetization model β
 
 **Scope**: formalise the curator's key-distribution lifecycle as explicit `KeyGrant` / `KeyRotate` messages (§5.5) and add the `PaidAccessGrant` protocol for per-assertion monetization (§5.7).

packages/agent/src/index.ts

@@ -70,6 +70,24 @@ export {
   type PolicyApprovalBinding,
 } from './ccl-policy.js';
 export { DKGAgent } from './dkg-agent.js';
+export {
+  verifyBatch,
+  buildBatchRejectionRecord,
+  type VerifyBatchInput,
+  type VerifyBatchResult,
+  type BatchRejectionRecord,
+} from './swm/verify-batch.js';
+export { createCGHostEnumerator, type CGHostEnumerator, type CGHostEnumeratorDeps } from './swm/enumerate-cg-hosts.js';
+export {
+  mintMemberAttestation,
+  verifyMemberAttestation,
+  computeAttestationDigest,
+  type MemberAttestation,
+  type MemberAttestationPayload,
+  type MintMemberAttestationInput,
+  type VerifyMemberAttestationInput,
+  type VerifyMemberAttestationResult,
+} from './swm/member-attestation.js';
 export {
   ContextGraphNotFoundError,
   InvalidContentError,

packages/agent/src/swm/enumerate-cg-hosts.ts

@@ -0,0 +1,78 @@
+/**
+ * OT-RFC-38 LU-6 (minimal) — CG hosting-peer enumeration.
+ *
+ * Distinct from `enumerate-cg-members` (which returns peers eligible
+ * to *decrypt* the CG content):
+ *
+ *   - **members** receive a copy of the chain key and can decrypt
+ *     curated CG payloads. Resolved via the CG allowlist or topic
+ *     subscribers.
+ *   - **hosts** receive the ciphertext (curated) or plaintext (public)
+ *     bytes and store them on behalf of the network, without the
+ *     ability to decrypt. Per SPEC_CG_HOSTING_MEMBERSHIP §3, hosting
+ *     and membership are orthogonal node roles.
+ *
+ * Phase A simplification (per RFC §4.6): the sharding table is single-
+ * shard — every sharding-table member hosts every CG. The on-chain
+ * `nodeId` field is a random 32-byte token, not a libp2p PeerID, so the
+ * Phase A enumerator does NOT consult chain state directly. It instead
+ * returns the currently-connected libp2p peers — they are the dialable
+ * candidates the LU-7 catchup protocol will try in turn, with each peer
+ * able to decline if it doesn't actually hold the CG (e.g. because it's
+ * an edge node, not a core).
+ *
+ * Future shard-aware enumeration (Phase B) will:
+ *   1. Maintain a local (identityId → peerId) map from observed peer
+ *      announcements (libp2p identify protocol carries peer addrs, and
+ *      `getPeerDiagnostics()` already surfaces a peer's identityId).
+ *   2. Cross-reference against `ShardingTable.getShardingTable()` to
+ *      filter to actual sharding-table members.
+ *   3. Apply the per-CG sharding function once shards >1 ship.
+ *
+ * Cache-free by design: callers (LU-7 catchup) invoke this on each
+ * SWMCatchupRequest send, so the result must reflect the current
+ * connection state. `getConnectedPeers()` is already an in-memory
+ * libp2p lookup; no I/O budget to amortise.
+ */
+
+export interface CGHostEnumeratorDeps {
+  /**
+   * Returns the currently-connected libp2p peer IDs (strings in the
+   * canonical base58/base32 multiaddr form). Self is included or not
+   * depending on the caller's wiring; this enumerator strips self
+   * regardless.
+   */
+  getConnectedPeers: () => string[];
+  /** Lazy accessor for our own peer ID (excluded from results). */
+  getSelfPeerId: () => string;
+}
+
+export interface CGHostEnumerator {
+  /**
+   * Resolve the candidate hosting-peer set for {@link cgId}. Returns
+   * all currently-connected peers (minus self) in Phase A; Phase B
+   * will filter to the sharding-table-eligible subset for the CG.
+   *
+   * Non-async because the underlying source (libp2p connections list)
+   * is in-memory; kept as a Promise-returning signature anyway so
+   * Phase B can layer a chain query in without breaking callers.
+   */
+  enumerate(cgId: string): Promise<string[]>;
+}
+
+export function createCGHostEnumerator(deps: CGHostEnumeratorDeps): CGHostEnumerator {
+  return {
+    async enumerate(_cgId: string): Promise<string[]> {
+      const self = deps.getSelfPeerId();
+      const seen = new Set<string>();
+      const out: string[] = [];
+      for (const peer of deps.getConnectedPeers()) {
+        if (peer === self) continue;
+        if (seen.has(peer)) continue;
+        seen.add(peer);
+        out.push(peer);
+      }
+      return out;
+    },
+  };
+}