Skip to content

Commit 6ac7297

Browse files
committed
Add production readiness e2e gate
1 parent e569ade commit 6ac7297

6 files changed

Lines changed: 314 additions & 0 deletions

File tree

.github/workflows/ci.yml

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -143,6 +143,18 @@ jobs:
143143
node --check ui/app.js
144144
node -e "JSON.parse(require('fs').readFileSync('src-tauri/tauri.conf.json')); JSON.parse(require('fs').readFileSync('src-tauri/capabilities/default.json'))"
145145
146+
production-readiness:
147+
name: Production readiness e2e gate
148+
runs-on: ubuntu-latest
149+
steps:
150+
- uses: actions/checkout@v5
151+
- uses: actions/setup-node@v5
152+
with:
153+
node-version: 20
154+
# Deterministic end-to-end gate: real feed server, temporary recorded Coral state,
155+
# proof_receipts ledger write, threads/runs HTTP APIs, and desk parse/config smoke.
156+
- run: npm run readiness:e2e
157+
146158
escrow:
147159
name: Escrow + arbiter contracts (Rust compile check)
148160
runs-on: ubuntu-latest

README.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -148,6 +148,7 @@ One `npm run` per example, from the repo root. **Each command installs that exam
148148
| Command | Runs | Needs |
149149
|---|---|---|
150150
| `npm run dev` | **the default demo** — the oracle proxy + UI, auto-settles on delivery | funded wallet + LLM key (renders demo data without) |
151+
| `npm run readiness:e2e` | **production-readiness e2e gate** - recorded Coral state -> feed -> ledger/proof receipts + desk smoke | no Docker/devnet/LLM key |
151152
| `npm run demo:coral` | the same oracle as a **multi-agent CoralOS round** (buyer + competing sellers) | Docker + `docker compose up -d coral` + a TxLINE token (`npm --prefix examples/txodds run mint`) |
152153
| `npm run marketplace` | **the full market** — a buyer + LLM sellers bidding in one session | Docker + `docker compose up -d coral` + `bash build-agents.sh` |
153154
| `npm run marketplace:web` | the market **visualizer** — live rounds, bids, settlement badges (:5173) | the marketplace feed running |
@@ -167,6 +168,11 @@ One `npm run` per example, from the repo root. **Each command installs that exam
167168
> **No-Docker ones** (`dev`, `agent-economy:quickstart`, `:web`, `marketplace:web`, `desk`) run straight
168169
> from Node; `desk:app` adds the Tauri/Rust shell.
169170
171+
Production-readiness status and the launch blockers live in
172+
[`docs/PRODUCTION_READINESS.md`](docs/PRODUCTION_READINESS.md). The automated gate is
173+
`npm run readiness:e2e`: it boots the real marketplace feed with a recorded Coral transcript,
174+
asserts feed/threads/runs/proof receipts over HTTP, and smokes the Agent Desk.
175+
170176
## Make it yours
171177

172178
1. **Change the service.** Open [`examples/txodds/agent/service.ts`](examples/txodds/agent/service.ts)

docs/PRODUCTION_READINESS.md

Lines changed: 123 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,123 @@
1+
# Production Readiness
2+
3+
This document applies the production-readiness checklist supplied during the audit to this repo. It is
4+
not a declaration that the kit is mainnet-production ready. The current production stance is:
5+
6+
- **Devnet-ready demo infrastructure:** yes.
7+
- **Deterministic CI/readiness gates for the core flows:** yes.
8+
- **Mainnet custody or real provider payments:** no.
9+
- **External launch with user funds:** blocked until the open risks below are closed.
10+
11+
## Critical Journeys
12+
13+
These are the user-visible flows that must stay healthy:
14+
15+
| Journey | Evidence |
16+
|---|---|
17+
| Single-agent TxODDS oracle | `npm run dev`; proxy persists runs with delivery hashes, escrow txs, and proof receipts. |
18+
| Coral marketplace | `npm run marketplace`; feed folds Coral session state into rounds, the run ledger, reputation, threads, and proof receipts. |
19+
| Marketplace visualizer | `examples/marketplace/web` unit tests and Playwright e2e against recorded Coral state. |
20+
| Payment rail procurement | `@pay/payment-runtime` tests plus `PAYMENT_*` folding into `proofReceipts` and `proof_receipts.json`. |
21+
| Agent desk | `npm run desk` for browser mode; CI parses UI JS and Tauri configs. |
22+
| Agent economy front doors | Typecheck/build smoke for autonomous, bridge, quickstart, and web dashboard. |
23+
| Escrow programs | CI `cargo check --workspace` on escrow + arbiter. |
24+
25+
## Automated Gate
26+
27+
Run the readiness e2e locally:
28+
29+
```sh
30+
npm run readiness:e2e
31+
```
32+
33+
What it proves, without Docker/devnet/LLM keys:
34+
35+
- Builds `@pay/agent-runtime`, because local `file:` dependents import its `dist`.
36+
- Starts the real marketplace feed against a temporary Coral extended-state fixture.
37+
- Asserts `/api/health`, `/api/feed`, `/api/threads`, and `/api/runs`.
38+
- Verifies a settled round with verifier pass and a Pay.sh proof receipt.
39+
- Verifies `proof_receipts.json` is written to the run ledger.
40+
- Checks the TxODDS Agent Desk static JS parses and the Tauri JSON configs are valid.
41+
42+
This complements, rather than replaces, the existing package tests and Playwright e2e.
43+
44+
## CI Coverage
45+
46+
The GitHub Actions workflow blocks PRs/pushes on:
47+
48+
- `packages/agent-runtime`: typecheck, tests.
49+
- `packages/harness-runtime`: runtime build, typecheck, tests.
50+
- `packages/payment-runtime`: runtime build, typecheck, tests.
51+
- `examples/txodds`: typecheck, tests.
52+
- `examples/marketplace`: launcher typecheck, feed typecheck/tests, web typecheck/tests, Playwright e2e.
53+
- `examples/agent-economy`: autonomous/quickstart/bridge typechecks and web production build smoke.
54+
- `coral-agents`: buyer/seller/verifier/broker/echo checks.
55+
- `examples/txodds-agent-desk`: static UI and config smoke.
56+
- `scripts/readiness-e2e.mjs`: production-readiness e2e gate.
57+
- `examples/txodds/escrow`: Rust `cargo check --workspace`.
58+
59+
## Production Checklist Status
60+
61+
| Area | Current answer | Status |
62+
|---|---|---|
63+
| Product and launch reality | Starter kit for paid agent services; default product is TxODDS oracle, not the final product. | Demo-ready |
64+
| Service levels | No public SLA/SLO promised. Devnet demos should degrade clearly when upstreams are down. | Needs launch owner |
65+
| Architecture | Documented in `README.md`, `CORAL.md`, package READMEs, and example READMEs. Coral coordinates; agents hold keys; Solana settles. | Covered for repo |
66+
| Source control | CI gates are present. Branch protection, CODEOWNERS, signed commits, and release tags are GitHub-org settings. | External setting |
67+
| Local development | `npm run setup`, `npm run dev`, example scripts, and `.env.example` cover local setup. Secrets are gitignored. | Covered |
68+
| Code quality | TypeScript strict packages, tests, protocol parsers, policy choke points, and devnet guard. | Covered for demo |
69+
| API contracts | Internal HTTP APIs are documented in READMEs. No OpenAPI contract yet. | Needs production API spec |
70+
| Data/storage | Run ledger is JSON-on-disk, replayable, and proof-receipt aware. No production DB/backup plan. | Demo-ready only |
71+
| Caching/performance | TxODDS proxy has short board caching. No capacity testing or cache SLO. | Needs launch work |
72+
| Async/events | Coral thread messages are replayable; watcher queue is simple in-memory demo state. | Demo-ready only |
73+
| Containers/runtime | Coral agents have Dockerfiles; desk Tauri shell is no-IPC. Resource limits/scanning not configured here. | Needs deployment platform |
74+
| Cloud/IaC | No cloud production deployment in this repo. | Out of repo |
75+
| CI/CD | Broad CI exists plus readiness e2e. No artifact signing/SBOM/release promotion. | CI covered, release hardening needed |
76+
| Security | Devnet guard, gitignored secrets, policy checks, verifier gate. No full threat model or pen test. | Needs security review |
77+
| Supply chain | Lockfiles are generated locally and ignored by repo policy. No SBOM/SLSA provenance. | Needs production policy |
78+
| Privacy/legal | No customer PII model in repo; LLM/provider data flow documented at a high level. | Needs product owner |
79+
| Reliability/resilience | Deterministic replay and degraded UI states exist. No paging, SLO alerts, or chaos testing. | Needs operations plan |
80+
| Observability | Ledger/transcripts/reputation are audit artifacts; no OpenTelemetry/log aggregation. | Needs production observability |
81+
| Testing | Unit/integration/e2e/readiness gates cover critical demo journeys. Load/security/restore tests not present. | Demo gate covered |
82+
| Deployment/rollback | No production deploy pipeline. Devnet examples are run manually. | Needs deployment plan |
83+
| Incident response | No on-call, severity model, or runbooks beyond troubleshooting docs. | Needs owner |
84+
| Backup/restore | Ledger is local disk; replay tested. No backup/restore drill. | Needs production storage plan |
85+
| Cost/FinOps | Devnet costs are negligible; LLM/provider costs bounded by env/policy only. | Needs launch budget |
86+
| Multi-tenancy | Not a multi-tenant app today. | Not applicable until hosted |
87+
| Admin/support | Agent Desk is an operator console, not a secured admin system. | Needs auth/RBAC before hosting |
88+
| Abuse/fraud | Spend caps and devnet guard exist; no bot/rate-limit/fraud controls. | Needs production controls |
89+
| Notifications/webhooks | Not a notification product. | Not applicable |
90+
| Analytics/reporting | Reputation and runs are product/audit metrics; no analytics pipeline. | Optional |
91+
| AI/LLM | Provider abstraction, fallbacks, verifier checks, and harness isolation exist. Prompt/version evals are not formalized. | Needs eval set |
92+
| Desktop/client | Desk has no IPC/fs/shell permissions beyond Tauri default; Tauri build still needs platform validation. | Needs signed build |
93+
| Vendor risk | CoralOS, Solana devnet/RPC, TxODDS, LLM providers, Pay.sh/x402 are critical vendors. No vendor SLA map. | Needs launch owner |
94+
| Docs/ownership | Broad docs exist. Concrete production owner/on-call owner is not assigned in repo. | Needs organization decision |
95+
96+
## Must-Answer Before Mainnet Or Real Users
97+
98+
- Who owns support, incident response, and launch approval?
99+
- What SLOs are promised for proxy, marketplace feed, desk, and settlement flows?
100+
- What is the production storage backend for the run ledger, and how is restore tested?
101+
- How are wallets, RPC keys, provider keys, and arbiter keys injected and rotated?
102+
- Which payment rails are live-provider backed, and how are provider failures handled?
103+
- What are the spend limits, kill switches, and operator approvals for real money?
104+
- How are admin/desk actions authenticated, authorized, and audited?
105+
- What is the threat model for prompt injection, payment proof replay, malicious sellers, and compromised agents?
106+
- What deployment target, rollback plan, artifact signing, and SBOM policy are used?
107+
- What telemetry pages a human, and what is the runbook?
108+
109+
## Open Production Blockers
110+
111+
1. Mainnet remains disabled by policy; `assertDevnet` must stay on unless a separate launch review says otherwise.
112+
2. Pay.sh/x402/USDC/embedded-wallet/payout rails are scaffolds except where explicitly marked live in `packages/payment-runtime/README.md`.
113+
3. No hosted auth/RBAC layer protects the desk or feed.
114+
4. No production database, backup, restore, or retention policy exists for run ledgers.
115+
5. No formal SLOs, incident response process, or on-call rotation exists.
116+
6. No SBOM/artifact-signing/supply-chain provenance gate exists.
117+
7. No load, abuse, or security test suite exists beyond functional checks.
118+
119+
## Operating Rule
120+
121+
The repo may claim **devnet demo readiness** when CI and `npm run readiness:e2e` are green. It may not
122+
claim **production readiness for real users or real funds** until the open production blockers have
123+
named owners, mitigations, and tested evidence.

package.json

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@
77
"engines": { "node": ">=20" },
88
"scripts": {
99
"setup": "npm install --prefix scripts && node scripts/setup.js",
10+
"readiness:e2e": "node scripts/readiness-e2e.mjs",
1011
"dev": "node scripts/txodds.js",
1112
"desk": "node scripts/run-example.js examples/txodds-agent-desk ui",
1213
"desk:app": "node scripts/run-example.js examples/txodds-agent-desk dev",

scripts/README.md

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,3 +35,14 @@ npm run dev # = node scripts/txodds.js
3535
```
3636

3737
Starts the data/escrow proxy (:8801) + the Oracle UI (:3020) and opens the browser. Devnet only.
38+
39+
## `readiness-e2e.mjs` - production-readiness gate
40+
41+
```sh
42+
npm run readiness:e2e
43+
```
44+
45+
Boots the real marketplace feed against a temporary recorded Coral session, asserts the critical
46+
HTTP surfaces (`/api/health`, `/api/feed`, `/api/threads`, `/api/runs`), verifies a
47+
`proof_receipts.json` ledger artifact, and smokes the TxODDS Agent Desk JS/config. No Docker, devnet,
48+
LLM key, or wallet needed.

scripts/readiness-e2e.mjs

Lines changed: 161 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,161 @@
1+
#!/usr/bin/env node
2+
/**
3+
* Production-readiness e2e gate.
4+
*
5+
* Deterministic, no devnet/Docker/LLM required:
6+
* 1. build the local runtime package the feed imports,
7+
* 2. start the real marketplace feed server against a temporary Coral extended-state fixture,
8+
* 3. assert health/feed/threads/runs/proof-receipts over HTTP,
9+
* 4. smoke the TxODDS Agent Desk static JS + Tauri JSON configs.
10+
*/
11+
import { spawn, spawnSync } from 'node:child_process'
12+
import { existsSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs'
13+
import { platform, tmpdir } from 'node:os'
14+
import { dirname, join } from 'node:path'
15+
import { fileURLToPath } from 'node:url'
16+
17+
const root = join(dirname(fileURLToPath(import.meta.url)), '..')
18+
const feedDir = join(root, 'examples', 'marketplace', 'feed')
19+
const runtimeDir = join(root, 'packages', 'agent-runtime')
20+
const deskDir = join(root, 'examples', 'txodds-agent-desk')
21+
const port = process.env.READINESS_PORT ?? String(4100 + Math.floor(Math.random() * 1000))
22+
const base = `http://localhost:${port}`
23+
const tmp = mkdtempSync(join(tmpdir(), 'pay-readiness-'))
24+
const fixturePath = join(tmp, 'coral-session-with-proof.json')
25+
const runsDir = join(tmp, 'runs')
26+
27+
function run(cwd, cmd, args) {
28+
const r = spawnSync(cmd, args, { cwd, shell: true, stdio: 'inherit' })
29+
if (r.status !== 0) throw new Error(`${cmd} ${args.join(' ')} failed in ${cwd}`)
30+
}
31+
32+
function stop(child) {
33+
if (child.exitCode != null || child.pid == null) return
34+
if (platform() === 'win32') {
35+
spawnSync('taskkill', ['/pid', String(child.pid), '/T', '/F'], { shell: true, stdio: 'ignore' })
36+
} else {
37+
try { process.kill(-child.pid, 'SIGTERM') } catch { child.kill('SIGTERM') }
38+
}
39+
}
40+
41+
function assert(condition, message) {
42+
if (!condition) throw new Error(message)
43+
}
44+
45+
function fixture() {
46+
const threadId = 'readiness-thread'
47+
const ts = (n) => `2026-07-06T12:00:${String(n).padStart(2, '0')}.000Z`
48+
const msg = (senderName, text, mentionNames, i) => ({ senderName, text, threadId, mentionNames, timestamp: ts(i) })
49+
const agents = ['buyer-agent', 'seller-premium', 'verifier-agent'].map((name) => ({
50+
name,
51+
status: { type: 'running' },
52+
registryAgentIdentifier: { name, version: '0.1.0', registrySourceId: { type: 'local' } },
53+
}))
54+
const messages = [
55+
msg('buyer-agent', 'WANT round=42 service=txline arg=edge-9001 budget=0.001', ['seller-premium'], 1),
56+
msg('seller-premium', 'BID round=42 price=0.0005 by=seller-premium note=readiness-proof', ['buyer-agent'], 2),
57+
msg('buyer-agent', 'AWARD round=42 to=seller-premium reason="readiness e2e winner"', ['seller-premium'], 3),
58+
msg('seller-premium', 'ESCROW_REQUIRED round=42 reference=Ref42 seller=7jwB amount=0.0005 deadline=600 settlement=arbiter', ['buyer-agent'], 4),
59+
msg('buyer-agent', 'DEPOSITED round=42 reference=Ref42 buyer=47Dp sig=dep42 settlement=arbiter vault=Vault42 arbiter=Arb42', ['seller-premium'], 5),
60+
msg('seller-premium', 'PAYMENT_REQUIRED round=42 rail=pay-sh amount=0.03 currency=USDC reference=pay-42 seller=pay.sh/txodds-context url=https://pay.sh/api/quicknode', [], 6),
61+
msg('seller-premium', 'PAYMENT_PROOF round=42 rail=pay-sh reference=pay-42 proof=pay-sh-demo:readiness buyer=seller-premium', [], 7),
62+
msg('seller-premium', 'PAYMENT_CONFIRMED round=42 rail=pay-sh reference=pay-42 paid=true amount=0.03 currency=USDC', [], 8),
63+
msg('seller-premium', 'DELIVERED round=42 {"service":"txline-edge","fixtureId":"9001","analysis":{"call":"readiness fixture delivered"}}', ['buyer-agent'], 9),
64+
msg('buyer-agent', 'VERIFY round=42 sha=abc service=txline arg=edge-9001 payload={"ok":true}', ['verifier-agent'], 10),
65+
msg('verifier-agent', 'VERIFIED round=42 verdict=pass by=verifier-agent reason="readiness hash + structure"', ['buyer-agent'], 11),
66+
msg('buyer-agent', 'ARBITER_RELEASED round=42 sig=rel42 settlement=arbiter', ['seller-premium'], 12),
67+
]
68+
return {
69+
base: { id: 'readiness-session', namespace: 'default', status: { type: 'executed' } },
70+
agents,
71+
threads: [{ id: threadId, name: 'market', creatorName: 'buyer-agent', participants: agents.map((a) => a.name), messages }],
72+
}
73+
}
74+
75+
async function waitForHealth(child) {
76+
const deadline = Date.now() + 30_000
77+
let last = ''
78+
while (Date.now() < deadline) {
79+
if (child.exitCode != null) throw new Error(`feed server exited early: ${child.exitCode}`)
80+
try {
81+
const r = await fetch(`${base}/api/health`)
82+
if (r.ok) return
83+
last = `HTTP ${r.status}`
84+
} catch (e) {
85+
last = e.message
86+
}
87+
await new Promise((resolve) => setTimeout(resolve, 500))
88+
}
89+
throw new Error(`feed server did not become healthy: ${last}`)
90+
}
91+
92+
async function json(path) {
93+
const r = await fetch(`${base}${path}`)
94+
const body = await r.text()
95+
assert(r.ok, `${path} failed: HTTP ${r.status} ${body.slice(0, 200)}`)
96+
return JSON.parse(body)
97+
}
98+
99+
async function main() {
100+
if (Number(process.versions.node.split('.')[0]) < 20) {
101+
throw new Error(`Node ${process.version} detected; readiness gate requires Node 20+`)
102+
}
103+
assert(existsSync(join(root, 'docs', 'PRODUCTION_READINESS.md')), 'docs/PRODUCTION_READINESS.md is missing')
104+
105+
if (!existsSync(join(runtimeDir, 'node_modules'))) run(runtimeDir, 'npm', ['install', '--no-audit', '--no-fund'])
106+
run(runtimeDir, 'npm', ['run', 'build'])
107+
if (!existsSync(join(feedDir, 'node_modules'))) run(feedDir, 'npm', ['install', '--no-audit', '--no-fund'])
108+
109+
writeFileSync(fixturePath, JSON.stringify(fixture(), null, 2), 'utf8')
110+
111+
const feed = spawn('npm', ['start'], {
112+
cwd: feedDir,
113+
shell: true,
114+
detached: platform() !== 'win32',
115+
stdio: ['ignore', 'pipe', 'pipe'],
116+
env: { ...process.env, FEED_FIXTURE: fixturePath, RUNS_DIR: runsDir, PORT: port },
117+
})
118+
feed.stdout.on('data', (d) => process.stdout.write(`[feed] ${d}`))
119+
feed.stderr.on('data', (d) => process.stderr.write(`[feed] ${d}`))
120+
121+
try {
122+
await waitForHealth(feed)
123+
124+
const health = await json('/api/health')
125+
assert(health.ok === true, 'health endpoint did not return ok=true')
126+
127+
const feedBody = await json('/api/feed?session=fixture')
128+
const round = feedBody.rounds?.find((r) => r.round === 42)
129+
assert(round?.status === 'settled', 'readiness round did not settle')
130+
assert(round?.verification?.verdict === 'pass', 'verifier pass did not fold into the round')
131+
assert(round?.proofReceipts?.[0]?.rail === 'pay-sh', 'proof receipt did not fold into the round')
132+
assert(round.proofReceipts[0].simulated === true, 'demo Pay.sh proof receipt was not marked simulated')
133+
134+
const runs = await json('/api/runs')
135+
const ledgerRun = runs.runs?.find((r) => r.runId === 'fixture/round-42')
136+
assert(ledgerRun?.proofReceipts?.[0]?.proof === 'pay-sh-demo:readiness', 'ledger run is missing proof receipt')
137+
138+
const threads = await json('/api/threads?session=fixture')
139+
assert(threads.threads?.[0]?.messages?.length >= 12, 'thread replay did not expose the bus transcript')
140+
assert(threads.agents?.some((a) => a.name === 'verifier-agent'), 'agent roster missing verifier-agent')
141+
142+
const receiptFile = join(runsDir, 'fixture', 'round-42', 'proof_receipts.json')
143+
assert(existsSync(receiptFile), 'proof_receipts.json was not written')
144+
const receipt = JSON.parse(readFileSync(receiptFile, 'utf8'))[0]
145+
assert(receipt.provider === 'pay.sh/txodds-context', 'proof receipt provider was not preserved')
146+
147+
run(deskDir, 'node', ['--check', 'ui/app.js'])
148+
JSON.parse(readFileSync(join(deskDir, 'src-tauri', 'tauri.conf.json'), 'utf8'))
149+
JSON.parse(readFileSync(join(deskDir, 'src-tauri', 'capabilities', 'default.json'), 'utf8'))
150+
151+
console.log('[readiness] PASS production-readiness e2e gate')
152+
} finally {
153+
stop(feed)
154+
rmSync(tmp, { recursive: true, force: true })
155+
}
156+
}
157+
158+
main().catch((e) => {
159+
console.error(`[readiness] FAIL ${e.stack ?? e}`)
160+
process.exitCode = 1
161+
})

0 commit comments

Comments
 (0)