feat(archive): durable Maker strategy acceptance

.github/workflows/ci.yml

@@ -12,6 +12,20 @@ jobs:
   test:
     runs-on: ubuntu-latest
 
+    services:
+      clickhouse:
+        image: clickhouse/clickhouse-server:24.8
+        env:
+          CLICKHOUSE_USER: default
+          CLICKHOUSE_PASSWORD: integration-test
+        ports:
+          - 8123:8123
+        options: >-
+          --health-cmd "wget --no-verbose --tries=1 --spider http://localhost:8123/ping || exit 1"
+          --health-interval 5s
+          --health-timeout 3s
+          --health-retries 20
+
     steps:
       - name: Checkout repo
         uses: actions/checkout@v3
@@ -32,6 +46,11 @@ jobs:
 
       - name: Run tests
         run: bun test
+        env:
+          CLICKHOUSE_REQUIRED: "1"
+          CLICKHOUSE_TEST_URL: http://localhost:8123
+          CLICKHOUSE_USER: default
+          CLICKHOUSE_PASSWORD: integration-test
 
       # Gate PRs on the same strict build the publish workflow runs on tag push
       # (dts-bundle-generator + strict tsc). Without this, develop can merge code

.gitignore

@@ -26,6 +26,7 @@ report.[0-9]_.[0-9]_.[0-9]_.[0-9]_.json
 .eslintcache
 .cache
 *.tsbuildinfo
+archive-forwarder-spool.sqlite*
 
 # research outputs
 research/python/examples/output/

README.md

@@ -24,6 +24,10 @@ A high-performance gRPC-based cryptocurrency exchange broker service that provid
 - API keys for supported exchanges (e.g., Binance, Bybit, etc.)
 - Optional: Verity prover URL for zero-knowledge proof integration
 
+## Service architecture
+
+See [SERVICES_ARCHITECTURE.md](SERVICES_ARCHITECTURE.md) for the authoritative boundaries between the full broker, operator market-data collector, archive-forwarder, research viewer, offline tools, and externally owned systems.
+
 ## 🛠️ Installation
 
 1. **Clone the repository:**
@@ -97,8 +101,15 @@ CLICKHOUSE_PORT=8123 bun run start-candle-viewer   # http://localhost:8091
 
 Dev watchers: `dev:candle-viewer`, `dev:archive-forwarder`, `dev:archive-watch` (see [research/README.md](research/README.md)).
 
+Maker `hb_runtime` delivery uses the forwarder's durable strategy-only SQLite
+acceptance boundary. See [docs/archive-forwarder-durable-acceptance.md](docs/archive-forwarder-durable-acceptance.md)
+for the wire versions, fixed quota/retention, retry ownership, health semantics,
+and required production volume.
+
 Key env vars: `CEX_BROKER_ARCHIVE_ENABLED=true`, `CEX_BROKER_ARCHIVE_FORWARDER_URL`, `CEX_BROKER_ARCHIVE_DEAD_LETTER_PATH`, and `CEX_BROKER_DEPLOYMENT_ID`. The archive is disabled for every enable value except the exact string `true`. Production durability requires the dead-letter file to reside on persistent writable storage or a mounted volume; a container-local ephemeral path is not durable.
 
+Canonical replay capture for FIET-901/FIET-903 adds deployment-owned archive roles, capture bundles, four-feed collector configuration, deterministic checksums, conflict-blocking replay views, and mandatory legacy-to-canonical table migration before canonical-only deployment. Credential resolution remains environment-loaded broker first, request metadata second, and supported public fallback last. See [docs/canonical-market-data-replay.md](docs/canonical-market-data-replay.md) for the complete deployment and cutover contract.
+
 #### Wallet-authenticated exchanges
 
 Some exchanges (for example Hyperliquid, Vertex, Paradex, and Derive) authenticate with an on-chain wallet instead of exchange-issued API keys. The broker keeps the same `API_KEY` / `API_SECRET` interface and maps credentials internally based on each exchange's CCXT `requiredCredentials`:

docker/clickhouse-research.compose.yml

@@ -5,6 +5,9 @@ services:
     ports:
       - "8123:8123"
       - "9000:9000"
+    environment:
+      - CLICKHOUSE_USER=default
+      - CLICKHOUSE_PASSWORD
     volumes:
       - clickhouse_data:/var/lib/clickhouse
       - ../schema/clickhouse/market_data.sql:/docker-entrypoint-initdb.d/01_market_data.sql:ro
@@ -24,19 +27,22 @@ services:
     ports:
       - "8090:8090"
     environment:
-      ARCHIVE_FORWARDER_PORT: "8090"
-      CLICKHOUSE_HOST: clickhouse
-      CLICKHOUSE_PORT: "8123"
-      CLICKHOUSE_USER: default
-      CLICKHOUSE_PASSWORD: ""
-      CLICKHOUSE_DATABASE: market_data
+      - ARCHIVE_FORWARDER_PORT=8090
+      - CLICKHOUSE_HOST=clickhouse
+      - CLICKHOUSE_PORT=8123
+      - CLICKHOUSE_USER=default
+      - CLICKHOUSE_PASSWORD
+      - CLICKHOUSE_DATABASE=market_data
+      - ARCHIVE_FORWARDER_SPOOL_PATH=/var/lib/archive-forwarder/spool.sqlite
     volumes:
       - ..:/app
+      - archive_forwarder_spool:/var/lib/archive-forwarder
     depends_on:
       - clickhouse
 
 volumes:
   clickhouse_data:
+  archive_forwarder_spool:
 
 networks:
   default:

docs/archive-forwarder-durable-acceptance.md

@@ -0,0 +1,96 @@
+# Maker archive-forwarder durable acceptance
+
+## Purpose and ownership boundary
+
+Maker's runtime archive bridge makes one non-blocking HTTP attempt. The archive
+forwarder therefore owns a conforming request only after it commits the request
+to its SQLite spool. It then returns HTTP 202 and drains each represented table
+to ClickHouse independently. Maker does not retry an acknowledged batch.
+
+This path is deliberately narrow: `source=hb_runtime` and exclusively these five
+tables:
+
+- `strategy_data.policy_evaluation_events`
+- `strategy_data.strategy_policy_snapshots`
+- `strategy_data.market_identity`
+- `strategy_data.symbol_mapping`
+- `strategy_data.inventory_settlement_events`
+
+Mixed requests are rejected atomically. Strategy tables under another source
+are also rejected. `broker_read`, `broker_write`, account, execution, market-data,
+and Maker replay requests keep their existing direct synchronous ClickHouse path
+and do not consume spool capacity.
+
+## Wire versions
+
+The envelope requires non-empty `source`, non-empty `deployment_id`, and 1–1000
+rows. Missing/empty `schema_version` and version `1` use the legacy contract.
+Version `2` additionally requires non-empty `producer_id`, `producer_run_id`,
+`stream_name`, and `archive_event_id`, plus positive integer `stream_seq` and
+`seq`. Unknown versions and row/envelope provenance mismatches return HTTP 400.
+
+`test/fixtures/archive_forwarder_envelope.json` is byte-identical to the fixture
+at Maker commit `563594435853c88cca5b187b8c999f845e31136b`; its pinned SHA-256 is
+`784f647e048052a6c3382309b1a86abfbe08bc162363ead9fc88eaa1ba3d50c9`.
+
+## Spool storage and fixed service limits
+
+The forwarder uses Bun SQLite with WAL, foreign keys, a 5-second busy timeout,
+and `synchronous=FULL`. Local runs default to:
+
+```text
+./archive-forwarder-spool.sqlite
+```
+
+Production must configure and persist:
+
+```text
+ARCHIVE_FORWARDER_SPOOL_PATH=/var/lib/archive-forwarder/spool.sqlite
+```
+
+Mount `/var/lib/archive-forwarder` on durable storage. The supplied Dockerfile
+and compose service do so. This is an archive-forwarder service option, not a
+core broker `CEX_BROKER_*` option.
+
+Quota is fixed at 1 GiB of deterministic payload/metadata accounting. Retention
+is fixed at 72 hours from admission. Neither has an environment override. Quota
+reservation, expiry cleanup, batch insertion, and per-table work insertion are
+transactional.
+
+## Responses and retry behavior
+
+- `202`: the complete Maker batch is durably owned; ClickHouse completion may be pending.
+- `400`: invalid envelope, table/source mix, provenance, or schema version/identity.
+- `429`: accepting the batch would exceed the fixed spool quota; ownership was not accepted.
+- `503`: the spool is missing, corrupt, unwritable, or cannot commit; ownership was not accepted.
+
+Transient ClickHouse failures retry after 1, 2, 4, 8, 16, 32, then 60 seconds,
+with ±20% jitter, through retention expiry. Completion is tracked per table, so
+a failed table never replays successful siblings. Schema and authentication
+failures become terminal and remain visible until expiry instead of hot-looping.
+
+Each batch/table work item persists one stable SHA-256 deduplication token. The
+worker passes it as `insert_deduplication_token`, while every strategy table has
+`non_replicated_deduplication_window=1000000`. A crash after a ClickHouse commit
+but before SQLite completion therefore retries with the same token.
+
+## Health and operations
+
+`GET /health` reports ClickHouse/schema readiness separately from spool
+writability and exposes queued batches/work, terminal/expired work, accounted
+bytes, oldest age, and the last bounded error class.
+
+- ClickHouse down + spool healthy: HTTP 200, `status=degraded`, durable admission available.
+- ClickHouse ready + spool healthy: HTTP 200, `status=ok`.
+- Spool unhealthy: HTTP 503, `status=unavailable`, even when ClickHouse is reachable.
+
+On restart, the service attempts to open/migrate the spool before listening and
+resumes incomplete work after ClickHouse schema readiness. If the persisted file
+is unavailable or corrupt, the service remains up only to expose unavailable
+health and the existing direct path; strategy admission fails with 503 until an
+operator preserves/repairs the file and restarts. Never delete or replace a spool
+file with pending work during rollback. Stop admission, preserve the volume, and
+retain the additive ClickHouse columns.
+
+The production feed/archive observation window is a separate FIET-937 cutover
+gate. Repository conformance checks do not claim that observation has run.

docs/canonical-market-data-replay.md

@@ -0,0 +1,126 @@
+# Canonical CEX market-data replay archive
+
+This is the deployment and operations contract for the CEX-broker portions of FIET-901 and FIET-903. FIET-903's broker RPC/capability/current-snapshot/live-stream sub-scope was completed by `cex-broker-order-book-depth-sourcing`; this change adds capture, storage, migration, and replay integrity. It does not add or gate RPCs.
+
+## Deployment identity and credential precedence
+
+TEE and non-TEE deployments run the same broker binary and register the full `ExecuteAction` and `Subscribe` service. Archive role does not reduce that service surface.
+
+Set the deployment-owned archive identity:
+
+```env
+CEX_BROKER_ARCHIVE_ENABLED=true
+CEX_BROKER_ARCHIVE_SOURCE=broker_read
+CEX_BROKER_DEPLOYMENT_ID=market-reader-eu-1
+CEX_BROKER_CAPTURE_BUNDLE_ID=cex-2026-08-03-eu-1
+CEX_BROKER_MARKET_CAPTURE_ENVIRONMENT=production
+CEX_BROKER_ARCHIVE_FORWARDER_URL=http://archive-forwarder:8090/archive
+CEX_BROKER_ARCHIVE_DEAD_LETTER_PATH=/var/lib/cex-broker/archive-loss.jsonl
+```
+
+`CEX_BROKER_ARCHIVE_SOURCE` is closed to `broker_read|broker_write` and defaults to `broker_write` for existing deployments. The writer stamps this immutable value into envelopes, rows, and loss records. It is never inferred from API-key presence. Production FIET-901 collector startup requires `broker_read` and a non-empty deployment-owned capture bundle. Development capture explicitly uses `CEX_BROKER_MARKET_CAPTURE_ENVIRONMENT=development` and generates a `development:<deployment>` bundle when none is supplied.
+
+Credential resolution uses the broker's established fixed precedence and requires no archive-specific credential configuration:
+
+1. Use a matching broker account loaded from `.env`/deployment configuration.
+2. If no matching environment broker exists, use a complete request `api-key` and `api-secret` pair.
+3. If neither source exists, construct a credentialless exchange only for operations that already support public access.
+
+When environment and request credentials are both present, the environment-loaded broker wins. The broker does not classify keys as public, read-only, or write-capable and does not introduce credential-source or permission-attestation settings. For a non-TEE reader, ensure every credential that deployment or its trusted callers may supply has exchange-side trading and withdrawal permissions disabled. Those exchange permissions—not archive source or credential location—establish effective privilege.
+
+## Four-feed collector
+
+The collector is an independent keep-alive client of a separately deployed full broker. Set its broker target and point its canonical configuration at a JSON document containing only feed intent:
+
+```env
+CEX_BROKER_URL=cex-broker.internal:8086
+CEX_BROKER_MARKET_DATA_COLLECTOR_CONFIG=/etc/cex-broker/market-data-subscriptions.json
+```
+
+```json
+{
+  "subscriptions": [
+    { "exchange": "binance", "symbol": "BTC/USDT", "feed": "ORDERBOOK", "depthLimit": 50 },
+    { "exchange": "binance", "symbol": "BTC/USDT", "feed": "TICKER" },
+    { "exchange": "binance", "symbol": "BTC/USDT", "feed": "TRADES" },
+    { "exchange": "binance", "symbol": "BTC/USDT", "feed": "OHLCV", "timeframe": "1m", "bootstrapLimit": 100 }
+  ]
+}
+```
+
+Run it with `bun run start-market-data-collector`. The collector starts no loopback broker, loads no CEX credentials, sends no API-key metadata, owns no archive writer, and does not connect to ClickHouse. The remote broker resolves its environment-first credentials and attaches the production environment, deployment, capture bundle, source, and integrity provenance configured in the preceding section. Collector JSON containing `environment`, `captureBundleId`, or other archive identity is rejected.
+
+Each entry has an independent reconnect supervisor and health state. OHLCV retains bootstrap/catch-up and stamps `broker_bootstrap_fetch_v1` separately from live capture. ORDERBOOK, TICKER, and TRADES record unrecoverable gaps after reconnect rather than synthesize missing events.
+
+`CEX_BROKER_OHLCV_COLLECTOR_CONFIG` remains supported as an OHLCV-only array. Move those entries into the canonical document and add `feed: "OHLCV"`; archive identity stays on the broker. The `start-ohlcv-collector` command and existing service directory/image remain compatibility wrappers for both formats.
+
+An external CCXT or Hummingbot fallback is an optional out-of-band producer of the shared capture contract, not a broker-collector implementation. It must declare its provider, versioned fallback source mode, reason, configured exchange, and configured pair. Cross-venue or cross-pair substitution is rejected.
+
+## Capture and integrity contract
+
+All canonical rows carry source, deployment, capture bundle, exchange, trading pair/source symbol, provider, feed, source mode, source/received timestamps, raw capture ID/scope, schema version, checksum algorithm, raw checksum, normalized checksum, and provenance completeness. The current versions are:
+
+- schema: `1.0.0`
+- checksum: `sha256-canonical-json-v1`
+- construction: sampled top-N snapshots for live/current broker order books
+
+Raw payloads are redacted before identity/checksum calculation. Canonical JSON sorts object keys, uses finite plain-decimal numbers, normalizes negative zero, omits undefined object values, and excludes checksum fields from their own projections. The TypeScript fixture is `test/fixtures/canonical-market-capture-v1.json`; `research/hummingbot/canonical_capture_fixture.py` is the matching Maker-side verifier.
+
+Exact L2 is future-facing and non-blocking for this delivery. The broker reports it unsupported and never silently labels sampled evidence exact. A future exact producer must supply complete continuity proof.
+
+## ClickHouse and replay
+
+Canonical storage is:
+
+- `market_data.cex_stream_events` for the redacted raw ledger
+- `market_data.cex_ticker_events`, `market_data.cex_trades`, and `market_data.cex_ohlcv`
+- append-only `market_data.cex_order_book_levels` and `market_data.cex_order_book_depth_summary`
+
+`cex_ohlcv` uses `ReplacingMergeTree(broker_version)`; `cex_ohlcv_closed` applies `FINAL` and closed-bar semantics. Order-book physical duplicates remain auditable. The `_canonical` views expose one checksum-consistent logical row, while `_conflicts` expose keys with multiple checksums. Same-batch conflicts are rejected by the forwarder; cross-batch conflicts remain stored and must block the affected replay bundle.
+
+Use `schema/clickhouse/canonical_market_data_replay.sql` for bounded bundle/exchange/pair/source-time replay. Its conflict preflights must return no rows before consuming the canonical views.
+
+The retained FIET-907 reference exporter materializes a conflict-free order-book window directly from ClickHouse to Maker-compatible Parquet files:
+
+```bash
+CEX_BROKER_REPLAY_EXPORT_DIRECTORY=/tmp/maker-capture \
+CEX_BROKER_REPLAY_CAPTURE_BUNDLE_IDS=cex-2026-08-03-eu-1 \
+CEX_BROKER_REPLAY_EXCHANGE=binance \
+CEX_BROKER_REPLAY_TRADING_PAIR=BTC-USDT \
+CEX_BROKER_REPLAY_START_TIME_MS=1785715200000 \
+CEX_BROKER_REPLAY_END_TIME_MS=1785801600000 \
+bun scripts/export-canonical-orderbook-parquet.ts
+
+uv run --project research/python --extra dev \
+  python research/hummingbot/order_book_parquet_fixture.py \
+  /tmp/maker-capture/order_book_levels.parquet \
+  /tmp/maker-capture/order_book_depth_summary.parquet
+```
+
+The exporter refuses to overwrite existing files or export a selected window with an order-book checksum conflict. The Python verifier checks capture-core field presence and recomputes every normalized-row checksum from the Parquet values. Neither tool calls the broker or an exchange. Fixture materialization, coverage reports, and replay-bundle assembly are owned by [FIET-907](https://linear.app/usherlabs/issue/FIET-907/clickhouse-backtest-fixture-materializers-and-coveragereplay-bundles), not by the live capture runtime.
+
+For complete strategy-pair validation, point `CEX_BROKER_REPLAY_VALIDATION_CONFIG` at a JSON document whose `windows` array contains `captureBundleIds`, `exchange`, `tradingPair`, `startTimeMs`, and `endTimeMs`, then run `bun scripts/validate-canonical-market-replay.ts`. Every configured window must contain raw and normalized ORDERBOOK, TICKER, TRADES, and OHLCV evidence; any missing feed or checksum conflict fails validation.
+
+## Migration, cutover, rollback, and deployment observation
+
+The upgraded broker always writes the latest canonical schema. There is no runtime legacy/dual/canonical write setting. Upgrading an existing legacy deployment therefore requires the ClickHouse table migration before the new broker version is deployed.
+
+Follow `schema/clickhouse/migrations/canonical_market_data_replay_cutover.sql` phase by phase: apply canonical DDL, quiesce legacy writers, migrate every retained bounded window, validate parity, switch consumers, and only then deploy the canonical-only broker.
+
+```bash
+CEX_BROKER_MIGRATION_START_TIME_MS=1700000000000 \
+CEX_BROKER_MIGRATION_END_TIME_MS=1700086400000 \
+bun run scripts/migrate-legacy-market-data-to-canonical.ts
+
+# Repeat after reviewing dry-run counts:
+CEX_BROKER_CANONICAL_MIGRATION_CONFIRM=true \
+CEX_BROKER_MIGRATION_START_TIME_MS=1700000000000 \
+CEX_BROKER_MIGRATION_END_TIME_MS=1700086400000 \
+bun run scripts/migrate-legacy-market-data-to-canonical.ts
+```
+
+The script reads `market_data.orderbook_snapshots` and `market_data.candles` directly from ClickHouse and writes their canonical equivalents. It never reads fixture files and never calls a broker or exchange. Legacy migration stamps `legacy_migration_v1` and `provenance_complete=0`; unavailable bundle/raw ID/raw scope/raw checksum remain `NULL`. Reruns preserve identical logical checksums: order-book canonical views collapse agreeing physical deliveries and OHLCV replacement semantics select the recorded broker version.
+
+Before cutover, run the parity and replay queries for every configured pair/window and complete the FIET-937 production observation window. Record feed health, last-frame age, reconnects, unrecoverable gaps, received/archived/invalid/sampled rows, queue saturation, journaled rows, checksum conflicts, parity mismatches, and Maker replay consumption. Any unaccounted row, conflict, parity mismatch, stale feed, or persistent journal growth blocks deployment cutover. This operational gate is not an implementation-completion requirement for the archived OpenSpec change and repository checks do not claim that it has occurred.
+
+Rollback stops the upgraded broker, restores retained legacy names if necessary, and rolls back to the previous legacy-writing application version. The runbook uses renames rather than drops; canonical and legacy base data remain recoverable throughout the retention period.