|
| 1 | +# Canonical CEX market-data replay archive |
| 2 | + |
| 3 | +This is the deployment and operations contract for the CEX-broker portions of FIET-901 and FIET-903. FIET-903's broker RPC/capability/current-snapshot/live-stream sub-scope was completed by `cex-broker-order-book-depth-sourcing`; this change adds capture, storage, migration, and replay integrity. It does not add or gate RPCs. |
| 4 | + |
| 5 | +## Deployment identity and credential precedence |
| 6 | + |
| 7 | +TEE and non-TEE deployments run the same broker binary and register the full `ExecuteAction` and `Subscribe` service. Archive role does not reduce that service surface. |
| 8 | + |
| 9 | +Set the deployment-owned archive identity: |
| 10 | + |
| 11 | +```env |
| 12 | +CEX_BROKER_ARCHIVE_ENABLED=true |
| 13 | +CEX_BROKER_ARCHIVE_SOURCE=broker_read |
| 14 | +CEX_BROKER_DEPLOYMENT_ID=market-reader-eu-1 |
| 15 | +CEX_BROKER_CAPTURE_BUNDLE_ID=cex-2026-08-03-eu-1 |
| 16 | +CEX_BROKER_MARKET_CAPTURE_ENVIRONMENT=production |
| 17 | +CEX_BROKER_ARCHIVE_FORWARDER_URL=http://archive-forwarder:8090/archive |
| 18 | +CEX_BROKER_ARCHIVE_DEAD_LETTER_PATH=/var/lib/cex-broker/archive-loss.jsonl |
| 19 | +``` |
| 20 | + |
| 21 | +`CEX_BROKER_ARCHIVE_SOURCE` is closed to `broker_read|broker_write` and defaults to `broker_write` for existing deployments. The writer stamps this immutable value into envelopes, rows, and loss records. It is never inferred from API-key presence. Production FIET-901 collector startup requires `broker_read` and a non-empty deployment-owned capture bundle. Development capture explicitly uses `CEX_BROKER_MARKET_CAPTURE_ENVIRONMENT=development` and generates a `development:<deployment>` bundle when none is supplied. |
| 22 | + |
| 23 | +Credential resolution uses the broker's established fixed precedence and requires no archive-specific credential configuration: |
| 24 | + |
| 25 | +1. Use a matching broker account loaded from `.env`/deployment configuration. |
| 26 | +2. If no matching environment broker exists, use a complete request `api-key` and `api-secret` pair. |
| 27 | +3. If neither source exists, construct a credentialless exchange only for operations that already support public access. |
| 28 | + |
| 29 | +When environment and request credentials are both present, the environment-loaded broker wins. The broker does not classify keys as public, read-only, or write-capable and does not introduce credential-source or permission-attestation settings. For a non-TEE reader, ensure every credential that deployment or its trusted callers may supply has exchange-side trading and withdrawal permissions disabled. Those exchange permissions—not archive source or credential location—establish effective privilege. |
| 30 | + |
| 31 | +## Four-feed collector |
| 32 | + |
| 33 | +The collector is an independent keep-alive client of a separately deployed full broker. Set its broker target and point its canonical configuration at a JSON document containing only feed intent: |
| 34 | + |
| 35 | +```env |
| 36 | +CEX_BROKER_URL=cex-broker.internal:8086 |
| 37 | +CEX_BROKER_MARKET_DATA_COLLECTOR_CONFIG=/etc/cex-broker/market-data-subscriptions.json |
| 38 | +``` |
| 39 | + |
| 40 | +```json |
| 41 | +{ |
| 42 | + "subscriptions": [ |
| 43 | + { "exchange": "binance", "symbol": "BTC/USDT", "feed": "ORDERBOOK", "depthLimit": 50 }, |
| 44 | + { "exchange": "binance", "symbol": "BTC/USDT", "feed": "TICKER" }, |
| 45 | + { "exchange": "binance", "symbol": "BTC/USDT", "feed": "TRADES" }, |
| 46 | + { "exchange": "binance", "symbol": "BTC/USDT", "feed": "OHLCV", "timeframe": "1m", "bootstrapLimit": 100 } |
| 47 | + ] |
| 48 | +} |
| 49 | +``` |
| 50 | + |
| 51 | +Run it with `bun run start-market-data-collector`. The collector starts no loopback broker, loads no CEX credentials, sends no API-key metadata, owns no archive writer, and does not connect to ClickHouse. The remote broker resolves its environment-first credentials and attaches the production environment, deployment, capture bundle, source, and integrity provenance configured in the preceding section. Collector JSON containing `environment`, `captureBundleId`, or other archive identity is rejected. |
| 52 | + |
| 53 | +Each entry has an independent reconnect supervisor and health state. OHLCV retains bootstrap/catch-up and stamps `broker_bootstrap_fetch_v1` separately from live capture. ORDERBOOK, TICKER, and TRADES record unrecoverable gaps after reconnect rather than synthesize missing events. |
| 54 | + |
| 55 | +`CEX_BROKER_OHLCV_COLLECTOR_CONFIG` remains supported as an OHLCV-only array. Move those entries into the canonical document and add `feed: "OHLCV"`; archive identity stays on the broker. The `start-ohlcv-collector` command and existing service directory/image remain compatibility wrappers for both formats. |
| 56 | + |
| 57 | +An external CCXT or Hummingbot fallback is an optional out-of-band producer of the shared capture contract, not a broker-collector implementation. It must declare its provider, versioned fallback source mode, reason, configured exchange, and configured pair. Cross-venue or cross-pair substitution is rejected. |
| 58 | + |
| 59 | +## Capture and integrity contract |
| 60 | + |
| 61 | +All canonical rows carry source, deployment, capture bundle, exchange, trading pair/source symbol, provider, feed, source mode, source/received timestamps, raw capture ID/scope, schema version, checksum algorithm, raw checksum, normalized checksum, and provenance completeness. The current versions are: |
| 62 | + |
| 63 | +- schema: `1.0.0` |
| 64 | +- checksum: `sha256-canonical-json-v1` |
| 65 | +- construction: sampled top-N snapshots for live/current broker order books |
| 66 | + |
| 67 | +Raw payloads are redacted before identity/checksum calculation. Canonical JSON sorts object keys, uses finite plain-decimal numbers, normalizes negative zero, omits undefined object values, and excludes checksum fields from their own projections. The TypeScript fixture is `test/fixtures/canonical-market-capture-v1.json`; `research/hummingbot/canonical_capture_fixture.py` is the matching Maker-side verifier. |
| 68 | + |
| 69 | +Exact L2 is future-facing and non-blocking for this delivery. The broker reports it unsupported and never silently labels sampled evidence exact. A future exact producer must supply complete continuity proof. |
| 70 | + |
| 71 | +## ClickHouse and replay |
| 72 | + |
| 73 | +Canonical storage is: |
| 74 | + |
| 75 | +- `market_data.cex_stream_events` for the redacted raw ledger |
| 76 | +- `market_data.cex_ticker_events`, `market_data.cex_trades`, and `market_data.cex_ohlcv` |
| 77 | +- append-only `market_data.cex_order_book_levels` and `market_data.cex_order_book_depth_summary` |
| 78 | + |
| 79 | +`cex_ohlcv` uses `ReplacingMergeTree(broker_version)`; `cex_ohlcv_closed` applies `FINAL` and closed-bar semantics. Order-book physical duplicates remain auditable. The `_canonical` views expose one checksum-consistent logical row, while `_conflicts` expose keys with multiple checksums. Same-batch conflicts are rejected by the forwarder; cross-batch conflicts remain stored and must block the affected replay bundle. |
| 80 | + |
| 81 | +Use `schema/clickhouse/canonical_market_data_replay.sql` for bounded bundle/exchange/pair/source-time replay. Its conflict preflights must return no rows before consuming the canonical views. |
| 82 | + |
| 83 | +The retained FIET-907 reference exporter materializes a conflict-free order-book window directly from ClickHouse to Maker-compatible Parquet files: |
| 84 | + |
| 85 | +```bash |
| 86 | +CEX_BROKER_REPLAY_EXPORT_DIRECTORY=/tmp/maker-capture \ |
| 87 | +CEX_BROKER_REPLAY_CAPTURE_BUNDLE_IDS=cex-2026-08-03-eu-1 \ |
| 88 | +CEX_BROKER_REPLAY_EXCHANGE=binance \ |
| 89 | +CEX_BROKER_REPLAY_TRADING_PAIR=BTC-USDT \ |
| 90 | +CEX_BROKER_REPLAY_START_TIME_MS=1785715200000 \ |
| 91 | +CEX_BROKER_REPLAY_END_TIME_MS=1785801600000 \ |
| 92 | +bun scripts/export-canonical-orderbook-parquet.ts |
| 93 | + |
| 94 | +uv run --project research/python --extra dev \ |
| 95 | + python research/hummingbot/order_book_parquet_fixture.py \ |
| 96 | + /tmp/maker-capture/order_book_levels.parquet \ |
| 97 | + /tmp/maker-capture/order_book_depth_summary.parquet |
| 98 | +``` |
| 99 | + |
| 100 | +The exporter refuses to overwrite existing files or export a selected window with an order-book checksum conflict. The Python verifier checks capture-core field presence and recomputes every normalized-row checksum from the Parquet values. Neither tool calls the broker or an exchange. Fixture materialization, coverage reports, and replay-bundle assembly are owned by [FIET-907](https://linear.app/usherlabs/issue/FIET-907/clickhouse-backtest-fixture-materializers-and-coveragereplay-bundles), not by the live capture runtime. |
| 101 | + |
| 102 | +For complete strategy-pair validation, point `CEX_BROKER_REPLAY_VALIDATION_CONFIG` at a JSON document whose `windows` array contains `captureBundleIds`, `exchange`, `tradingPair`, `startTimeMs`, and `endTimeMs`, then run `bun scripts/validate-canonical-market-replay.ts`. Every configured window must contain raw and normalized ORDERBOOK, TICKER, TRADES, and OHLCV evidence; any missing feed or checksum conflict fails validation. |
| 103 | + |
| 104 | +## Migration, cutover, rollback, and deployment observation |
| 105 | + |
| 106 | +The upgraded broker always writes the latest canonical schema. There is no runtime legacy/dual/canonical write setting. Upgrading an existing legacy deployment therefore requires the ClickHouse table migration before the new broker version is deployed. |
| 107 | + |
| 108 | +Follow `schema/clickhouse/migrations/canonical_market_data_replay_cutover.sql` phase by phase: apply canonical DDL, quiesce legacy writers, migrate every retained bounded window, validate parity, switch consumers, and only then deploy the canonical-only broker. |
| 109 | + |
| 110 | +```bash |
| 111 | +CEX_BROKER_MIGRATION_START_TIME_MS=1700000000000 \ |
| 112 | +CEX_BROKER_MIGRATION_END_TIME_MS=1700086400000 \ |
| 113 | +bun run scripts/migrate-legacy-market-data-to-canonical.ts |
| 114 | + |
| 115 | +# Repeat after reviewing dry-run counts: |
| 116 | +CEX_BROKER_CANONICAL_MIGRATION_CONFIRM=true \ |
| 117 | +CEX_BROKER_MIGRATION_START_TIME_MS=1700000000000 \ |
| 118 | +CEX_BROKER_MIGRATION_END_TIME_MS=1700086400000 \ |
| 119 | +bun run scripts/migrate-legacy-market-data-to-canonical.ts |
| 120 | +``` |
| 121 | + |
| 122 | +The script reads `market_data.orderbook_snapshots` and `market_data.candles` directly from ClickHouse and writes their canonical equivalents. It never reads fixture files and never calls a broker or exchange. Legacy migration stamps `legacy_migration_v1` and `provenance_complete=0`; unavailable bundle/raw ID/raw scope/raw checksum remain `NULL`. Reruns preserve identical logical checksums: order-book canonical views collapse agreeing physical deliveries and OHLCV replacement semantics select the recorded broker version. |
| 123 | + |
| 124 | +Before cutover, run the parity and replay queries for every configured pair/window and complete the FIET-937 production observation window. Record feed health, last-frame age, reconnects, unrecoverable gaps, received/archived/invalid/sampled rows, queue saturation, journaled rows, checksum conflicts, parity mismatches, and Maker replay consumption. Any unaccounted row, conflict, parity mismatch, stale feed, or persistent journal growth blocks deployment cutover. This operational gate is not an implementation-completion requirement for the archived OpenSpec change and repository checks do not claim that it has occurred. |
| 125 | + |
| 126 | +Rollback stops the upgraded broker, restores retained legacy names if necessary, and rolls back to the previous legacy-writing application version. The runbook uses renames rather than drops; canonical and legacy base data remain recoverable throughout the retention period. |
0 commit comments