diff --git a/openspec/changes/maker-archive-forwarder-conformance/.openspec.yaml b/openspec/changes/archive/2026-08-03-maker-archive-forwarder-conformance/.openspec.yaml similarity index 100% rename from openspec/changes/maker-archive-forwarder-conformance/.openspec.yaml rename to openspec/changes/archive/2026-08-03-maker-archive-forwarder-conformance/.openspec.yaml diff --git a/openspec/changes/maker-archive-forwarder-conformance/design.md b/openspec/changes/archive/2026-08-03-maker-archive-forwarder-conformance/design.md similarity index 100% rename from openspec/changes/maker-archive-forwarder-conformance/design.md rename to openspec/changes/archive/2026-08-03-maker-archive-forwarder-conformance/design.md diff --git a/openspec/changes/archive/2026-08-03-maker-archive-forwarder-conformance/implementation-evidence.md b/openspec/changes/archive/2026-08-03-maker-archive-forwarder-conformance/implementation-evidence.md new file mode 100644 index 0000000..152b9b8 --- /dev/null +++ b/openspec/changes/archive/2026-08-03-maker-archive-forwarder-conformance/implementation-evidence.md @@ -0,0 +1,91 @@ +# Implementation evidence + +## Released CEX boundary + +The durable Maker acceptance boundary shipped through +[usherlabs/cex-broker#99](https://github.com/usherlabs/cex-broker/pull/99). + +| Evidence | Immutable reference | +|---|---| +| Reviewed CEX change commit | `1bd2af1ef7ec7d4fc9d223cdcc9211c57d600526` | +| CEX merge commit | `5eec63dfcd947aae3b0f861d39cf96e05339cd0c` | +| Release | `v0.2.37`, dereferencing to the merge commit | +| npm integrity | `sha512-e4Cq665s3Be2E6QMADWLzIvXgj3Vd/PmLOrOCtkngCt3FjKdgE4SmHqoRDhMz2/reSv2ttfePclHVqG05s1CeA==` | +| Archive-forwarder image | `ghcr.io/usherlabs/cex-broker-archive-forwarder:0.2.37@sha256:713068ff7bc7b75640ea990baa68c317876d8efdd3f312cfa38b37a799eb8a44` | +| Core broker image | `ghcr.io/usherlabs/cex-broker:0.2.37@sha256:cbc959948f9b6caed2759ca40f3716af2354b12814078368aba73d22c14577b3` | +| Pinned Maker producer commit | `563594435853c88cca5b187b8c999f845e31136b` | +| Shared fixture SHA-256 | `784f647e048052a6c3382309b1a86abfbe08bc162363ead9fc88eaa1ba3d50c9` | + +Required CEX evidence passed: + +- [PR CI](https://github.com/usherlabs/cex-broker/actions/runs/30792787681) + used ClickHouse 24.8 and passed contract, SQLite durability/fault/restart, + legacy upgrade, all-five-table schema-v2, and stable-token deduplication + coverage; +- [post-merge CI](https://github.com/usherlabs/cex-broker/actions/runs/30792875090) + passed; +- [package and core-image publication](https://github.com/usherlabs/cex-broker/actions/runs/30792910567) + passed; and +- [archive-forwarder image publication](https://github.com/usherlabs/cex-broker/actions/runs/30792910562) + passed. + +Local verification before release recorded 593 repository tests passing with no +failures, 48 focused forwarder tests, 12 real ClickHouse integration tests, +type-check, build, lint, Compose validation, strict OpenSpec validation, and a +clean diff check. + +## Maker consumer closure + +Maker consumed the release through +[usherlabs/fiet-maker#1067](https://github.com/usherlabs/fiet-maker/pull/1067). +The dependency/evidence commit is +`f84ebf3291a7b0060b9d90c44ea785d9ad6ea820`; the merge commit is +`1ae1d0faf87396a35089ec51eedc55ff65d675d9`. + +Maker's production build authority is its parent `pnpm-workspace.yaml` and root +`pnpm-lock.yaml`. They pin the workspace's wrapper resolution to exact +`@usherlabs/cex-broker@0.2.37` and the npm integrity above. The separately +versioned fiet-tee submodule manifest and lock remain standalone repository +release inputs; they do not override the parent production workspace pin. + +The installed wrapper passed 18 tests, the SGX archive-storage path contract, +and the CommonJS production bundle check against `0.2.37`. Maker strict +OpenSpec validation passed, all 64 tasks in +`complete-nonblocking-strategy-replay-archive` are complete, and the shared +fixture hash is identical. Maker post-merge advisory evidence is +[run 30793571785](https://github.com/usherlabs/fiet-maker/actions/runs/30793571785); +the production-context CEX SGX staging build is +[run 30793571854](https://github.com/usherlabs/fiet-maker/actions/runs/30793571854). +Both runs passed. + +## Linear scope transition + +- [FIET-901](https://linear.app/usherlabs/issue/FIET-901/read-broker-cex-market-data-into-clickhouse) + records the released implementation, same-full-broker deployment model, + credential precedence, absence of credential profiles/attestation knobs, and + the FIET-937/FIET-907 handoff while remaining open/In Review. +- [FIET-903](https://linear.app/usherlabs/issue/FIET-903/canonical-cex-order-book-depth-tables-for-strategy-runtime-replay) + records the already-complete RPC/capability sub-scope, released storage + sub-scope, sampled-mode closure, and future-facing exact-L2 guardrail while + remaining open/In Review. +- [FIET-909](https://linear.app/usherlabs/issue/FIET-909/hb-runtime-telemetry-bridge-for-strategy-replay-sourcing) + records the Maker/CEX conformance release and immutable evidence. +- [FIET-937](https://linear.app/usherlabs/issue/FIET-937/production-cex-market-data-archive-into-clickhouse-for-strategy-replay) + retains the production observation window and go/no-go cutover decision. +- [FIET-924](https://linear.app/usherlabs/issue/FIET-924/review-and-implementation-of-data-warehouse-ingestion-packagetemplate) + remains Done and was not reopened or repurposed. + +## Ownership result + +- Maker makes exactly one bounded HTTP attempt per admitted strategy batch. +- HTTP 202 means the complete batch is durably owned by the forwarder's fixed + 1 GiB, 72-hour SQLite spool. +- Retry, restart recovery, per-table completion, expiry, and ClickHouse + deduplication belong to the forwarder after acceptance. +- Broker-origin traffic keeps its existing direct synchronous path. +- No new core full-broker `CEX_BROKER_*` environment variable was introduced; + only forwarder-local `ARCHIVE_FORWARDER_SPOOL_PATH` selects the durable file. +- Production observation remains a deployment/cutover gate owned by FIET-937, + not an OpenSpec implementation or archive prerequisite. +- Parquet materialization remains an out-of-band ClickHouse consumer owned by + FIET-907. diff --git a/openspec/changes/maker-archive-forwarder-conformance/proposal.md b/openspec/changes/archive/2026-08-03-maker-archive-forwarder-conformance/proposal.md similarity index 100% rename from openspec/changes/maker-archive-forwarder-conformance/proposal.md rename to openspec/changes/archive/2026-08-03-maker-archive-forwarder-conformance/proposal.md diff --git a/openspec/changes/maker-archive-forwarder-conformance/specs/archive-forwarder-durable-acceptance/spec.md b/openspec/changes/archive/2026-08-03-maker-archive-forwarder-conformance/specs/archive-forwarder-durable-acceptance/spec.md similarity index 100% rename from openspec/changes/maker-archive-forwarder-conformance/specs/archive-forwarder-durable-acceptance/spec.md rename to openspec/changes/archive/2026-08-03-maker-archive-forwarder-conformance/specs/archive-forwarder-durable-acceptance/spec.md diff --git a/openspec/changes/maker-archive-forwarder-conformance/specs/strategy-runtime-archive-ingestion/spec.md b/openspec/changes/archive/2026-08-03-maker-archive-forwarder-conformance/specs/strategy-runtime-archive-ingestion/spec.md similarity index 100% rename from openspec/changes/maker-archive-forwarder-conformance/specs/strategy-runtime-archive-ingestion/spec.md rename to openspec/changes/archive/2026-08-03-maker-archive-forwarder-conformance/specs/strategy-runtime-archive-ingestion/spec.md diff --git a/openspec/changes/maker-archive-forwarder-conformance/tasks.md b/openspec/changes/archive/2026-08-03-maker-archive-forwarder-conformance/tasks.md similarity index 94% rename from openspec/changes/maker-archive-forwarder-conformance/tasks.md rename to openspec/changes/archive/2026-08-03-maker-archive-forwarder-conformance/tasks.md index e1f6df5..520d396 100644 --- a/openspec/changes/maker-archive-forwarder-conformance/tasks.md +++ b/openspec/changes/archive/2026-08-03-maker-archive-forwarder-conformance/tasks.md @@ -44,7 +44,7 @@ ## 7. Coordinated Release And Maker Closure -- [ ] 7.1 Select the smallest unused patch above `0.2.36`, update CEX package/image metadata, and publish matching version/digest evidence after required Actions pass. -- [ ] 7.2 Update Maker's CEX Broker dependency and lockfile, run Maker strict checks, record CEX commit/PR/Actions/version/digest evidence, and complete Maker task 8.6. -- [ ] 7.3 Update FIET-937 with the retained production observation cutover gate, link FIET-901/FIET-903 to the scope transition, and comment FIET-909 with conformance evidence while leaving FIET-924 closed. -- [ ] 7.4 Sync and archive this change only after both repositories satisfy the cross-service definition of done. +- [x] 7.1 Select the smallest unused patch above `0.2.36`, update CEX package/image metadata, and publish matching version/digest evidence after required Actions pass. +- [x] 7.2 Update Maker's CEX Broker dependency and lockfile, run Maker strict checks, record CEX commit/PR/Actions/version/digest evidence, and complete Maker task 8.6. +- [x] 7.3 Update FIET-937 with the retained production observation cutover gate, link FIET-901/FIET-903 to the scope transition, and comment FIET-909 with conformance evidence while leaving FIET-924 closed. +- [x] 7.4 Sync and archive this change only after both repositories satisfy the cross-service definition of done. diff --git a/openspec/specs/archive-forwarder-durable-acceptance/spec.md b/openspec/specs/archive-forwarder-durable-acceptance/spec.md new file mode 100644 index 0000000..0ae0fc3 --- /dev/null +++ b/openspec/specs/archive-forwarder-durable-acceptance/spec.md @@ -0,0 +1,127 @@ +# archive-forwarder-durable-acceptance Specification + +## Purpose +TBD - created by archiving change maker-archive-forwarder-conformance. Update Purpose after archive. +## Requirements +### Requirement: Accepted Maker batches are durable before acknowledgement + +The forwarder MUST atomically persist a validated Maker envelope and one pending work item per represented strategy table before returning HTTP 202. It MUST NOT acknowledge acceptance when the spool transaction has not committed. + +#### Scenario: Strategy batch is durably admitted +- **WHEN** a conforming Maker batch commits to the SQLite spool +- **THEN** the forwarder MUST return HTTP 202 without waiting for ClickHouse + +#### Scenario: ClickHouse is unavailable during admission +- **WHEN** ClickHouse is unavailable but the spool is writable and within quota +- **THEN** the forwarder MUST still durably admit the batch and return HTTP 202 + +#### Scenario: Spool is unavailable +- **WHEN** the spool cannot open, write, or commit +- **THEN** the forwarder MUST return HTTP 503 and MUST NOT claim ownership + +### Requirement: Spool bounds are fixed and atomic + +The strategy spool SHALL use a fixed 1 GiB quota and fixed 72-hour retention. Admission accounting, expiry cleanup, and quota reservation MUST be serialized so concurrent requests cannot over-admit. Neither value SHALL be configurable through environment variables. + +#### Scenario: Admission fits quota +- **WHEN** the committed spool bytes plus the new batch's deterministic accounted bytes do not exceed 1 GiB +- **THEN** the batch MUST be eligible for atomic admission + +#### Scenario: Admission exceeds quota +- **WHEN** accepting a batch would exceed the fixed quota +- **THEN** the forwarder MUST return HTTP 429 without persisting any part of that batch + +#### Scenario: Work reaches retention limit +- **WHEN** pending or terminal work reaches 72 hours since admission +- **THEN** it MUST be expired transactionally and the loss MUST be recorded in bounded telemetry + +### Requirement: Spool storage is restart recoverable + +The forwarder MUST use Bun SQLite with WAL, foreign keys, a busy timeout, and full synchronous durability. `ARCHIVE_FORWARDER_SPOOL_PATH` SHALL select the database path and SHALL default to `./archive-forwarder-spool.sqlite`; production documentation MUST require persistent storage for that path. + +#### Scenario: Forwarder restarts with pending work +- **WHEN** a process exits after acknowledgement and restarts with the same spool path +- **THEN** every incomplete table work item MUST remain eligible for drainage + +#### Scenario: Local path is not configured +- **WHEN** the service starts without `ARCHIVE_FORWARDER_SPOOL_PATH` +- **THEN** it MUST use `./archive-forwarder-spool.sqlite` + +#### Scenario: SQLite is corrupt or read-only +- **WHEN** startup or a health transaction detects corruption or inability to write +- **THEN** strategy admission MUST fail closed with HTTP 503 and health MUST report the spool unhealthy + +### Requirement: Delivery retries are isolated and idempotent + +The worker MUST track completion independently for each table in a batch. Transient failures MUST retry after exponential delays starting at 1 second, doubling to a 60-second cap, with ±20 percent jitter until retention expiry. Successful siblings MUST NOT be retried. + +Each table work item MUST use one stable ClickHouse `insert_deduplication_token`, and each strategy table MUST enable non-replicated deduplication over a window large enough for the admitted delivery horizon. + +#### Scenario: One table fails after siblings succeed +- **WHEN** a multi-table batch inserts some table groups and one table returns a transient error +- **THEN** only the failed table work item MUST be rescheduled + +#### Scenario: Insert outcome is ambiguous across restart +- **WHEN** ClickHouse commits a table insert but the worker exits before recording completion +- **THEN** the restarted retry MUST use the same deduplication token and MUST not create duplicate logical delivery rows + +#### Scenario: Permanent ClickHouse failure occurs +- **WHEN** a table insert fails with a schema, authentication, or other classified permanent error +- **THEN** the work MUST become terminal, MUST NOT hot-loop, and MUST remain observable until expiry + +#### Scenario: All table work completes +- **WHEN** every represented table work item is marked complete +- **THEN** the spool MUST transactionally remove the completed batch and its work records + +### Requirement: Health reports acceptance capacity separately from ClickHouse drainage + +Health MUST expose spool writability, queued batches, queued table work, accounted bytes, oldest age, terminal/expired work, and a bounded last-error class. ClickHouse unavailability with a healthy spool MUST return HTTP 200 with degraded status; an unhealthy spool MUST return HTTP 503. + +#### Scenario: ClickHouse is down and spool is healthy +- **WHEN** the health endpoint cannot ping ClickHouse but can validate spool writability +- **THEN** it MUST return HTTP 200 and report degraded drainage with durable admission available + +#### Scenario: Spool is unhealthy +- **WHEN** the health endpoint cannot validate the spool +- **THEN** it MUST return HTTP 503 even if ClickHouse is reachable + +### Requirement: Durable acceptance emits bounded operational telemetry + +The forwarder MUST record bounded-cardinality counters and gauges for admitted/rejected strategy batches and rows, quota rejections, spool failures, pending work, accounted bytes, oldest age, retry attempts, table completions, terminal failures, expirations, and last successful drain. + +#### Scenario: Retry telemetry is emitted +- **WHEN** one table work item is rescheduled +- **THEN** retry metrics MUST use only approved table and bounded error-class labels + +#### Scenario: Untrusted request values are submitted +- **WHEN** a client supplies arbitrary source, table, deployment, or error text +- **THEN** unbounded values MUST NOT become persistent metric labels + +### Requirement: CI proves the durability and ClickHouse contract + +Required CI MUST run the legacy/v1/v2/mixed/unknown/missing-identity/all-five-table matrix, SQLite quota/expiry/restart/corruption/write-failure/partial-retry tests, and real ClickHouse 24.8 schema/insert/deduplication tests. ClickHouse integration tests MUST fail rather than skip when the required CI service is unavailable. + +#### Scenario: ClickHouse service is missing in CI +- **WHEN** the mandatory integration job cannot reach ClickHouse 24.8 +- **THEN** the job MUST fail + +#### Scenario: Durable fault matrix passes +- **WHEN** repository CI completes successfully +- **THEN** it MUST provide evidence for quota, retention, restart recovery, partial retry isolation, spool failure, and deduplicated real inserts + +### Requirement: Release evidence closes the cross-service contract + +CEX Broker and Maker completion MUST reference a published CEX Broker package and image with the same unused patch version above `0.2.36`, an immutable image digest, the CEX PR and required Actions evidence, Maker's updated dependency and lockfile, and strict validation in both repositories. + +#### Scenario: CEX conformance release is published +- **WHEN** the CEX implementation and required CI pass +- **THEN** package and image evidence MUST identify the same version and immutable digest + +#### Scenario: Maker task 8.6 is completed +- **WHEN** Maker consumes the conformance release and its strict checks pass +- **THEN** its OpenSpec evidence MUST record the CEX commit/PR/Actions/version/digest and task 8.6 MAY be marked complete + +#### Scenario: Production observation has not run +- **WHEN** repository conformance is complete but FIET-937's observation window remains pending +- **THEN** this change MAY complete while FIET-937 continues to block deployment cutover + diff --git a/openspec/specs/strategy-runtime-archive-ingestion/spec.md b/openspec/specs/strategy-runtime-archive-ingestion/spec.md new file mode 100644 index 0000000..9faea6c --- /dev/null +++ b/openspec/specs/strategy-runtime-archive-ingestion/spec.md @@ -0,0 +1,99 @@ +# strategy-runtime-archive-ingestion Specification + +## Purpose +TBD - created by archiving change maker-archive-forwarder-conformance. Update Purpose after archive. +## Requirements +### Requirement: Strategy runtime requests use a closed envelope contract + +The archive forwarder MUST accept a strategy runtime request only when its non-empty envelope source is `hb_runtime`, its deployment id is non-empty, it contains between one and 1000 rows, and every row targets an approved strategy runtime table. + +The approved table set SHALL be exactly `strategy_data.policy_evaluation_events`, `strategy_data.strategy_policy_snapshots`, `strategy_data.market_identity`, `strategy_data.symbol_mapping`, and `strategy_data.inventory_settlement_events`. + +#### Scenario: All five strategy tables are submitted +- **WHEN** one `hb_runtime` envelope contains conforming rows for all five approved tables +- **THEN** the request MUST pass strategy contract validation + +#### Scenario: Strategy and non-strategy rows are mixed +- **WHEN** an `hb_runtime` envelope contains an approved strategy table and any non-strategy table +- **THEN** the entire request MUST be rejected with HTTP 400 before durable admission or ClickHouse insertion + +#### Scenario: Strategy table uses another source +- **WHEN** an envelope under `broker_read`, `broker_write`, or another source targets an approved strategy table +- **THEN** the entire request MUST be rejected with HTTP 400 + +#### Scenario: Envelope identity is empty +- **WHEN** source or deployment id is missing, empty, or whitespace-only +- **THEN** the request MUST be rejected with HTTP 400 + +### Requirement: Strategy schema versions are validated before ownership + +Each strategy row MUST use a missing/empty legacy version, version `1`, or version `2`. Unknown versions MUST be rejected with HTTP 400 before the forwarder accepts ownership. + +Version `2` rows MUST contain non-empty `producer_id`, `producer_run_id`, `stream_name`, and `archive_event_id`, and positive UInt64-compatible integer `stream_seq` and `seq` values. + +#### Scenario: Legacy schema version is omitted +- **WHEN** a conforming strategy row omits `schema_version` or supplies an empty value +- **THEN** the forwarder MUST admit it under the legacy compatibility contract + +#### Scenario: Version one row is submitted +- **WHEN** a conforming strategy row supplies `schema_version=1` +- **THEN** the forwarder MUST admit it without requiring v2 producer and stream fields + +#### Scenario: Version two identity is complete +- **WHEN** a version `2` row supplies every required producer, stream, event, and sequence identity +- **THEN** the forwarder MUST admit it + +#### Scenario: Version two identity is incomplete +- **WHEN** any required v2 identity is blank, missing, non-integral, zero, negative, or outside UInt64 range +- **THEN** the entire request MUST be rejected with HTTP 400 + +#### Scenario: Unknown version is submitted +- **WHEN** a strategy row supplies any schema version other than legacy/empty, `1`, or `2` +- **THEN** the entire request MUST be rejected with HTTP 400 + +### Requirement: Envelope and row provenance agree + +When a strategy row supplies `source` or `deployment_id`, its value MUST equal the corresponding envelope value. Credentials and broker configuration MUST NOT supply or override these Maker identities. + +#### Scenario: Row source differs from envelope +- **WHEN** a row declares a source other than the envelope source +- **THEN** the entire request MUST be rejected before admission + +#### Scenario: Row deployment differs from envelope +- **WHEN** a row declares a deployment id other than the envelope deployment id +- **THEN** the entire request MUST be rejected before admission + +### Requirement: Strategy tables implement the pinned additive schema + +All five strategy tables MUST expose compatibility-defaulted `producer_id`, `producer_run_id`, `stream_name`, `stream_seq`, `seq`, and `archive_event_id` columns. Policy rows MUST expose the pinned policy decision fields, and policy snapshots, market identity, and symbol mapping MUST expose the pinned content/revision/market mapping fields. + +DDL upgrades MUST use additive idempotent migration statements so existing v1 data remains readable and a new deployment writes the latest schema. + +#### Scenario: Existing v1 tables are upgraded +- **WHEN** schema initialization runs against strategy tables created by the prior release +- **THEN** every pinned v2 column MUST be added without rewriting or dropping historical rows + +#### Scenario: Fresh strategy database is initialized +- **WHEN** schema initialization runs on an empty ClickHouse 24.8 database +- **THEN** all five tables MUST be insert-compatible with the pinned Maker v2 rows + +### Requirement: The Maker fixture is the cross-repository wire evidence + +The CEX Broker contract fixture MUST equal the pinned Maker `archive_forwarder_envelope.json` and contract tests MUST validate every fixture row without local field translation. + +#### Scenario: Pinned Maker fixture is checked in CEX Broker +- **WHEN** the cross-repository contract test runs +- **THEN** it MUST prove fixture equality and successful v2 validation + +#### Scenario: Fixture contract drifts +- **WHEN** Maker or CEX changes a table, schema version, or required identity without the matching repository change +- **THEN** a required contract check MUST fail + +### Requirement: Non-strategy archive behavior remains compatible + +Requests that contain no strategy table and do not use `hb_runtime` MUST retain the existing direct ClickHouse insertion and synchronous success/failure contract. + +#### Scenario: Broker market rows are submitted +- **WHEN** a valid `broker_read` or `broker_write` request contains only supported non-strategy rows +- **THEN** the forwarder MUST insert them through the existing direct path and MUST NOT consume strategy spool quota +