docs(openspec): archive Maker forwarder conformance

openspec/changes/archive/2026-08-03-maker-archive-forwarder-conformance/implementation-evidence.md

@@ -0,0 +1,91 @@
+# Implementation evidence
+
+## Released CEX boundary
+
+The durable Maker acceptance boundary shipped through
+[usherlabs/cex-broker#99](https://github.com/usherlabs/cex-broker/pull/99).
+
+| Evidence | Immutable reference |
+|---|---|
+| Reviewed CEX change commit | `1bd2af1ef7ec7d4fc9d223cdcc9211c57d600526` |
+| CEX merge commit | `5eec63dfcd947aae3b0f861d39cf96e05339cd0c` |
+| Release | `v0.2.37`, dereferencing to the merge commit |
+| npm integrity | `sha512-e4Cq665s3Be2E6QMADWLzIvXgj3Vd/PmLOrOCtkngCt3FjKdgE4SmHqoRDhMz2/reSv2ttfePclHVqG05s1CeA==` |
+| Archive-forwarder image | `ghcr.io/usherlabs/cex-broker-archive-forwarder:0.2.37@sha256:713068ff7bc7b75640ea990baa68c317876d8efdd3f312cfa38b37a799eb8a44` |
+| Core broker image | `ghcr.io/usherlabs/cex-broker:0.2.37@sha256:cbc959948f9b6caed2759ca40f3716af2354b12814078368aba73d22c14577b3` |
+| Pinned Maker producer commit | `563594435853c88cca5b187b8c999f845e31136b` |
+| Shared fixture SHA-256 | `784f647e048052a6c3382309b1a86abfbe08bc162363ead9fc88eaa1ba3d50c9` |
+
+Required CEX evidence passed:
+
+- [PR CI](https://github.com/usherlabs/cex-broker/actions/runs/30792787681)
+  used ClickHouse 24.8 and passed contract, SQLite durability/fault/restart,
+  legacy upgrade, all-five-table schema-v2, and stable-token deduplication
+  coverage;
+- [post-merge CI](https://github.com/usherlabs/cex-broker/actions/runs/30792875090)
+  passed;
+- [package and core-image publication](https://github.com/usherlabs/cex-broker/actions/runs/30792910567)
+  passed; and
+- [archive-forwarder image publication](https://github.com/usherlabs/cex-broker/actions/runs/30792910562)
+  passed.
+
+Local verification before release recorded 593 repository tests passing with no
+failures, 48 focused forwarder tests, 12 real ClickHouse integration tests,
+type-check, build, lint, Compose validation, strict OpenSpec validation, and a
+clean diff check.
+
+## Maker consumer closure
+
+Maker consumed the release through
+[usherlabs/fiet-maker#1067](https://github.com/usherlabs/fiet-maker/pull/1067).
+The dependency/evidence commit is
+`f84ebf3291a7b0060b9d90c44ea785d9ad6ea820`; the merge commit is
+`1ae1d0faf87396a35089ec51eedc55ff65d675d9`.
+
+Maker's production build authority is its parent `pnpm-workspace.yaml` and root
+`pnpm-lock.yaml`. They pin the workspace's wrapper resolution to exact
+`@usherlabs/cex-broker@0.2.37` and the npm integrity above. The separately
+versioned fiet-tee submodule manifest and lock remain standalone repository
+release inputs; they do not override the parent production workspace pin.
+
+The installed wrapper passed 18 tests, the SGX archive-storage path contract,
+and the CommonJS production bundle check against `0.2.37`. Maker strict
+OpenSpec validation passed, all 64 tasks in
+`complete-nonblocking-strategy-replay-archive` are complete, and the shared
+fixture hash is identical. Maker post-merge advisory evidence is
+[run 30793571785](https://github.com/usherlabs/fiet-maker/actions/runs/30793571785);
+the production-context CEX SGX staging build is
+[run 30793571854](https://github.com/usherlabs/fiet-maker/actions/runs/30793571854).
+Both runs passed.
+
+## Linear scope transition
+
+- [FIET-901](https://linear.app/usherlabs/issue/FIET-901/read-broker-cex-market-data-into-clickhouse)
+  records the released implementation, same-full-broker deployment model,
+  credential precedence, absence of credential profiles/attestation knobs, and
+  the FIET-937/FIET-907 handoff while remaining open/In Review.
+- [FIET-903](https://linear.app/usherlabs/issue/FIET-903/canonical-cex-order-book-depth-tables-for-strategy-runtime-replay)
+  records the already-complete RPC/capability sub-scope, released storage
+  sub-scope, sampled-mode closure, and future-facing exact-L2 guardrail while
+  remaining open/In Review.
+- [FIET-909](https://linear.app/usherlabs/issue/FIET-909/hb-runtime-telemetry-bridge-for-strategy-replay-sourcing)
+  records the Maker/CEX conformance release and immutable evidence.
+- [FIET-937](https://linear.app/usherlabs/issue/FIET-937/production-cex-market-data-archive-into-clickhouse-for-strategy-replay)
+  retains the production observation window and go/no-go cutover decision.
+- [FIET-924](https://linear.app/usherlabs/issue/FIET-924/review-and-implementation-of-data-warehouse-ingestion-packagetemplate)
+  remains Done and was not reopened or repurposed.
+
+## Ownership result
+
+- Maker makes exactly one bounded HTTP attempt per admitted strategy batch.
+- HTTP 202 means the complete batch is durably owned by the forwarder's fixed
+  1 GiB, 72-hour SQLite spool.
+- Retry, restart recovery, per-table completion, expiry, and ClickHouse
+  deduplication belong to the forwarder after acceptance.
+- Broker-origin traffic keeps its existing direct synchronous path.
+- No new core full-broker `CEX_BROKER_*` environment variable was introduced;
+  only forwarder-local `ARCHIVE_FORWARDER_SPOOL_PATH` selects the durable file.
+- Production observation remains a deployment/cutover gate owned by FIET-937,
+  not an OpenSpec implementation or archive prerequisite.
+- Parquet materialization remains an out-of-band ClickHouse consumer owned by
+  FIET-907.

openspec/changes/maker-archive-forwarder-conformance/tasks.md → openspec/changes/archive/2026-08-03-maker-archive-forwarder-conformance/tasks.md

@@ -44,7 +44,7 @@
 
 ## 7. Coordinated Release And Maker Closure
 
-- [ ] 7.1 Select the smallest unused patch above `0.2.36`, update CEX package/image metadata, and publish matching version/digest evidence after required Actions pass.
-- [ ] 7.2 Update Maker's CEX Broker dependency and lockfile, run Maker strict checks, record CEX commit/PR/Actions/version/digest evidence, and complete Maker task 8.6.
-- [ ] 7.3 Update FIET-937 with the retained production observation cutover gate, link FIET-901/FIET-903 to the scope transition, and comment FIET-909 with conformance evidence while leaving FIET-924 closed.
-- [ ] 7.4 Sync and archive this change only after both repositories satisfy the cross-service definition of done.
+- [x] 7.1 Select the smallest unused patch above `0.2.36`, update CEX package/image metadata, and publish matching version/digest evidence after required Actions pass.
+- [x] 7.2 Update Maker's CEX Broker dependency and lockfile, run Maker strict checks, record CEX commit/PR/Actions/version/digest evidence, and complete Maker task 8.6.
+- [x] 7.3 Update FIET-937 with the retained production observation cutover gate, link FIET-901/FIET-903 to the scope transition, and comment FIET-909 with conformance evidence while leaving FIET-924 closed.
+- [x] 7.4 Sync and archive this change only after both repositories satisfy the cross-service definition of done.

openspec/specs/archive-forwarder-durable-acceptance/spec.md

@@ -0,0 +1,127 @@
+# archive-forwarder-durable-acceptance Specification
+
+## Purpose
+TBD - created by archiving change maker-archive-forwarder-conformance. Update Purpose after archive.
+## Requirements
+### Requirement: Accepted Maker batches are durable before acknowledgement
+
+The forwarder MUST atomically persist a validated Maker envelope and one pending work item per represented strategy table before returning HTTP 202. It MUST NOT acknowledge acceptance when the spool transaction has not committed.
+
+#### Scenario: Strategy batch is durably admitted
+- **WHEN** a conforming Maker batch commits to the SQLite spool
+- **THEN** the forwarder MUST return HTTP 202 without waiting for ClickHouse
+
+#### Scenario: ClickHouse is unavailable during admission
+- **WHEN** ClickHouse is unavailable but the spool is writable and within quota
+- **THEN** the forwarder MUST still durably admit the batch and return HTTP 202
+
+#### Scenario: Spool is unavailable
+- **WHEN** the spool cannot open, write, or commit
+- **THEN** the forwarder MUST return HTTP 503 and MUST NOT claim ownership
+
+### Requirement: Spool bounds are fixed and atomic
+
+The strategy spool SHALL use a fixed 1 GiB quota and fixed 72-hour retention. Admission accounting, expiry cleanup, and quota reservation MUST be serialized so concurrent requests cannot over-admit. Neither value SHALL be configurable through environment variables.
+
+#### Scenario: Admission fits quota
+- **WHEN** the committed spool bytes plus the new batch's deterministic accounted bytes do not exceed 1 GiB
+- **THEN** the batch MUST be eligible for atomic admission
+
+#### Scenario: Admission exceeds quota
+- **WHEN** accepting a batch would exceed the fixed quota
+- **THEN** the forwarder MUST return HTTP 429 without persisting any part of that batch
+
+#### Scenario: Work reaches retention limit
+- **WHEN** pending or terminal work reaches 72 hours since admission
+- **THEN** it MUST be expired transactionally and the loss MUST be recorded in bounded telemetry
+
+### Requirement: Spool storage is restart recoverable
+
+The forwarder MUST use Bun SQLite with WAL, foreign keys, a busy timeout, and full synchronous durability. `ARCHIVE_FORWARDER_SPOOL_PATH` SHALL select the database path and SHALL default to `./archive-forwarder-spool.sqlite`; production documentation MUST require persistent storage for that path.
+
+#### Scenario: Forwarder restarts with pending work
+- **WHEN** a process exits after acknowledgement and restarts with the same spool path
+- **THEN** every incomplete table work item MUST remain eligible for drainage
+
+#### Scenario: Local path is not configured
+- **WHEN** the service starts without `ARCHIVE_FORWARDER_SPOOL_PATH`
+- **THEN** it MUST use `./archive-forwarder-spool.sqlite`
+
+#### Scenario: SQLite is corrupt or read-only
+- **WHEN** startup or a health transaction detects corruption or inability to write
+- **THEN** strategy admission MUST fail closed with HTTP 503 and health MUST report the spool unhealthy
+
+### Requirement: Delivery retries are isolated and idempotent
+
+The worker MUST track completion independently for each table in a batch. Transient failures MUST retry after exponential delays starting at 1 second, doubling to a 60-second cap, with ±20 percent jitter until retention expiry. Successful siblings MUST NOT be retried.
+
+Each table work item MUST use one stable ClickHouse `insert_deduplication_token`, and each strategy table MUST enable non-replicated deduplication over a window large enough for the admitted delivery horizon.
+
+#### Scenario: One table fails after siblings succeed
+- **WHEN** a multi-table batch inserts some table groups and one table returns a transient error
+- **THEN** only the failed table work item MUST be rescheduled
+
+#### Scenario: Insert outcome is ambiguous across restart
+- **WHEN** ClickHouse commits a table insert but the worker exits before recording completion
+- **THEN** the restarted retry MUST use the same deduplication token and MUST not create duplicate logical delivery rows
+
+#### Scenario: Permanent ClickHouse failure occurs
+- **WHEN** a table insert fails with a schema, authentication, or other classified permanent error
+- **THEN** the work MUST become terminal, MUST NOT hot-loop, and MUST remain observable until expiry
+
+#### Scenario: All table work completes
+- **WHEN** every represented table work item is marked complete
+- **THEN** the spool MUST transactionally remove the completed batch and its work records
+
+### Requirement: Health reports acceptance capacity separately from ClickHouse drainage
+
+Health MUST expose spool writability, queued batches, queued table work, accounted bytes, oldest age, terminal/expired work, and a bounded last-error class. ClickHouse unavailability with a healthy spool MUST return HTTP 200 with degraded status; an unhealthy spool MUST return HTTP 503.
+
+#### Scenario: ClickHouse is down and spool is healthy
+- **WHEN** the health endpoint cannot ping ClickHouse but can validate spool writability
+- **THEN** it MUST return HTTP 200 and report degraded drainage with durable admission available
+
+#### Scenario: Spool is unhealthy
+- **WHEN** the health endpoint cannot validate the spool
+- **THEN** it MUST return HTTP 503 even if ClickHouse is reachable
+
+### Requirement: Durable acceptance emits bounded operational telemetry
+
+The forwarder MUST record bounded-cardinality counters and gauges for admitted/rejected strategy batches and rows, quota rejections, spool failures, pending work, accounted bytes, oldest age, retry attempts, table completions, terminal failures, expirations, and last successful drain.
+
+#### Scenario: Retry telemetry is emitted
+- **WHEN** one table work item is rescheduled
+- **THEN** retry metrics MUST use only approved table and bounded error-class labels
+
+#### Scenario: Untrusted request values are submitted
+- **WHEN** a client supplies arbitrary source, table, deployment, or error text
+- **THEN** unbounded values MUST NOT become persistent metric labels
+
+### Requirement: CI proves the durability and ClickHouse contract
+
+Required CI MUST run the legacy/v1/v2/mixed/unknown/missing-identity/all-five-table matrix, SQLite quota/expiry/restart/corruption/write-failure/partial-retry tests, and real ClickHouse 24.8 schema/insert/deduplication tests. ClickHouse integration tests MUST fail rather than skip when the required CI service is unavailable.
+
+#### Scenario: ClickHouse service is missing in CI
+- **WHEN** the mandatory integration job cannot reach ClickHouse 24.8
+- **THEN** the job MUST fail
+
+#### Scenario: Durable fault matrix passes
+- **WHEN** repository CI completes successfully
+- **THEN** it MUST provide evidence for quota, retention, restart recovery, partial retry isolation, spool failure, and deduplicated real inserts
+
+### Requirement: Release evidence closes the cross-service contract
+
+CEX Broker and Maker completion MUST reference a published CEX Broker package and image with the same unused patch version above `0.2.36`, an immutable image digest, the CEX PR and required Actions evidence, Maker's updated dependency and lockfile, and strict validation in both repositories.
+
+#### Scenario: CEX conformance release is published
+- **WHEN** the CEX implementation and required CI pass
+- **THEN** package and image evidence MUST identify the same version and immutable digest
+
+#### Scenario: Maker task 8.6 is completed
+- **WHEN** Maker consumes the conformance release and its strict checks pass
+- **THEN** its OpenSpec evidence MUST record the CEX commit/PR/Actions/version/digest and task 8.6 MAY be marked complete
+
+#### Scenario: Production observation has not run
+- **WHEN** repository conformance is complete but FIET-937's observation window remains pending
+- **THEN** this change MAY complete while FIET-937 continues to block deployment cutover
+