Skip to content

Archive durable fill and transfer execution facts - #61

Merged
outerlook merged 1 commit into
developfrom
archive-fills-transfers
Jul 7, 2026
Merged

Archive durable fill and transfer execution facts#61
outerlook merged 1 commit into
developfrom
archive-fills-transfers

Conversation

@outerlook

Copy link
Copy Markdown
Member

Summary

Adds durable CEX execution facts for the two grains that today require repeated, truncation-prone venue-history scans: broker_execution.transfer_events and broker_execution.fill_events, end to end (row builders, emit sites, forwarder support, schema).

transfer_events

  • Withdrawals emit at the execute-action site, including the venue fee the ccxt transaction already carries (previously discarded) and failed attempts (status="failed" with error_summary).
  • Observed deposits emit from the deposit polling path; sub↔master internal transfers emit as event_kind="internal_transfer".
  • Withdrawal fees dominate measurable cost at small commits, and deposit-account attribution enables cheap wrong-account-credit detection — both currently reconstructed via paginated venue APIs.

fill_events

  • Production createOrder/GetOrderDetails responses carry no per-trade data and no fees (verified against live archive rows: 0 of ~520k detail rows have trades[] or fee fields), so per-fill truth requires the trade-history endpoint.
  • A FillArchivePoller (constructed only when archiving is enabled; mirrors the deposit-reconciler lifecycle) polls ccxt trade history per (account, symbol) seen through order activity, with an in-memory since-cursor. Restart re-scans a bounded lookback; duplicates resolve at read time (plain MergeTree, per the consumer contract). Symbols poll sequentially to stagger venue calls; per-symbol failures are isolated. Enclave-safe (node http/https only).

Contract alignment

Column shapes follow fiet-maker's docs/CEX_EXECUTION_ARCHIVE_CONTRACT.md (and its sandbox proof harness); a test asserts a fully-populated row covers every contract column for both tables. Deliberate divergences are additive only:

  • transfer_events gains fee_amount/fee_currency (contract lacks them; ccxt provides them).
  • fill_events.event_kind = trade_history_fill (the contract's create_order_fill grain would yield zero rows in production, per the finding above); fill_index is the trade's position within the poll batch.
  • internal_transfer / submit_internal_transfer values in existing enum-style columns; withdrawal failures use the contract's error_summary.

These should be reflected back into the contract doc/fixture on the fiet-maker side.

Archive self-health

  • Writer now emits cex_archive_rows_flushed_total and a cex_archive_last_flush_success gauge (the enqueued/shed/failure counters existed but flushed/last-success did not).
  • The forwarder names rejected tables in its warning log and response body.

Verification

bun test: 405 pass / 0 fail; tsc --noEmit clean; biome clean. New tests cover builder shapes against the consumer contract, fee-as-string precision preference, forwarder acceptance of the new tables + rejected-table naming, order-activity tracking, cursor advancement, and poller failure isolation.

Rollout

  • Broker and forwarder must release together: the forwarder 400-rejects any batch containing an unknown table and the writer requeues rejected batches indefinitely, so a broker emitting the new tables against an old forwarder poison-pills its archive queue.
  • No new environment variables — poller tuning is constants — so the SGX/Gramine manifest allowlist is unchanged; the usual enclave redeploy applies.

Add broker_execution.transfer_events and broker_execution.fill_events
end to end: row builders, emit sites, forwarder support, and schema.

Transfers: withdrawals (including the venue fee ccxt already returns,
previously discarded, and failed attempts with error_summary), observed
deposits, and sub<->master internal transfers.

Fills: a FillArchivePoller drives ccxt trade-history per (account,
symbol) seen through order activity, with an in-memory since-cursor.
Venue createOrder/GetOrderDetails responses carry no per-trade data in
production, so trade-history polling is the only source of per-fill
fees. Restart re-scans a bounded lookback; duplicates resolve at read
time (plain MergeTree, per the consumer contract).

Column shapes follow the fiet-maker CEX execution archive contract;
divergences are additive only (transfer fee columns, trade_history_fill
event kind, internal_transfer lifecycle values).

Archive self-health: writer now emits cex_archive_rows_flushed_total and
a last-flush-success gauge; the forwarder names rejected tables in its
warning and response body. Unknown tables 400-reject the whole batch and
requeue, so broker and forwarder must release together.

No new environment variables (poller tuning is constants), keeping the
Gramine manifest allowlist unchanged.
@coderabbitai

coderabbitai Bot commented Jul 7, 2026

Copy link
Copy Markdown

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your recent review volume is higher than typical usage, so adaptive limits are currently applied.

Next review available in: 59 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 0289e52b-8575-4c4d-9b1a-28af6c3db7fa

📥 Commits

Reviewing files that changed from the base of the PR and between f6b5ed7 and 84b591e.

📒 Files selected for processing (22)
  • schema/clickhouse/broker_execution.sql
  • services/archive-forwarder/index.ts
  • services/archive-forwarder/router.ts
  • services/archive-forwarder/types.ts
  • src/handlers/execute-action/context.ts
  • src/handlers/execute-action/deposit.ts
  • src/handlers/execute-action/handler.ts
  • src/handlers/execute-action/internal-transfer.ts
  • src/handlers/execute-action/orders.ts
  • src/handlers/execute-action/withdraw.ts
  • src/helpers/broker-execution-archive/capture.ts
  • src/helpers/broker-execution-archive/index.ts
  • src/helpers/broker-execution-archive/rows.ts
  • src/helpers/broker-execution-archive/types.ts
  • src/helpers/broker-execution-archive/writer.ts
  • src/helpers/fill-archive-poller.ts
  • src/helpers/order-activity-tracker.ts
  • src/index.ts
  • src/server.ts
  • test/archive-forwarder.test.ts
  • test/broker-execution-archive.test.ts
  • test/fill-archive-poller.test.ts
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch archive-fills-transfers

Comment @coderabbitai help to get the list of available commands.

@outerlook
outerlook merged commit abca49c into develop Jul 7, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant