Skip to content

Bump master for telemtry, in-TEE proto handling, etc. - #42

Merged
rsoury merged 182 commits into
masterfrom
develop
Aug 2, 2026
Merged

Bump master for telemtry, in-TEE proto handling, etc.#42
rsoury merged 182 commits into
masterfrom
develop

Conversation

@rsoury

@rsoury rsoury commented May 14, 2026

Copy link
Copy Markdown
Member

Summary by CodeRabbit

  • New Features
    • Order-execution OpenTelemetry telemetry (non-blocking counters/histograms) plus expanded tests/fixtures
    • Order-book Call APIs (capability/current snapshot/historical unsupported responses) and richer Subscribe behavior with improved marketType/perp config support
    • Wallet-based exchange credential support and improved broker/credential selection
  • Refactor
    • Server modularization (handlers/helpers/transport/wiring split) and shared proto-loading configuration
  • Documentation
    • Updated README with broker configs, Call/stream semantics, telemetry guidance, and project structure
    • Updated AGENTS/POLICY and added OpenSpec/spec docs (order-book depth sourcing, modularization)
  • Tests
    • New/expanded Bun test suites for telemetry, order-book RPC/compatibility, proto loading, auth, subscriptions, perp config, and treasury/discovery flows
  • Chores
    • Version bump, CI trigger adjustment, .gitignore rule, and server line-budget script

outerlook and others added 7 commits April 27, 2026 10:11
Share the proto-loader option set across descriptor and proto loads so
the bundled server matches existing runtime materialization behavior.

Normalize enum names at the server boundary because enums are now
deserialized as strings, and cover descriptor defaults, enums, and longs
with a focused regression test.
Validate numeric action and subscription inputs against the enum values before
accepting them. Use own-property checks for string names so inherited keys fall
through to the existing unknown labels and subscription fallback behavior.
Switch gRPC schema loading from runtime protobuf files to precompiled descriptor
feat: capture CEX order execution telemetry
@coderabbitai

coderabbitai Bot commented May 14, 2026

Copy link
Copy Markdown

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

Walkthrough

Extracts server.ts into modular handlers and helpers, adds proto descriptor/package loading, expands order-book, perp, telemetry, and streaming behavior, and updates docs, tests, CI, and packaging.

Changes

Proto, docs, config, and scaffolding

Layer / File(s) Summary
Proto, docs, config, and scaffolding
.gitignore, README.md, AGENTS.md, POLICY.md, package.json, biome.json, .github/workflows/ci.yml, Dockerfile, scripts/check-server-line-budget.sh, policy/policy.binance-mexc-usdc-bep20.example.json, openspec/..., examples/kraken-orderbook-demo.ts, src/proto-loader-options.ts, src/proto-package-definition.ts, src/proto/node.descriptor.ts, src/proto/node.proto
Adds shared proto loading, generated descriptor packaging, updated repo docs and policy examples, CI and packaging metadata, and supporting scaffolding files.
Shared helpers and broker contracts
src/helpers/*, src/schemas/action-payloads.ts
Adds shared helper modules for auth, broker pooling/selection, gRPC payload/status helpers, market/order-book normalization, telemetry, transfer-network discovery, treasury discovery, verity HTTP overrides, and related schema updates.
ExecuteAction modular handlers and subscribe runtime
src/handlers/execute-action/*, src/handlers/subscribe/*, src/helpers/binance-user-data-stream.ts, src/server.ts
Extracts execute-action dispatch/handlers, adds subscribe stream handling with Binance user-data support, and rewires server registration to the modular handlers.
Tests and runtime validation
test/*
Adds and updates unit/integration coverage for proto loading, helper behavior, action dispatch, order-book RPCs, treasury/deposit flows, perp config, telemetry, auth, and subscribe streaming.

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant Server
  participant ExecuteActionHandler
  participant ActionRegistry
  participant Broker
  Client->>Server: ExecuteAction request
  Server->>ExecuteActionHandler: createExecuteActionHandler(...)
  ExecuteActionHandler->>ActionRegistry: dispatchExecuteAction(ctx)
  ActionRegistry->>Broker: invoke action handler
  Broker-->>ExecuteActionHandler: result or error
  ExecuteActionHandler-->>Client: wrappedCallback(response)
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

Suggested reviewers

  • outerlook
  • victorshevtsov

Poem

🐰 I hopped through handlers, neat and trim,
With proto paws and telemetry hymns.
Streams and orders race in line,
While helpers burrow, bright and fine. 🥕

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch develop

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/helpers/order-telemetry.ts`:
- Line 83: The telemetry payload logged via log.info("CEX market action
execution telemetry", telemetry) currently includes raw upstream error text
(error.message stored earlier), so before emitting telemetry remove or redact
the error.message field on the telemetry object (or replace it with a generic
code/value like "redacted_error" or an errorCode) — locate where telemetry is
assembled and where error.message is assigned and ensure you strip or replace
that property prior to calling log.info so structured telemetry never contains
raw exchange error strings.

In `@src/server.ts`:
- Around line 882-897: The RPC path is blocking on emitOrderExecutionTelemetry;
change the three await usages to fire-and-forget so telemetry won't delay
responses: replace awaiting calls to emitOrderExecutionTelemetry(otelMetrics,
{...}, order) with non-blocking invocations such as void
emitOrderExecutionTelemetry(otelMetrics, {...}, order).catch(err =>
processLogger.warn("Telemetry emit failed", err)) (or equivalent error handling)
so errors are surfaced but do not block the RPC; apply the same change for each
occurrence where emitOrderExecutionTelemetry is currently awaited.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 2ec8dab9-e263-44ab-99df-4ce7a782082b

📥 Commits

Reviewing files that changed from the base of the PR and between 632d3da and dfcd5fd.

📒 Files selected for processing (16)
  • .gitignore
  • README.md
  • examples/kraken-orderbook-demo.ts
  • package.json
  • src/helpers/constants.ts
  • src/helpers/order-telemetry.ts
  • src/proto-loader-options.ts
  • src/proto-package-definition.ts
  • src/proto/node.descriptor.ts
  • src/server.ts
  • test/helpers-constants.test.ts
  • test/internal-transfer-rpc.test.ts
  • test/order-telemetry-fixtures.ts
  • test/order-telemetry.test.ts
  • test/proto-descriptor.test.ts
  • test/server-proto-loader-options.test.ts
📜 Review details
🧰 Additional context used
🧠 Learnings (1)
📚 Learning: 2026-04-14T06:47:01.283Z
Learnt from: csmithington
Repo: usherlabs/cex-broker PR: 38
File: src/client.dev.ts:0-0
Timestamp: 2026-04-14T06:47:01.283Z
Learning: In this codebase, gRPC action constants such as `FetchTicker`, `FetchFees`, and `FetchAccountId` should be sourced from `src/helpers/constants.ts` and imported from there (e.g., used by both `src/client.dev.ts` and `src/server.ts`). Do not import or reference generated proto TypeScript artifacts (for example `./proto/cex_broker/Action`), since those generated files are git-ignored and won’t be available/committed consistently.

Applied to files:

  • src/proto-loader-options.ts
  • src/proto-package-definition.ts
  • src/proto/node.descriptor.ts
  • src/helpers/constants.ts
  • src/helpers/order-telemetry.ts
  • src/server.ts
🪛 LanguageTool
README.md

[grammar] ~546-~546: Use a hyphen to join words.
Context: ...change timestamp when present and broker observed timestamp Use metrics for aggr...

(QB_NEW_EN_HYPHEN)

🔇 Additional comments (13)
package.json (1)

3-3: LGTM!

src/proto-loader-options.ts (1)

1-9: LGTM!

examples/kraken-orderbook-demo.ts (1)

9-9: LGTM!

Also applies to: 16-19

src/proto/node.descriptor.ts (1)

1-127: LGTM!

test/proto-descriptor.test.ts (1)

1-10: LGTM!

src/proto-package-definition.ts (1)

1-8: LGTM!

.gitignore (1)

39-39: LGTM!

test/server-proto-loader-options.test.ts (1)

1-54: LGTM!

src/helpers/constants.ts (1)

44-110: LGTM!

test/helpers-constants.test.ts (1)

4-45: LGTM!

test/internal-transfer-rpc.test.ts (1)

6-13: LGTM!

test/order-telemetry-fixtures.ts (1)

1-135: LGTM!

test/order-telemetry.test.ts (1)

1-370: LGTM!

Comment thread src/helpers/order-telemetry.ts
Comment thread src/server.ts Outdated
csmithington and others added 21 commits May 14, 2026 04:13
…ew' (#3) from feat/create-a-new-pr-for-cex-broker-enable-github-ci-on-develop-a into develop

Reviewed-on: https://git.aos.usher.so/ao/cex-broker/pulls/3
Route Binance BALANCE and ORDERS Subscribe requests through a signature-based Spot WebSocket API user-data adapter instead of CCXT watchBalance/watchOrders.
Normalize Subscribe exchange lookup and install stream lifecycle handling before long-running subscription work starts.
Add focused Subscribe tests for Binance balance and order frames, uppercase exchange lookup, secondary account selection, and avoiding the retired listenKey-backed watch paths.
Document Subscribe enum values to match the proto/constants contract, including BALANCE and ORDERS account streams.
Describe Binance account streams as using the WebSocket API user-data subscription path without Spot listenKey REST lifecycle.
Refresh the checked-in proto descriptor generated by the build workflow.
Allow callable treasury methods when capability metadata is missing while still rejecting methods explicitly marked unsupported.
Add regression coverage for missing capability metadata and explicit false capability handling.
Default missing, empty, and unknown deposit statuses to pending while preserving explicit terminal status mappings.
Report observed deposit address mismatches with the deposit_address_mismatch prefix and remove unused handler bindings.
Add focused regression coverage for deposit status normalization and address mismatch RPC errors, with related lint-only cleanup.
End subscribe streams after writing per-subscription error responses and outer internal-error responses.
Report the resolved subscription type from the outer subscribe error path.
Add focused subscribe handler coverage for stream closure and outer error type reporting.
outerlook added 29 commits July 29, 2026 14:34
Archive the authoring component on order events
Archive rows are delivered at-most-once over a bounded queue, so a lost row
and an event that never happened are currently indistinguishable. Add `seq`,
a monotonic per-(controller_id, run_id) counter shared across every
strategy_data table, so gap detection becomes a query over holes in the
sequence.

The column ships on both the CREATE TABLE definitions and as idempotent
ALTER ... ADD COLUMN IF NOT EXISTS statements, so existing deployments pick
it up when the forwarder applies the schema at startup.

Document the delivery contract in the schema header, including the ordering
constraint: inserts use JSONEachRow, so a producer emitting a column
ClickHouse does not have fails the whole per-table batch. The forwarder must
be deployed before the producer.
Add per-run seq column to strategy_data archive tables
fix(orders): preserve venue error codes for passive submissions
Capture venue deposits in the execution archive
chore(release): bump cex-broker to 0.2.35
fix(deposits): preserve venue status transitions for locked Binance deposits
…eartbeat

Two issues from review, both real.

Rejected rows are precisely the rows whose table is unknown, so their names came
straight from the request payload and became metric labels verbatim. Every
distinct label value is a permanent series in the metrics SDK, so any client
could grow our memory without bound — a metrics pipeline that traffic can
exhaust is worse than no metrics. Labels are now bounded to the supported set
plus fixed malformed/unsupported buckets, enforced at the metric boundary rather
than at the single call site so a future caller cannot reintroduce it. Raw names
still reach the response and the log, where they are bounded per request and are
what an operator actually needs.

The flush heartbeat also advanced on batches that inserted nothing, because
zero failures is trivially true for an empty batch. An empty POST would have kept
a staleness alert green while no data reached ClickHouse — the exact condition
the heartbeat exists to detect, since its only job is separating quiet from dead.

Tests: a client inventing 50 table names yields one series rather than fifty, and
an empty batch leaves the heartbeat untouched. Full suite 423 pass, 0 fail.
feat(archive): add forwarder OpenTelemetry metrics
A submission row carried only the caller's withdrawal id and the venue
observations that terminate it carried only the venue id, so the two
halves of one movement had no shared key and the lifecycle could not be
joined. Amounts differ by the withdrawal fee, so they are not a fallback.

Binance echoes the caller's withdrawOrderId on withdrawal-history
records; carry it onto observe_withdrawal rows as client_withdrawal_id.
The column and its migration already exist.
…elation-id

fix(archive): correlate withdrawal observations with their submission
…archive

fix(archive): index Binance internal transfer venue ids
@rsoury
rsoury merged commit 60d8418 into master Aug 2, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants