Fix OHLCV subscriptions: stream via watchOHLCV and guard bar re-emits - #73
Conversation
Shedding is silent data loss; the shed metric is invisible when OTel is disabled, so emit a rate-limited WARN with the running shed total.
|
Warning Review limit reachedYou’ve reached a temporary PR review limit under our Fair Usage Limits Policy. Next review available in: 1 minute Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (6)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
Problem
Subscribe(OHLCV)streams were unusable for archival in production. The stream loop calledfetchOHLCVWs, which in our ccxt fork is a Binance ws-api request/response klines fetch — it returns the latest ~500 bars in ~300ms rather than blocking on exchange updates. The loop therefore spun ~3x/s, andOhlcvBarTracker.processBatchre-emitted every bar of every frame (~1,500 archive rows/s per pair) into an archive writer that drains far slower, shedding ~99% of rows. Observed in production:market_data.candlesreceived a sparse, hours-delayed crawl of real-but-old bars (stored values verified identical to Binance's klines at those old timestamps), while also hammering Binance with continuous klines requests. Every other subscription type already blocks on push-basedwatch*calls.Changes
Subscribe(OHLCV)now useswatchOHLCV, matching the sibling orderbook/trades/ticker cases: each loop iteration blocks until the exchange pushes a kline update. Bootstrap backfill and frame writing are unchanged.OhlcvBarTracker.processBatchnow filters incoming frames to bars at-or-after the last tracked open time, so overlapping or snapshot-shaped payloads can never re-emit already-archived bars: a repeated identical frame yields at most one open-bar update; a frame advancing by one bar yields exactly the close of the previous bar plus the new open bar. First-frame behavior (bootstrap archiving its full window) and the gap/reconnect close-out are preserved.Verification
bun test: 457 pass / 0 fail, including new tracker tests for repeated-snapshot, advancing-overlap, and first-frame batch semantics, and the OHLCV subscribe-path tests updated towatchOHLCV.bunx tscandbunx biome lint: clean (117 pre-existing warnings unchanged).Deployment note
Fixes the standing OHLCV collector (#70) in production; should ship as the next release train so the collector image picks it up.