-
Notifications
You must be signed in to change notification settings - Fork 2.2k
Open
Labels
adapterNautilus integration with external systemsNautilus integration with external systemsenhancementNew feature or requestNew feature or request
Description
Summary
Implement a fixed-interval batching mechanism in the Databento data client to handle rate limiting (3 requests/second). The batcher automatically groups individual subscription/request calls and flushes them at fixed intervals.
Key insight: By batching at the data client level, we remove the need for actors to explicitly batch via params. The rate limiting concern is fully encapsulated.
Important distinction: This is NOT debouncing (where the timer resets on each new item). This is fixed-interval batching - a timer starts on the first item, and ALL items accumulated during that interval are flushed together when the timer fires.
Proposed Config Options
subscription_batch_interval_ms: int | None = 300 # Live subscriptions
http_request_batch_interval_ms: int | None = 300 # HTTP requestsArchitecture
Actor/Strategy
│
│ Individual calls (no batching needed)
▼
┌─────────────────────────────────────────────────────────────┐
│ DatabentoDataClient │
│ │
│ _subscribe_quote_ticks(instrument_id) │
│ _subscribe_trade_ticks(instrument_id) │
│ _request_quote_ticks(instrument_id) │
│ │ │
│ ▼ │
│ ┌─────────────────────────┐ ┌─────────────────────────┐ │
│ │ SubscriptionBatcher │ │ HttpRequestBatcher │ │
│ │ │ │ │ │
│ │ Collects by: │ │ Collects by: │ │
│ │ (dataset, schema) │ │ (dataset, endpoint, │ │
│ │ │ │ schema, start, end) │ │
│ │ After 300ms, batches │ │ │ │
│ │ into single call │ │ After 300ms, batches │ │
│ └───────────┬─────────────┘ └───────────┬─────────────┘ │
│ │ │ │
│ ▼ ▼ │
│ live_client.subscribe() http_client.get_range_*() │
│ (all instrument_ids) (all instrument_ids) │
│ │
└─────────────────────────────────────────────────────────────┘
Methods to Update
Live Subscriptions:
_subscribe_quote_ticks_subscribe_trade_ticks_subscribe_bars_subscribe_instrument_status
HTTP Requests:
_request_quote_ticks_request_trade_ticks_request_bars
Benefits
- Simpler actor code - no need to batch via params
- Rate limiting encapsulated - data client handles it transparently
- Cleaner data client code - single instrument per method call
- Configurable - can disable batching if not needed
Note: I'm actively working on this feature. Implementation is in progress.
Metadata
Metadata
Assignees
Labels
adapterNautilus integration with external systemsNautilus integration with external systemsenhancementNew feature or requestNew feature or request