Skip to content

[Databento] Add fixed-interval batching for subscriptions and HTTP requests #3457

@shzhng

Description

@shzhng

Summary

Implement a fixed-interval batching mechanism in the Databento data client to handle rate limiting (3 requests/second). The batcher automatically groups individual subscription/request calls and flushes them at fixed intervals.

Key insight: By batching at the data client level, we remove the need for actors to explicitly batch via params. The rate limiting concern is fully encapsulated.

Important distinction: This is NOT debouncing (where the timer resets on each new item). This is fixed-interval batching - a timer starts on the first item, and ALL items accumulated during that interval are flushed together when the timer fires.

Proposed Config Options

subscription_batch_interval_ms: int | None = 300  # Live subscriptions
http_request_batch_interval_ms: int | None = 300  # HTTP requests

Architecture

Actor/Strategy
    │
    │  Individual calls (no batching needed)
    ▼
┌─────────────────────────────────────────────────────────────┐
│                    DatabentoDataClient                       │
│                                                              │
│  _subscribe_quote_ticks(instrument_id)                      │
│  _subscribe_trade_ticks(instrument_id)                      │
│  _request_quote_ticks(instrument_id)                        │
│           │                                                  │
│           ▼                                                  │
│  ┌─────────────────────────┐  ┌─────────────────────────┐   │
│  │ SubscriptionBatcher     │  │  HttpRequestBatcher     │   │
│  │                         │  │                         │   │
│  │ Collects by:            │  │ Collects by:            │   │
│  │   (dataset, schema)     │  │   (dataset, endpoint,   │   │
│  │                         │  │    schema, start, end)  │   │
│  │ After 300ms, batches    │  │                         │   │
│  │ into single call        │  │ After 300ms, batches    │   │
│  └───────────┬─────────────┘  └───────────┬─────────────┘   │
│              │                             │                 │
│              ▼                             ▼                 │
│     live_client.subscribe()      http_client.get_range_*()  │
│     (all instrument_ids)         (all instrument_ids)       │
│                                                              │
└─────────────────────────────────────────────────────────────┘

Methods to Update

Live Subscriptions:

  • _subscribe_quote_ticks
  • _subscribe_trade_ticks
  • _subscribe_bars
  • _subscribe_instrument_status

HTTP Requests:

  • _request_quote_ticks
  • _request_trade_ticks
  • _request_bars

Benefits

  1. Simpler actor code - no need to batch via params
  2. Rate limiting encapsulated - data client handles it transparently
  3. Cleaner data client code - single instrument per method call
  4. Configurable - can disable batching if not needed

Note: I'm actively working on this feature. Implementation is in progress.

Metadata

Metadata

Assignees

No one assigned

    Labels

    adapterNautilus integration with external systemsenhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions