[Databento] Add `symbol_venue_overrides` config for Databento adapter

[shzhng]

Problem

When using both Databento and Interactive Brokers adapters, options (specifically index and equity, unconfirmed for futures options) instruments can end up cached under different venue IDs for the same underlying symbol, causing cache lookup failures.

Background Discussion

From Discord (#interactive_brokers channel) between myself and @faysou:

• Databento uses OPRA as the venue for options (e.g., SPXW 260120P06835000.OPRA)
• IB uses XCBO for the same options (e.g., SPXW260120P06835000.XCBO)
• The venue mismatch means instruments subscribed via Databento can't be found in cache when placing orders via IB

Use Case

Trading SPX/SPXW options where:

• Market data comes from Databento (which now will have OPRA as the "venue" for all options due the consolidated tape changes)
• Order execution goes through IB (using XCBO venue)
• Need instruments to be cached under the same venue ID

Proposed Solution

Add a symbol_venue_overrides configuration option to DatabentoDataClientConfig that allows mapping symbol prefixes to specific venues. This would override the default venue resolution from the publisher map.

Example Configuration

DatabentoDataClientConfig(
    api_key="...",
    symbol_venue_overrides={
        "SPX": "XCBO",
        "SPXW": "XCBO",
        "VIX": "XCBO",
        "VIXW": "XCBO",
    }
)

How it would work

1. Symbol prefixes are matched against raw_symbol from Databento
2. If a prefix matches, use the override venue instead of OPRA
3. Instruments get cached under the override venue (e.g., SPXW 260120P06835000.XCBO)
4. IB orders can now find the instruments in cache

Implementation Notes

The feature would require changes to:

1. Python side: Add symbol_venue_overrides: dict[str, str] | None field to DatabentoDataClientConfig
2. Rust side:
   • Add the field to DatabentoLiveClient in python/live.rs
   • Add the field to DatabentoFeedHandler in live.rs
   • Update update_instrument_id_map() to check prefix matches before falling back to publisher_venue_map

Related

This is similar to how the IB adapter handles symbol_to_mic_venue mappings for routing purposes.

[shzhng]

I can get around to working on this, just wanted to create an issue to keep track and remind myself to manage prioritization

[shzhng]

Note that an additional issue that causes discrepancies between the symbols from DB vs IB is that DB client strictly uses OCC option symbols (eg. SPXW 260120P06835000 - this is the industry standard), while the IB client does some unintended munging by stripping whitespace (eg. SPXW260120P06835000). I believe this whitespace stripping is unnecessary and a bug, so my PR #3452 fixes this

[faysou]

I think this issue regarding venues has been solved before, there's a venue dataset mapping in databento, I solved it, someone had a similar issue. Basically the same symbol can be associated with various datasets in databento and it's possible to tell which dataset we want for a given venue. If you search in the code you should be able to find this. The venue to use is the real MIC of a symbol XCBO. There is also in the IB adapter an override mechanism for the venue when an exchange doesn't return the right MIC (some IB exchanges are associated with several MICs).

[shzhng]

I think this issue regarding venues has been solved before, there's a venue dataset mapping in databento, I solved it, someone had a similar issue. Basically the same symbol can be associated with various datasets in databento and it's possible to tell which dataset we want for a given venue. If you search in the code you should be able to find this. The venue to use is the real MIC of a symbol XCBO. There is also in the IB adapter an override mechanism for the venue when an exchange doesn't return the right MIC (some IB exchanges are associated with several MICs).

Hey @faysou, so the issue is actually a little different. I do use this in my config already:

    databento_data_config = DatabentoDataClientConfig(
        api_key=None,  # Uses DATABENTO_API_KEY env var
        http_gateway=None,
        use_exchange_as_venue=True,  # Use MIC codes (XCBO) as venue - this is default
        # Map XCBO venue to OPRA.PILLAR dataset for subscriptions
        venue_dataset_map={"XCBO": "OPRA.PILLAR"}, # <-- this is what you're talking about
    )

And in my strategy code:

        self.request_instruments(
            venue=Venue("XCBO"),
            client_id=ClientId(DATABENTO),
            params={
                "parent_symbols": ["SPXW.OPT"],
            },
        )

Databento will indeed map to the correct dataset, but when the instrument symbols actually come back, they will come back as SPXW 260120P06835000.OPRA rather than the SPXW 260120P06835000.XCBO as I would expect.

This is because we're using use_exchange_as_venue=True and Databento now only uses OPRA as the publisher (ie. publisher_id=30) for options data because of the consolidated tape changes.

The venue_dataset_map={"XCBO": "OPRA.PILLAR"} on the the DatabentoDataClientConfig basically serves as an "outgoing" mapper, rather than the "incoming" mapper that I want

[shzhng]

Additional context: OPRA definitions lack listing exchange

For those wondering why use_exchange_as_venue doesn't work for OPRA (like it does for GLBX), here's the raw data from Databento:

OPRA Option Definition:

raw_symbol:      SPXW  250131P06100000
publisher_id:    30      # OPRA consolidated
exchange:        OPRA    # Just the feed name, not the actual exchange
underlying:      SPXW
strike_price:    6100.0
expiration:      2025-01-31

The exchange field is just OPRA (the consolidated feed), not the actual listing exchange (CBOE). Compare this to GLBX futures where the definition has exchange: XCME, exchange: XCBT, etc.

Interestingly, OPRA trades do have per-exchange publisher IDs (e.g., publisher_id: 22 = XCBO), but:

1. A single option can trade on all 17 OPRA exchanges
2. There's no "primary listing exchange" field in the definition
3. Creating separate InstrumentIds per-exchange would be confusing

This confirms that a static symbol_venue_overrides config is the right approach for the IB interop use case.