[Backtesting] - Dynamically Changing the Instrument Universe

[Tverous]

Hi everyone,

I’m building strategies that rebalance weekly, where the set of tradable instruments changes over time. From what I understand, in NautilusTrader, you need to load the instruments before the backtest starts, which means predefining the entire universe (e.g., the top 500 stocks by market cap).

That approach feels inefficient, and it may also introduce point-in-time issues (today’s “top 500” isn’t necessarily the correct universe historically). Is there a way in NautilusTrader to dynamically load instruments and their historical bar data during a backtest - i.e., load data on demand as the backtest progresses - rather than requiring everything to be preloaded upfront?

Thanks

[Tverous]

Hi @cjdsellers,

If this functionality isn’t supported today, I’m happy to help implement it. My rough proposal is:

• Add dynamic instrument loading to the BacktestEngine
• Add deferred subscription support to BacktestDataClient
• Have the BacktestEngine process deferred subscriptions during the run
• Expose a data-loading callback/hook on Strategy, so a strategy can request instruments/bars on demand (e.g., at each rebalance)

Conceptually, the example usage could look like this:

class DynamicUniverseStrategy(Strategy):
    def __init__(self, catalog_path, constituent_history):
        self.catalog = ParquetDataCatalog(catalog_path)
        self.constituent_history = constituent_history  # date -> [InstrumentId]
        self.loaded = set()

    def on_start(self):
        self.clock.set_timer("rebalance", timedelta(days=7), self.on_rebalance)

    def on_rebalance(self, event):
        current_date = self.clock.utc_now().date()
        universe = self.constituent_history[current_date]

        for instrument_id in universe:
            if instrument_id not in self.loaded:
                # Load instrument and data dynamically
                instrument = self.catalog.instruments([instrument_id])[0]
                bars = list(self.catalog.bars([instrument_id]))

                # Inject into running backtest
                self._engine.add_instrument_dynamic(instrument, bars)
                self.loaded.add(instrument_id)

        self.execute_rebalance(universe)

Please let me know if this approach aligns with NautilusTrader’s architecture and whether you’d be open to a PR in this direction.

Thanks