Backtesting Multi-processing and data streaming

[Spad0ut]

Hi everyone,

I am trying to run an extensive backtest with a relatively complex strategy. I am using 1-minute bars, 5 venues, 50 instruments and 1-2 years of data. Considering it will take ages to process all 1-minute bars on one CPU core, I was wondering if there are any built-in ways to split the backtest among several cores or I have to run a separate smaller backtest on every core? Also the RAM usage is very high. I saw that you have chunk_size parameter in the high API BacktestNode, but I was wondering if something similar is available in BacktestEngine? I am also using Nautilus Trader 1.221, so please let me know if I should bother updating, in order to solve my problem.

@cjdsellers I woud be very happy, if you could help me with my issue :)

[Aliipou]

NautilusTrader does not have built-in multi-core parallelism within a single BacktestEngine run — the simulation loop is single-threaded by design for determinism. Here is the practical approach for your setup:

1. Parallel backtests via BacktestNode (high-level API)

BacktestNode is built exactly for this. It runs multiple independent configs in parallel using a process pool:

from nautilus_trader.backtest.node import BacktestNode

node = BacktestNode(configs=[config1, config2, config3])
results = node.run()

2. chunk_size is a BacktestNode-only feature

chunk_size lives on BacktestDataConfig and only applies when using BacktestNode. BacktestEngine loads all data upfront — so to reduce RAM you need to move to BacktestNode + chunk_size.

3. Practical split strategy for 50 instruments / 5 venues

Split instruments across workers:

n_workers = 8
groups = [instruments[i::n_workers] for i in range(n_workers)]
configs = [build_config(group) for group in groups]
node = BacktestNode(configs=configs)

Or split by time window if your strategy has no state dependency across windows (walk-forward style):

windows = [("2023-01-01","2023-07-01"), ("2023-07-01","2024-01-01")]
configs = [build_config(instruments, start=s, end=e) for s, e in windows]

4. If you must stay on BacktestEngine directly

Filter engine.add_data() calls manually. There is no built-in chunking at the engine level.

Full BacktestNode docs: https://nautilustrader.io/docs/nightly/concepts/backtesting

[faysou]

there's no process pool for the BacktestNode, but actually if one wanted this would be a good place to do it where you've shown it