SystemError: no locals found

[quant-exchange]

The 00:00 UTC 2026-05-10 BTC pipeline batch crashed with:

SystemError: no locals found when setting up annotations
at neuralprophet/components/seasonality/fourier.py:88
inside Lightning training_step / forward

This is a CPython frame-state corruption that fires sporadically inside torch / Lightning forward paths under memory pressure or right after a CUDA context teardown. The line itself is innocent (for name, features in s.items():); the interpreter's locals dict transiently disappears. Resetting CUDA caches + GC between attempts clears the condition.

Add _fit_with_retry() that wraps model.fit() with up to 2 attempts, catching SystemError specifically (other exceptions propagate unchanged), and use it at both the forecast and validation fit sites.

Without this, a single transient interpreter glitch silently kills the batch for a training asset.

def _fit_with_retry(model, training_data, *, label, max_attempts=2, **fit_kwargs):
    """Wrap model.fit with a retry on transient interpreter / CUDA glitches.

    Specifically guards against `SystemError: no locals found when setting
    up annotations`, a CPython frame-state corruption that fires
    sporadically inside torch.compile / Lightning forward paths under
    memory pressure. Resetting CUDA caches + GC between attempts clears
    the condition. Other exceptions propagate unchanged.
    """
    last_err = None
    for attempt in range(1, max_attempts + 1):
        try:
            return model.fit(training_data, **fit_kwargs)
        except SystemError as err:
            last_err = err
            print(f"[fit_retry] {label}: SystemError on attempt {attempt}/{max_attempts}: {err}")
            try:
                gc.collect()
                if torch.cuda.is_available():
                    torch.cuda.empty_cache()
                    torch.cuda.synchronize()
            except Exception as cleanup_err:
                print(f"[fit_retry] cleanup error: {cleanup_err}")
            if attempt >= max_attempts:
                raise
            time.sleep(2)
    if last_err is not None:
        raise last_err

How to use:

_fit_with_retry(model, training_data, freq=engine['freq'], epochs=_epochs,
                            minimal=engine['minimal'], label='validation', **_ws_kwargs)

[quant-exchange]

We're still working on getting together the dozens of features we've added to NeuralProphet for the community. You can see how we're exploring over at PriceRanger.ai!

NOTE: minimal=engine['minimal'] (this is custom to NeuralProfit to reduce log burdens on lightning for performance reasons, and you can ignore that*; also, our engine[] structure is just for our hyperparameters container you can do whatever you want here).