fix(training,calibration): resolve DataLoader workers everywhere, and guard it (#207)

[jayhesselberth]

Fixes #207.

#206 fixed eval test and added resolve_dataloader_workers, whose docstring says:

Every caller that builds a loader goes through this function, so that guard lives in one place.

Nothing enforced that. Two callers didn't.

What was broken

site	symptom
training.py val loader	hardcoded num_workers=0, three lines below the call that resolves the train loader
calibration.py ×2	num_workers=num_workers with default 0 — on CUDA that is literally zero, not AUTO

Observed on an A30 with a 1,176,763-chunk val set:

fully fed for a while, then dropped to 30% for 5 minutes, now back

≈ 75 minutes of near-idle GPU per 15-epoch run, scaling with val size. Same root cause as #205: __getitem__ being cheap does not mean one process can saturate a GPU — collate, pin, host-to-device and the forward pass all serialize onto that core.

Why not just delete the zeros

The old val comment gave two reasons and they aged differently:

• "__getitem__ is trivially fast, so workers add no benefit" — refuted by the measurement.
• "avoids OOM-triggered segfaults on large multiclass datasets" — real, but narrower than a blanket 0.

LeechDataset stacks per-chunk tensors into contiguous buffers precisely so a fork COW-shares them (the _try_stack note in dataset.py). Only the list fallback — inconsistent per-chunk shapes — makes each worker fault N PyObject headers into private copies and multiply peak RSS by (1 + workers).

So resolve_val_dataloader_workers scopes the exception to exactly that case, and it wins over an explicit --num-workers N, because OOM is not a throughput tradeoff. It lives beside its sibling in dataset.py rather than inline in train(), which restores the "one place" claim and makes it testable without standing up a training loop.

The guard, and why it looks like this

num_workers may no longer be a bare literal anywhere in the package. It must come from a resolver, from a local whose name says it carries a resolved count, or carry a call-site marker dataloader-workers: unresolved with a reason. Two markers exist: commands/benchmark.py (the worker count is the independent variable being benchmarked) and the legacy SignalCNN path (SignalDataset has no _signals_tensor, so the val guard would force 0 and change behaviour).

It took three attempts, and the first two failed their own mutation test:

1. A file-scoped allow-list — worse than nothing. ("training.py", None) exempted the whole file, so reintroducing the The in-training validation loader bypasses resolve_dataloader_workers, starving the GPU once per epoch #207 bug passed.
2. Inspecting DataLoader(...) call sites — also passed the The in-training validation loader bypasses resolve_dataloader_workers, starving the GPU once per epoch #207 reintroduction, because that bug lives in a val_loader_kwargs dict reaching the loader via **kwargs, and the enclosing function resolves a different loader. That is exactly the shape The in-training validation loader bypasses resolve_dataloader_workers, starving the GPU once per epoch #207 had.

So the check is on the value wherever it is bound, not on loader construction.

Tests

TestValLoaderWorkers (6) covers the resolver: stacked datasets get workers, val matches train, the list fallback stays serial, CPU stays serial, an explicit N is honoured when stacked and overridden when not.

The guard is mutation-tested against both real regressions:

mutant	result
reintroduce the #207 literal in training.py	fails, names training.py:1614
reintroduce the calibration.py literal	fails, names calibration.py:202
remove the list-fallback guard	2 resolver tests fail
revert the resolver to always-zero	3 resolver tests fail

Full suite green; ruff format and check clean.