Skip to content

Phase 2: CPU core-count scaling sweep - #89

Merged
neuromechanist merged 5 commits into
feature/issue-84-epic-xplat-benchmarkfrom
feature/issue-86-phase2-cpu-scaling
Jul 8, 2026
Merged

Phase 2: CPU core-count scaling sweep#89
neuromechanist merged 5 commits into
feature/issue-84-epic-xplat-benchmarkfrom
feature/issue-86-phase2-cpu-scaling

Conversation

@neuromechanist

Copy link
Copy Markdown
Member

Summary

Adds a --threads CPU core-count scaling sweep to the dimension-sweep harness: the CPU
backends (torch-cpu via set_num_threads, numpy via threadpoolctl, native-fortran via
OMP_NUM_THREADS) run at each thread count; GPU backends run once. Answers "do cores catch
the GPU?"

Closes #86
Part of epic #84

What changed

  • --threads N1,N2,...: sweeps thread counts for CPU backends (_is_cpu gates which);
    each run records a threads field. _run_torch/_run_numpy/_run_fortran gained a unified
    threads knob threaded through _run_backend.
  • _report: a CPU-scaling block (threads x backend, per channel count) with a GPU reference
    line; the summary table now shows the fastest CPU thread count. Backward-compatible with
    thread-less JSONs.
  • numpy hygiene: AMICA_NumPy prints per-iteration progress even at verbose=False; its
    stdout/stderr are redirected to a sink during the fit (timing via perf_counter, unaffected).
  • threadpoolctl added as a dependency (in-process BLAS thread control for numpy).

Results (hallu: RTX 4090, 32-core x86; real ds002718; see .context/issue-84/phase2_cpu_scaling.md)

Fastest CPU vs CUDA f64 (ms/iter): native-fortran 3.16 / 10.0 / 12.1 / 24.2 at 16/32/48/70ch
vs CUDA 34 / 35 / 35 / 38.

  • Native Fortran+OpenMP is the only CPU backend that scales with cores, and on 32 cores beats
    the RTX 4090 at every EEG channel count
    (~11x at 16ch down to ~1.6x at 70ch).
  • torch-cpu does not scale and regresses 10-20x at 32 threads (oversubscription); best ~4-8
    threads, never catches the GPU.
  • numpy is thread-flat (BLAS/Python-bound), slowest, never approaches the GPU.

Test plan

  • pyAMICA/tests/test_dimsweep_threads.py: _is_cpu classification (CPU vs GPU backends) +
    full-registry coverage. ruff clean.
  • Validated end-to-end: 52/52 sweep points on hallu, 0 failures; LL agrees ~3 digits across
    backends. Report renders the scaling tables + GPU reference.

Caveats

@neuromechanist

Copy link
Copy Markdown
Member Author

Review summary (4 agents, Sonnet) + complete cross-platform data

Code review came back clean on correctness except one real issue, now fixed; test + silent-failure agents added coverage/robustness findings, all addressed in the follow-up commits:

  • torch.set_num_threads leak (code-reviewer, critical): the process-global thread count leaked into later GPU-reference runs (their preprocessing runs on CPU), silently biasing the CUDA/MPS number. Fixed: save/restore get_num_threads() in a finally, with a regression test.
  • numpy divergence swallowed (silent-failure, critical): the stdout/stderr redirect ate AMICA_NumPy's own "non-finite LL" warning/error (the NaN-suppression AGENTS.md forbids). Replaced with a log-level raise (keeps warnings visible) + a m.converged check that raises on divergence -- important before the 2000-iter Phase 3 runs (AMICA_NumPy long fit hits non-finite likelihood (NaN) ~iter 687 on sample data #39/AMICA_NumPy long fit drifts/degrades past ~150 iters (LL decreases, component match drops) #41 history).
  • _report hid fully-failed backends -> now surfaces them under a "failed" line.
  • --threads validation (>=1) + native-fortran non-sweep row now records its resolved thread count (not a misleading None).
  • Tests: _report fastest-thread/failed-backend, main() --threads end-to-end (real 32ch data), torch thread-restore. 15 tests green.

Complete results: channel x core, f64 + f32, both machines

Full grid run on hallu (Intel 32c + RTX 4090) and Mac (Apple 14c + MLX/MPS); see
.context/issue-84/phase2_cpu_scaling.md. Headline (70ch, ms/it):

  • native-fortran scales with cores (hallu 30 @24c beats cuda 38.5); torch-cpu peaks ~8c then collapses; numpy flat/slowest.
  • MLX ~33 ms is the efficiency winner -- a laptop GPU matching the 4090 and a 32-core CPU.
  • f32 is correct (LL matches f64 to ~4-5 digits on CUDA/MLX/MPS) and scales at least as well as f64.
  • GPU verified genuinely used (98% SM util f64 / 61% f32 sustained), overhead-bound at EEG scale.

Note: comment-analyzer agent stalled; docs self-reviewed against the measured numbers.

@neuromechanist
neuromechanist merged commit 6410e04 into feature/issue-84-epic-xplat-benchmark Jul 8, 2026
7 checks passed
@neuromechanist
neuromechanist deleted the feature/issue-86-phase2-cpu-scaling branch July 8, 2026 23:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant