Data-size (k-factor) sweep: cross-backend IC equivalence vs frames (#90)

[neuromechanist]

Closes #90.

What

Sweeps the data-size axis of the k-factor experiment: holds channels=70 and grows frames
using the full ds002718 sub-002 recording (73.5k -> 747.75k frames, k = 15 -> 152), to test
whether cross-backend IC equivalence rises toward 1.0 as the decomposition becomes
well-determined. Complements Phase 3 (#87), which swept k along the channel axis.

• benchmarks/benchmark_decompose.py: new --frames axis (comma-separated frame counts);
  per-run npz now records frames and filenames include it; _k_sweep() + _plot_ksweep()
  report mean pairwise Hungarian-matched |corr| vs k and emit the line plot.
• Fix: --compare wrote the k-sweep line plot and the per-config equivalence matrix to the
  same --figure path (matrix clobbered the line plot). The matrix now goes to a derived
  {stem}_matrix{suffix} path.
• Findings: .context/issue-90/ksweep_findings.md; figures benchmarks/figures/ksweep_70ch*.png.

Result

Equivalence climbs steeply from 0.91 (k=15) to 0.98 (k=60), then plateaus at ~0.982 (it does
not reach 1.0). Knee at k ~= 60 (frames ~= 3 * channels^2). At k=152 the two double-precision
implementations (native-fortran-f64 vs torch-cuda-f64) agree at 0.995 -- essentially the same
ICs, cross-implementation. The residual below 1.0 is dominated by float32 (rounding over the
larger data + an early lrate-floor stop at iter 1735), not a backend defect.

Practical guideline: budget k ~= 60 for backend-reproducible ICs, ~2x the EEGLAB k >= 20-30 rule
(which locks only the strongest components).

Tested

All 18 runs (3 backends x 6 frame counts, 2000 iters) completed on hallu (Linux x86_64, RTX 4090,
32 cores); LL rises smoothly with k. --compare regenerates both figures correctly after the fix.
ruff check + format clean. Real EEG only (no synthetic data).

[neuromechanist]

Review (2 Sonnet reviewers: code correctness + findings accuracy)

Findings-accuracy review: all quantitative claims verified exact against the raw results/ksweep.log and the regenerated --compare output (equivalence table, per-config matrix, timings, LL, the lrate-floor early-stop attribution). No numeric discrepancies. One interpretation note addressed below.

Code-correctness review — all findings addressed in 9afbc08:

1. (Moderate) Duplicate --frames values that clamp to the same length caused a silent redundant re-fit + npz clobber. Fixed: frame_list is now clamped and deduped up front (sorted({min(int(f), full.shape[1]) for f in ...})).
2. (Moderate) Mixing legacy pre-Data-size (k-factor) sweep: cross-backend IC equivalence vs frames at 70ch #90 npz (frames=-1) with per-frame npz in one --compare silently mis-grouped: a spurious negative-k row in the sweep and a dropped result in the matrix. Fixed: _k_sweep skips the frames<=0 sentinel, the sweep only triggers on >=2 real frame counts, and _compare prints a warning when the two formats are mixed for a channel count.
3. (Low) _k_sweep printed an empty section header when no frame group had >=2 backends. Fixed: rows are collected first; header prints only when non-empty, else an explicit "nothing to compare" line (matching the matrix path's messaging).
4. (Out-of-scope flag, but introduced-by-this-diff) The topomap de-sphere recomputed the sphere from the full-length array regardless of the fit's frame range; --frames truncation made this reachable. Fixed: _compare now passes full[:, :target_frames] to _plot_topomaps.

Findings-accuracy note (addressed): "knee at k~=60" reworded to "transition complete by k~=60" since the sweep samples no point between k=30 and k=60, so the exact knee is an upper bound, not a pinpoint.

Regression check: re-ran --compare on hallu after the refactor; the equivalence table and both figures are byte-identical to the pre-fix run. ruff check + format clean.

[neuromechanist]

Sweep complete (hallu, RTX 4090) — cross-backend equivalence vs k @ 70ch, 2000 iters

frames	k	mean |corr|	min	comps >0.95
73,500	15	0.892	0.838	49.5%
147,000	30	0.932	0.898	61.4%
294,000	60	0.983	0.974	92.9%
490,000	100	0.983	0.980	96.2%
747,750	152	0.982	0.970	92.4%

Equivalence rises sharply with k and saturates at ~0.98 once k >= 60; the two float64 backends agree best (native-fortran vs torch-cuda-f64 = 0.996), the float32 path slightly lower (0.97). All backends agree on LL to ~3 digits.

Caveats: the equivalence knee is between k=30 and k=60 for this recording only — the threshold is data-specific (SNR, effective rank, source structure), not a general law, and the 5-point sweep does not resolve where in [30,60] it sits. The ~0.98 plateau (not 1.0) is intrinsic estimator spread + f32 precision (cf. #27), i.e. "the same decomposition," not a defect.

Findings + figures added in 1d8e88e (.context/issue-90/datasize_sweep_findings.md, benchmarks/figures/equivalence_k90*.png).

[neuromechanist]

Reconciled the duplicate-findings issue from review: this branch already had the canonical ksweep_findings.md + ksweep_70ch*.png (commits d69d8d5/9afbc08, review-addressed), and a later commit added a redundant second run (datasize_sweep_findings.md + equivalence_k90*.png). Removed the duplicate in deb5039 so only the canonical ksweep set survives; the phase-3 cross-link and PR body already point to it. (My earlier comment's numbers were from the now-removed duplicate run; the canonical table is in ksweep_findings.md: k=15->0.911, k=30->0.929, k=60->0.982, plateau ~0.98.)