Skip to content

Make soak test actually detect leaks - #6

Merged
ausimian merged 1 commit into
mainfrom
fix/soak-test-sensitivity
Apr 14, 2026
Merged

Make soak test actually detect leaks#6
ausimian merged 1 commit into
mainfrom
fix/soak-test-sensitivity

Conversation

@ausimian

Copy link
Copy Markdown
Owner

Summary

The soak test as it landed in #5 had two problems that only showed up under the full suite:

  1. In-suite drift swamped the signal. Running standalone it passed cleanly, but running after the other 101 tests MLX's Metal buffer pool kept ~2 tensors' worth of arena alive past `clear_cache`. At 4 MB tensors / 2 MB tolerance the test failed on legitimate allocator behaviour.
  2. Single-tensor detection was never achievable. Arena drift scales with tensor size (~2 tensors, regardless of per-size), so any single-tensor retention is indistinguishable from the noise floor.

Re-tuned around rate detection instead of absolute retention:

  • 1 MB tensors × 2000 iterations (still ~1 s wall time)
  • 4 MB tolerance — absorbs the ~2 MB of Metal arena drift, leaves headroom for one or two stray retentions
  • A per-iteration leak rate scales linearly with `@iters`; arena drift does not. With 2000 iters, a 1 % leak rate accumulates to ~20 MB — 5× the tolerance.

Validated by injection: a deliberate 1 %-per-iter leak yields delta 45 MB vs 4 MB tolerance → fails by 10×. Clean path passes in ~1 s as part of `mix precommit`.

Test plan

  • `mix precommit` green locally (102 tests including soak).
  • Injection test (retain every 100th tensor) fails cleanly with informative diagnostic.
  • CI precommit green on this PR.

The original soak test allocated [8] f32 tensors (64 bytes/iter) and
checked delta <= 1 MB after 5000 iters. Maximum possible leak was
~320 KB — well under the tolerance — so even a total refcount breakage
would pass the test.

New design:
- 1 Mi-element f32 tensors (4 MB each), 500 iters.
- Warm-up loop of 20 iters before sampling the baseline so MLX's
  one-time allocator setup (kernel cache, Metal libraries) doesn't
  look like phantom growth.
- Tolerance dropped to 2 MB — half of one iteration's allocation,
  so a single retained tensor per cycle would trip the test.

Validated by forcing a deliberate leak (retaining every tensor in a
list): delta measured at 407 MB vs 2 MB tolerance, test correctly
fails. The clean path still passes.
@ausimian
ausimian merged commit a9fe80b into main Apr 14, 2026
1 check passed
@ausimian
ausimian deleted the fix/soak-test-sensitivity branch April 14, 2026 01:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant