Skip to content

Fix SIGABRT/SIGSEGV from concurrent mx::eval dispatch - #28

Merged
ausimian merged 1 commit into
mainfrom
investigate-sigsegv-streams
Apr 16, 2026
Merged

Fix SIGABRT/SIGSEGV from concurrent mx::eval dispatch#28
ausimian merged 1 commit into
mainfrom
investigate-sigsegv-streams

Conversation

@ausimian

Copy link
Copy Markdown
Owner

Summary

  • Root cause: MLX is not thread-safe (ml-explore/mlx#2133). The Metal CommandEncoder is shared state — concurrent mx::eval calls from different OS threads crash with "A command encoder is already encoding to this command buffer" (SIGABRT) or SIGSEGV. M14's soak tests (8 concurrent Task.async_stream workers) were the first to call mx::eval from multiple dirty-CPU scheduler threads simultaneously.
  • emily::safe_eval(): mutex-serialised mx::eval wrapper in tensor.hpp. All eval callsites route through it; graph-building ops remain lock-free.
  • Removed set_default_stream from with_stream/2: the NIF mutated MLX thread-local state which is unreliable under BEAM process migration between OS threads. Process-dictionary routing was already correct.
  • Hardened resolve_stream(-1): avoids reading the (potentially corrupted) thread-local default.
  • M15.5 milestone: plan for building MLX from source (0.32+) to restore true concurrent Metal dispatch via upstream ThreadLocalStream (ml-explore/mlx#3405).

Test plan

  • Reproduced crash before fix: amplified stress test (16 workers × 100 iters) crashed 2/5 with Metal CommandEncoder assertion
  • Full test suite 20× — 0 crashes
  • Soak tests 20× — 0 crashes
  • Amplified stress test 20× — 0 crashes

🤖 Generated with Claude Code

MLX is not thread-safe (ml-explore/mlx#2133). The Metal
CommandEncoder is shared state — concurrent mx::eval calls from
different OS threads crash with "A command encoder is already
encoding to this command buffer" (SIGABRT) or SIGSEGV. The M14
soak tests were the first to exercise concurrent eval from
multiple dirty-CPU scheduler threads.

Fix:
- Add emily::safe_eval() with a static mutex serialising all
  mx::eval calls (c_src/emily/tensor.hpp)
- Remove set_default_stream calls from with_stream/2 — the NIF
  mutated MLX thread-local state which is unreliable under BEAM
  process migration between OS threads
- Harden resolve_stream(-1) to avoid reading thread-local default

Also adds M15.5 milestone for building MLX from source once
upstream gains native thread-safety (0.32+).
@ausimian
ausimian merged commit ce4a91e into main Apr 16, 2026
1 check passed
@ausimian
ausimian deleted the investigate-sigsegv-streams branch April 16, 2026 14:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant