M22: compile-time debug flags - #36
Merged
Merged
Conversation
Two opt-in Application.compile_env flags re-enable runtime assertions on hot paths that GPU backends skip by default. Both default false with zero runtime cost when off — the gated branches are dead-code eliminated by the Elixir compiler and no Emily.Backend.DebugHelpers reference reaches the compiled BEAM. - :debug_bounds_check — raises on OOB / negative indices in gather, take, take_along_axis, indexed_add, indexed_put. Closes the silent- NaN-from-OOB-gather class of bug that surfaced in DistilBERT-QA conformance (vocab-30522 tokenizer vs. tiny-random embedding). - :debug_detect_nan_inf — scans matmul, fast_rms_norm, fast_layer_norm, and both fast_scaled_dot_product_attention variants for NaN/Inf so training-time numerics failures surface at the producing op. The Emily.Backend.DebugHelpers module holds the assertion bodies and is always compiled; Emily.Backend only references it from inside `if @flag` gates, so when the flags are off the entire module is unreachable from Emily.Backend's bytecode. A zero-cost verification test inspects :beam_disasm.file/1 output and asserts no reference survives under the default-off build. Emily's own config/test.exs keeps both prod flags false so mix test doesn't pay GPU-sync cost on every run; fixture-only flags drive the gate→helper composition tests in test/support/debug_fixture.ex. Standalone softmax has no Backend callback in Emily (it's Axon-composed from exp/sum/divide); the fused SDPA softmax is covered via the two SDPA callback paths. Closes #32.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Application.compile_envflags —:debug_bounds_checkand:debug_detect_nan_inf— that re-enable runtime assertions on hot paths GPU backends skip by default. Both default false with zero runtime cost; the gated branches are dead-code eliminated and noEmily.Backend.DebugHelpersreference reaches the compiled BEAM.:debug_bounds_checkcoversgather,take,take_along_axis,indexed_add,indexed_put— closes the silent-NaN-from-OOB-gather class of bug that surfaced in DistilBERT-QA conformance.:debug_detect_nan_infscansmatmul,fast_rms_norm,fast_layer_norm, and bothfast_scaled_dot_product_attentionvariants so training-time numerics failures surface at the producing op.:beam_disasm.file/1output and asserts noDebugHelpers/check_bounds!/check_nan_inf!reference survives under the default-off build — guards against a future refactor accidentally unconditionalising the gate.config/test.exskeeps both prod flags false somix testdoesn't pay GPU-sync cost; fixture-only flags drive the gate→helper composition tests.Test plan
mix precommit— 421 tests, 0 failures, credo clean.mix test test/emily/debug_flags_test.exs— 15 tests across four groups:DebugHelpersdirect-call tests (positive / negative / error-message format):beam_disasm.file/1config/test.exs's:debug_bounds_checktotrue, rerun the file — zero-cost tests fail (as designed), negative-control test starts raising. Flip back, confirm green.