Add FMA rewrites and fused d4 inverse codelets#36
Merged
Conversation
Inverse twiddle recombinations used ADD(MUL,MUL) instead of MADD/MSUB, missing FMA opportunities on every ISA. Fixed in norm_q_inv, norm2_inv_fused, and codelet_d3_tw_res_inv for both f32 (V4F) and double (V2) paths. NEON FMA count in the inverse schedule runner went from 8 to 38. Added codelet_d4_inv_f32 (SSE2/NEON) and codelet_d4_inv (AVX2+FMA) that fuse two d3_inv leaves with the q=8 norm_q_inv butterfly, keeping all intermediate values in registers instead of storing/reloading 32 elements through the data array. Wired into both the recursive path and the schedule runner for both precisions. Assembly impact: NEON total instructions 1162→961, SSE2 1535→1286. AVX2 double d4 fusion: 2212 instructions, 55 FMAs, 330 spill/reloads (expected for 16 YMM registers; spills hit L1 stack). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
codelet_d4_inv_f32(SSE2/NEON) fusing two d3_inv + norm_q_inv(q=8), keeping intermediates in registers instead of store/reload through the data arraycodelet_d4_inv(AVX2+FMA) — same fusion for double-precision using__m256dintrinsicsAssembly impact
Test plan
make testpasses (correctness ok, c api ok, backend=neon-128)--target=x86_64-apple-darwin -msse2 -mno-avx)--target=x86_64-apple-darwin -mavx2 -mfma)🤖 Generated with Claude Code