Enable AVX512 scheduled inverse FFT

Author: falseywinchnet

examples/benchmark.cpp

@@ -410,6 +410,7 @@ struct Result {
   double standard_ns = NAN;
   double native_f32_ns = NAN;
   double standard_f32_ns = NAN;
+  double inverse_ns = NAN;
   double roundtrip_ns = NAN;
   double standard_over_fftw = NAN;
   double standard_f32_over_fftwf = NAN;
@@ -612,6 +613,13 @@ Result bench_one(std::size_t n, int forced_iters, FFTW* fftw, FFTWf* fftwf, Inte
     result.sink += standard_f32[(static_cast<std::size_t>(result.sink) * 17) % nb].re;
   });
 
+  plan.forward(original.data(), standard.data(), work.data(), scratch.data());
+  result.inverse_ns = bench_ns(iters, [&] {
+    standard[(static_cast<std::size_t>(result.sink) * 17) % nb].re += 1e-12;
+    plan.inverse(standard.data(), inverse_out.data());
+    result.sink += inverse_out[(static_cast<std::size_t>(result.sink) * 17) % n];
+  });
+
   input = original;
   result.roundtrip_ns = bench_ns(iters, [&] {
     input[static_cast<std::size_t>(result.sink) & (n - 1)] += 1e-12;
@@ -696,6 +704,7 @@ Result bench_one(std::size_t n, int forced_iters, FFTW* fftw, FFTWf* fftwf, Inte
   print_ns(result.standard_ns);
   print_ns(result.native_f32_ns);
   print_ns(result.standard_f32_ns);
+  print_ns(result.inverse_ns);
   print_ns(result.roundtrip_ns);
 
   if (std::isnan(result.standard_over_fftw)) {
@@ -827,7 +836,7 @@ int main(int argc, char** argv) {
     std::printf("PFFFT disabled; compile with -DBFFT_WITH_PFFFT and link pffft to enable.\n");
 #endif
 
-    std::printf("%8s %8s %11s %11s %11s %11s %11s %11s %11s %11s %11s %11s %8s %8s %8s %8s %8s %8s  %s\n",
+    std::printf("%8s %8s %11s %11s %11s %11s %11s %11s %11s %11s %11s %11s %11s %8s %8s %8s %8s %8s %8s  %s\n",
                 "N",
                 "iters",
                 "FFTW_ns",
@@ -839,6 +848,7 @@ int main(int argc, char** argv) {
                 "Std_ns",
                 "F32Nat_ns",
                 "F32Std_ns",
+                "Inv_ns",
                 "RT_ns",
                 "S/F64",
                 "F32/Ff",

src/detail/bruun_kernel.hpp

@@ -4318,7 +4318,10 @@ class RFFT {
     // Fast scheduled residue inverse: exact reverse of forward_residues_recursive.
     // Requires N >= 64.
     void inverse_residues_recursive(double* RESTRICT v) const {
-#if BRUUN_LEVEL == 1
+#if BRUUN_LEVEL <= 3
+        // Wide x86 uses the same flat inverse schedule as 128-bit SIMD.  The
+        // fused d4 inverse codelet keeps the q==8 leaves register-resident, so
+        // the schedule no longer pays an extra store/reload penalty at the leaf.
         run_inv_residue_schedule(v);
 #else
         residue_spine_tail_inv(v);

src/detail/inverse_acceleration_notes.txt

@@ -20,8 +20,7 @@ Why this is first:
 
 ISA notes:
 - SSE/NEON benefit immediately from removing recursive calls and from the existing 128-bit `norm_q_inv` loop.
-- AVX2 keeps the existing recursive path for now because the first scheduled attempt did not beat it on this host.
-- AVX-512 is treated the same as AVX2 until the inverse d4 q==8 fusion is available; the schedule infrastructure is present, but not enabled for wide-vector builds yet.
+- AVX2 and AVX-512 now use the scheduled inverse as well.  After the double-precision d4 inverse fusion, q==8 leaves stay register-resident in both recursive and scheduled runners, and direct inverse benchmarking on wide x86 hosts showed the schedule paying for its flat dispatch.
 
 Completed:
 1. [done] Fused inverse d4 codelet for f32 q==8 (`codelet_d4_inv_f32`): two inverse d3 leaves + inverse q=8 norm in one function, values stay in registers.  Eliminates 8 loads + 8 stores per q==8 block.  Wired into both the schedule runner and the recursive path.
@@ -32,9 +31,9 @@ Completed:
 Follow-up candidates:
 2. Fuse native/standard gather with the inverse binomial tail to remove one complete pass over N for inverse-from-complex APIs.
 3. Add f32/scalar lane-doubling equivalents where the inverse leaf still falls back to generic inverse-norm ladders.
-6. Re-evaluate enabling the scheduled inverse for AVX2 now that the d4 fusion is available — the register-residency win may now overcome flat-dispatch overhead.
 
-SSE-first gate:
-- Initial AVX2 timing showed the flat inverse schedule is not yet a win on this host, likely because AVX2 already has the recursive leaf path inlined into large vector kernels while the schedule adds switch pressure.
-- The scheduled inverse is therefore enabled first for 128-bit SSE/NEON builds, where removing recursive calls is the current concrete win.
-- AVX2 and AVX-512 keep the existing recursive path until the d4 fusion is ported to those ISAs and benchmarked.
+Wide-x86 schedule gate:
+- Rechecked the scheduled inverse on Intel Xeon Platinum 8370C with `-O3 -mavx2 -mfma -mno-avx512*`.
+- Rechecked the AVX-512 scheduled inverse on the same host with `-O3 -march=native`.
+- The benchmark now prints a direct `Inv_ns` column so inverse-only changes are visible instead of inferred from round-trip timing.
+- The d4 fusion was correctness-checked on AVX2 and AVX-512 and is fast enough to enable the flat inverse schedule for BRUUN_LEVEL 2 and 3.