Add AVX2 encode prototype

Author: eldryoth

CHANGELOG.md

@@ -1,5 +1,16 @@
 # Changelog
 
+## Unreleased
+
+- Added a real non-dispatchable AVX2 fixed-block encode prototype for Standard
+  and URL-safe alphabets. The prototype remains test-only and active runtime
+  backend selection remains scalar-only.
+- Added AVX2 SIMD equivalence coverage for patterned blocks, all 64 emitted
+  six-bit Base64 values, and custom alphabets that must fall back to scalar
+  encoding.
+- Updated the SIMD unsafe inventory for the AVX2 prototype, including staged
+  stack-copy wiping and AVX/SSE cleanup with `vzeroupper`.
+
 ## 1.1.0 - 2026-06-20
 
 - Started the SIMD encode foundation line with a real SSSE3/SSE4.1 fixed-block

docs/SIMD.md

@@ -82,11 +82,11 @@ runtime behavior for that line.
   registers before returning. It is tested against scalar output only when
   SSSE3/SSE4.1 is available and is not compiled into release library builds.
 - An inactive AVX2 fixed-block encode prototype exists behind the SIMD boundary
-  as test-only scaffolding and is tested against scalar output only when AVX2
-  is available. It currently zeroes the destination with SIMD and then
-  overwrites the block with scalar encoding; this is scaffolding, not vectorized
-  Base64 correctness evidence, and it is not compiled into release library
-  builds.
+  as real non-dispatchable vector encode evidence for Standard and URL-safe
+  alphabets. It uses AVX2 lane-local byte shuffling, vector shifts/masks, and
+  byte blending for fixed 24-byte input blocks, then clears XMM/YMM state
+  before returning. It is tested against scalar output only when AVX2 is
+  available and is not compiled into release library builds.
 - An inactive NEON fixed-block encode prototype exists behind the same boundary
   as test-only scaffolding and is tested against scalar output only on
   NEON-capable ARM targets. It currently zeroes the destination with SIMD and

docs/SIMD_ADMISSION.md

@@ -45,7 +45,7 @@ added to `ActiveBackend` or used by runtime dispatch:
 | Backend | State | Required CPU features | Evidence |
 | --- | --- | --- | --- |
 | AVX-512 VBMI | candidate only | `avx512f`, `avx512bw`, `avx512vl`, `avx512vbmi` | inactive prototype equivalence only |
-| AVX2 | candidate only | `avx2` | inactive prototype equivalence only |
+| AVX2 | candidate only | `avx2` | real fixed-block encode prototype for Standard and URL-safe alphabets; non-dispatchable |
 | SSSE3/SSE4.1 | candidate only | `ssse3`, `sse4.1` | real fixed-block encode prototype for Standard and URL-safe alphabets; non-dispatchable |
 | NEON | candidate only | `neon` | inactive prototype equivalence only |
 | wasm `simd128` | candidate only | `simd128` | compile-time candidate detection only |

docs/UNSAFE.md

@@ -415,33 +415,117 @@ Purpose:
 
 - Exercise AVX2 target-feature plumbing.
 - Validate the unsafe boundary.
-- Provide scalar-equivalence scaffolding before any real vector path is
-  admitted. Current tests do not prove vectorized Base64 correctness.
+- Provide real fixed-block vector encode evidence for Standard and URL-safe
+  alphabets before any runtime dispatch is admitted.
 
 Preconditions:
 
 - Caller must prove AVX2 is available on the current CPU.
 - Input is exactly 24 bytes.
 - Output is exactly 32 bytes.
+- The vectorized path is used only for Standard-family alphabets (`A-Z`,
+  `a-z`, `0-9`, and either `+/` or `-_`). Other alphabets fall back to the
+  scalar prototype loop.
 
 Unsafe operation:
 
-- `_mm256_storeu_si256` stores one 256-bit zero vector into the output buffer.
+- `_mm256_loadu_si256` loads from a local 32-byte staging array that contains
+  two 12-byte input lanes plus four zero bytes per lane.
+- `_mm256_shuffle_epi8` reshapes each staged 128-bit lane into four 24-bit
+  groups without reading from caller memory beyond the fixed input array.
+- AVX2 shifts, masks, and OR operations produce thirty-two 6-bit indices.
+- `encode_standard_family_indices_avx2` maps those indices to Standard or
+  URL-safe alphabet bytes with AVX2 byte blends.
+- `_mm256_storeu_si256` stores the 32 encoded bytes into the output buffer.
+- `clear_ymm_registers_for_test_prototype` clears XMM lower halves and uses
+  `vzeroupper` before return to reduce register retention and AVX/SSE
+  transition state in this non-dispatchable test prototype.
+- The local staging array is wiped with the crate cleanup primitive before the
+  function returns.
 
 Safety argument:
 
-- The output type is `&mut [u8; 32]`, so the store has enough initialized,
-  writable memory.
-- The intrinsic is the unaligned store variant, so no stronger alignment is
-  required.
+- The input and output array types provide fixed readable and writable bounds.
+- The SIMD load reads only from a local 32-byte staging array, so the prototype
+  does not over-read the 24-byte caller input.
+- The staging array is mutable and wiped after the SIMD store and register
+  cleanup, reducing stack retention of the copied caller bytes in this inactive
+  prototype.
+- The load and store intrinsics are unaligned variants, so no stronger
+  alignment is required.
 - The function is guarded by an AVX2 target-feature contract.
-- The prototype then overwrites the block with scalar-equivalent Base64 output.
-  The SIMD zeroing is semantically overwritten and is not an implementation of
-  vectorized Base64.
-- Register-retention note: this prototype does not load caller bytes into SIMD
-  registers. Any future AVX2 implementation that does so must document and
-  implement explicit cleanup for every secret-bearing YMM/XMM register before
-  return, plus AVX transition cleanup such as `vzeroupper` where applicable.
+- The output length is fixed by the output array type.
+- The prototype remains test-only and non-dispatchable. Runtime acceleration is
+  still blocked by the SIMD admission manifest.
+- Register-retention note: the prototype now loads caller bytes into YMM/XMM
+  state. It calls `clear_ymm_registers_for_test_prototype` before return. This
+  is retention reduction for the inactive prototype, not a formal
+  microarchitectural side-channel proof.
+
+### `encode_standard_family_indices_avx2`
+
+Location: `src/simd/x86.rs`
+
+Status: private helper for the inactive AVX2 test-only prototype, not compiled
+into release library builds and not dispatchable.
+
+Purpose:
+
+- Map thirty-two 6-bit indices to Standard or URL-safe alphabet bytes with AVX2
+  blends instead of scalar per-byte table indexing.
+
+Preconditions:
+
+- Caller must prove AVX2 is available on the current CPU.
+- `indices` contains only byte values in `0..=63`.
+- The alphabet must be Standard-family as checked by the caller with a complete
+  comparison of positions `0..62` and an explicit check of the two terminal
+  symbols.
+
+Unsafe operation:
+
+- AVX2 byte comparisons and blends compute the ASCII offset for each index.
+
+Safety argument:
+
+- The helper does not dereference raw pointers or access memory.
+- The target-feature contract enables the required AVX2 instructions.
+- The caller constructs `indices` with masks that constrain every byte to a
+  six-bit Base64 value.
+- The helper is private to the test-only prototype path.
+
+### `clear_ymm_registers_for_test_prototype`
+
+Location: `src/simd/x86.rs`
+
+Status: private helper for inactive AVX2 test-only prototypes, not compiled
+into release library builds and not dispatchable.
+
+Purpose:
+
+- Clear lower XMM state and upper YMM state before returning from the AVX2
+  prototype path that processes caller bytes in vector registers.
+
+Preconditions:
+
+- Called only after the prototype has stored its output and no later AVX/SSE
+  value is needed by the function.
+
+Unsafe operation:
+
+- Calls `clear_xmm_registers_for_test_prototype` for lower XMM register state.
+- Inline assembly emits `vzeroupper` to clear upper YMM state and avoid
+  carrying AVX upper halves back to scalar/SSE code.
+
+Safety argument:
+
+- The helper does not read or write memory.
+- The helper runs at the end of the inactive prototype path.
+- `vzeroupper` is valid under the AVX2 target-feature precondition inherited
+  from the caller.
+- This is best-effort register-retention reduction for test evidence, not a
+  guarantee that historical register, stack, cache, or microarchitectural
+  copies do not exist.
 
 ### `encode_12_bytes_ssse3_sse41`
 

src/simd/tests.rs

@@ -38,6 +38,22 @@ fn fill_indices_pattern(output: &mut [u8; 12], seed: u8) {
     }
 }
 
+#[cfg(any(target_arch = "x86", target_arch = "x86_64"))]
+fn fill_indices_pattern_wide(output: &mut [u8; 24], seed: u8) {
+    let mut write = 0;
+    for group in 0..8 {
+        let i0 = seed.wrapping_add(group * 4) & 0x3f;
+        let i1 = seed.wrapping_add(group * 4 + 1) & 0x3f;
+        let i2 = seed.wrapping_add(group * 4 + 2) & 0x3f;
+        let i3 = seed.wrapping_add(group * 4 + 3) & 0x3f;
+
+        output[write] = (i0 << 2) | (i1 >> 4);
+        output[write + 1] = (i1 << 4) | (i2 >> 2);
+        output[write + 2] = (i2 << 6) | i3;
+        write += 3;
+    }
+}
+
 #[cfg(all(feature = "std", any(target_arch = "x86", target_arch = "x86_64")))]
 #[test]
 fn avx512_encode_prototype_matches_scalar_when_available() {
@@ -119,6 +135,50 @@ fn avx2_encode_prototype_matches_scalar_when_available() {
         assert_eq!(scalar_len, avx2_url_safe.len());
         assert_eq!(avx2_url_safe, scalar_url_safe);
     }
+
+    for seed in 0..64 {
+        fill_indices_pattern_wide(&mut input, seed);
+
+        let mut avx2_standard = [0x55; 32];
+        let mut scalar_standard = [0xaa; 32];
+        // SAFETY: The feature check above proves AVX2 availability for this
+        // test invocation.
+        unsafe {
+            encode_24_bytes_avx2::<Standard>(&input, &mut avx2_standard);
+        }
+        let scalar_len = Engine::<Standard, true>::new()
+            .encode_slice(&input, &mut scalar_standard)
+            .unwrap();
+        assert_eq!(scalar_len, avx2_standard.len());
+        assert_eq!(avx2_standard, scalar_standard);
+
+        let mut avx2_url_safe = [0x55; 32];
+        let mut scalar_url_safe = [0xaa; 32];
+        // SAFETY: The feature check above proves AVX2 availability for this
+        // test invocation.
+        unsafe {
+            encode_24_bytes_avx2::<UrlSafe>(&input, &mut avx2_url_safe);
+        }
+        let scalar_len = Engine::<UrlSafe, true>::new()
+            .encode_slice(&input, &mut scalar_url_safe)
+            .unwrap();
+        assert_eq!(scalar_len, avx2_url_safe.len());
+        assert_eq!(avx2_url_safe, scalar_url_safe);
+    }
+
+    fill_indices_pattern_wide(&mut input, 0);
+    let mut avx2_custom = [0x55; 32];
+    let mut scalar_custom = [0xaa; 32];
+    // SAFETY: The feature check above proves AVX2 availability for this test
+    // invocation.
+    unsafe {
+        encode_24_bytes_avx2::<AnchorMatchingCustom>(&input, &mut avx2_custom);
+    }
+    let scalar_len = Engine::<AnchorMatchingCustom, true>::new()
+        .encode_slice(&input, &mut scalar_custom)
+        .unwrap();
+    assert_eq!(scalar_len, avx2_custom.len());
+    assert_eq!(avx2_custom, scalar_custom);
 }
 
 #[cfg(all(feature = "std", any(target_arch = "x86", target_arch = "x86_64")))]

src/simd/x86.rs

@@ -6,15 +6,21 @@ use crate::{Alphabet, encode_base64_value};
 use core::arch::x86::{
     __m128i, __m256i, __m512i, _mm_add_epi8, _mm_and_si128, _mm_blendv_epi8, _mm_cmpeq_epi8,
     _mm_cmpgt_epi8, _mm_loadu_si128, _mm_or_si128, _mm_set1_epi8, _mm_set1_epi32, _mm_setr_epi8,
-    _mm_shuffle_epi8, _mm_slli_epi32, _mm_srli_epi32, _mm_storeu_si128, _mm256_setzero_si256,
-    _mm256_storeu_si256, _mm512_setzero_si512, _mm512_storeu_si512,
+    _mm_shuffle_epi8, _mm_slli_epi32, _mm_srli_epi32, _mm_storeu_si128, _mm256_add_epi8,
+    _mm256_and_si256, _mm256_blendv_epi8, _mm256_cmpeq_epi8, _mm256_cmpgt_epi8, _mm256_loadu_si256,
+    _mm256_or_si256, _mm256_set1_epi8, _mm256_set1_epi32, _mm256_setr_epi8, _mm256_shuffle_epi8,
+    _mm256_slli_epi32, _mm256_srli_epi32, _mm256_storeu_si256, _mm512_setzero_si512,
+    _mm512_storeu_si512,
 };
 #[cfg(target_arch = "x86_64")]
 use core::arch::x86_64::{
     __m128i, __m256i, __m512i, _mm_add_epi8, _mm_and_si128, _mm_blendv_epi8, _mm_cmpeq_epi8,
     _mm_cmpgt_epi8, _mm_loadu_si128, _mm_or_si128, _mm_set1_epi8, _mm_set1_epi32, _mm_setr_epi8,
-    _mm_shuffle_epi8, _mm_slli_epi32, _mm_srli_epi32, _mm_storeu_si128, _mm256_setzero_si256,
-    _mm256_storeu_si256, _mm512_setzero_si512, _mm512_storeu_si512,
+    _mm_shuffle_epi8, _mm_slli_epi32, _mm_srli_epi32, _mm_storeu_si128, _mm256_add_epi8,
+    _mm256_and_si256, _mm256_blendv_epi8, _mm256_cmpeq_epi8, _mm256_cmpgt_epi8, _mm256_loadu_si256,
+    _mm256_or_si256, _mm256_set1_epi8, _mm256_set1_epi32, _mm256_setr_epi8, _mm256_shuffle_epi8,
+    _mm256_slli_epi32, _mm256_srli_epi32, _mm256_storeu_si256, _mm512_setzero_si512,
+    _mm512_storeu_si512,
 };
 
 #[allow(dead_code, reason = "inactive prototype is not dispatchable yet")]
@@ -49,15 +55,44 @@ pub(crate) unsafe fn encode_24_bytes_avx2<A>(input: &[u8; 24], output: &mut [u8;
 where
     A: Alphabet,
 {
-    let zeros = _mm256_setzero_si256();
-    // SAFETY: `output` is a valid 32-byte mutable array. AVX2 is guaranteed by
-    // this function's target feature precondition, and the unaligned store does
-    // not require any stronger pointer alignment.
-    unsafe {
-        _mm256_storeu_si256(output.as_mut_ptr().cast::<__m256i>(), zeros);
+    if !is_standard_or_url_safe_family::<A>() {
+        scalar_encode_block::<A, 24, 32>(input, output);
+        return;
     }
 
-    scalar_encode_block::<A, 24, 32>(input, output);
+    let mut staged = [
+        input[0], input[1], input[2], input[3], input[4], input[5], input[6], input[7], input[8],
+        input[9], input[10], input[11], 0, 0, 0, 0, input[12], input[13], input[14], input[15],
+        input[16], input[17], input[18], input[19], input[20], input[21], input[22], input[23], 0,
+        0, 0, 0,
+    ];
+
+    // SAFETY: `staged` and `output` are valid 32-byte arrays. The function's
+    // target-feature contract enables AVX2. The load and store are unaligned
+    // variants, and the shuffle mask uses `0x80` lanes only for zero-filled
+    // bytes. The SIMD path is non-dispatchable test evidence.
+    unsafe {
+        let input_vec = _mm256_loadu_si256(staged.as_ptr().cast::<__m256i>());
+        let shuffle = _mm256_setr_epi8(
+            2, 1, 0, -128, 5, 4, 3, -128, 8, 7, 6, -128, 11, 10, 9, -128, 2, 1, 0, -128, 5, 4, 3,
+            -128, 8, 7, 6, -128, 11, 10, 9, -128,
+        );
+        let lanes = _mm256_shuffle_epi8(input_vec, shuffle);
+
+        let index0 = _mm256_and_si256(_mm256_srli_epi32(lanes, 18), _mm256_set1_epi32(0x0000_003f));
+        let index1 = _mm256_and_si256(_mm256_srli_epi32(lanes, 4), _mm256_set1_epi32(0x0000_3f00));
+        let index2 = _mm256_and_si256(_mm256_slli_epi32(lanes, 10), _mm256_set1_epi32(0x003f_0000));
+        let index3 = _mm256_and_si256(_mm256_slli_epi32(lanes, 24), _mm256_set1_epi32(0x3f00_0000));
+        let indices = _mm256_or_si256(
+            _mm256_or_si256(index0, index1),
+            _mm256_or_si256(index2, index3),
+        );
+
+        let encoded = encode_standard_family_indices_avx2::<A>(indices);
+        _mm256_storeu_si256(output.as_mut_ptr().cast::<__m256i>(), encoded);
+        clear_ymm_registers_for_test_prototype();
+    }
+    crate::wipe_bytes(&mut staged);
 }
 
 #[allow(dead_code, reason = "inactive prototype is not dispatchable yet")]
@@ -166,6 +201,38 @@ where
     _mm_add_epi8(indices, offset)
 }
 
+#[target_feature(enable = "avx2")]
+unsafe fn encode_standard_family_indices_avx2<A>(indices: __m256i) -> __m256i
+where
+    A: Alphabet,
+{
+    let offset62 = if A::ENCODE[62] == b'-' { -17 } else { -19 };
+    let offset63 = if A::ENCODE[63] == b'_' { 32 } else { -16 };
+
+    let lt26 = _mm256_cmpgt_epi8(_mm256_set1_epi8(26), indices);
+    let lt52 = _mm256_cmpgt_epi8(_mm256_set1_epi8(52), indices);
+    let lt62 = _mm256_cmpgt_epi8(_mm256_set1_epi8(62), indices);
+    let eq62 = _mm256_cmpeq_epi8(_mm256_set1_epi8(62), indices);
+
+    let mut offset = _mm256_set1_epi8(offset63);
+    offset = _mm256_blendv_epi8(offset, _mm256_set1_epi8(offset62), eq62);
+    offset = _mm256_blendv_epi8(offset, _mm256_set1_epi8(-4), lt62);
+    offset = _mm256_blendv_epi8(offset, _mm256_set1_epi8(71), lt52);
+    offset = _mm256_blendv_epi8(offset, _mm256_set1_epi8(65), lt26);
+
+    _mm256_add_epi8(indices, offset)
+}
+
+unsafe fn clear_ymm_registers_for_test_prototype() {
+    // SAFETY: The helper runs after the AVX2 prototype stores its output. The
+    // XMM cleanup zeroes the lower halves declared to the compiler, and
+    // `vzeroupper` clears upper YMM state before returning to scalar code.
+    unsafe {
+        clear_xmm_registers_for_test_prototype();
+        core::arch::asm!("vzeroupper", options(nostack, preserves_flags, nomem));
+    }
+}
+
 #[cfg(target_arch = "x86")]
 unsafe fn clear_xmm_registers_for_test_prototype() {
     // SAFETY: This test-only cleanup runs after the prototype stores its