Speed up AVX-512 byte and fixed-width array scans

[mrecachinas]

What

Improves the AVX-512 BITALG paths identified in #51 and adds a repeatable three-run x86 benchmark harness.

• Uses the existing masked AVX-512 tail implementation for byte inputs below 64 bytes instead of falling back through AVX2/SSE.
• Adds AVX-512 cross-record scanners for 16- and 32-byte catalogs, processing four records per batch.
• Preserves first-index ordering, lowest-index best ties, ordered all-results, threshold semantics, exact-match termination, and scalar tails.
• Keeps explicit classic/SSE/AVX2 selections unchanged; the new scanners engage only for native or avx512 when AVX-512 BITALG is available.
• Expands the x86 threshold and scanner benchmark matrices.

Advances the AVX-512 portion of #51. AVX2-only tuning remains a separately measured follow-up.

Performance

Three-run medians on a Google Cloud c4-standard-4, Intel Xeon Platinum 8581C (Emerald Rapids), with avx2, avx512bw, and avx512_bitalg exposed.

Python fixed-width scans

Workload	Before	After	Speedup
1024x16 first, random/no-match	3.222 us	0.679 us	4.75x
1024x16 best, random/no-match	3.720 us	0.651 us	5.72x
1024x16 all, random/no-match	3.466 us	0.710 us	4.88x
1024x32 first, random/no-match	3.199 us	1.278 us	2.50x
1024x32 best, random/no-match	3.729 us	1.383 us	2.70x
1024x32 all, random/no-match	3.445 us	1.377 us	2.50x

Short AVX-512 byte inputs

Bytes	Before	After	Improvement
16	5.380 ns	4.048 ns	32.9%
32	6.112 ns	4.068 ns	50.2%
48	6.924 ns	4.071 ns	70.1%
63	11.955 ns	4.068 ns	193.9%

The hybrid scanner scalar-checks the first record before batching. The accepted worst-case trade-off is approximately 0.5-0.6 ns for 16-byte first/best when an exact match is at index 0; full scans improve by 2.77x-4.07x in Rust and 2.50x-5.72x through Python.

Testing

• cargo test --lib --no-default-features on the AVX-512 C4 VM — 72 tests
• cargo test --no-default-features on ARM64 — 76 unit and 2 doc tests
• python -m pytest test/ -k "not bench" — 80 passed, 3 skipped
• x86 benchmark comparisons use three-run medians on the same VM
• Independent correctness and safety review found no scanner defects
• cargo fmt --check
• git diff --check

[github-actions]

📊 Benchmark Comparison Results

Status	Benchmark	Base	PR	Delta
❌	check_bytes_arrays_best_within_dist[16384 elems,s=64,at 0]	0.0539ms	0.1232ms	128.4% slower
❌	check_bytes_arrays_best_within_dist[16384 elems,s=64,mid]	0.0538ms	0.1229ms	128.3% slower
❌	check_bytes_arrays_best_within_dist[16384 elems,s=64,end]	0.0544ms	0.1230ms	126.2% slower
❌	check_bytes_arrays_all_within_dist[16384 elems,s=64,mid]	0.0539ms	0.1132ms	109.9% slower
❌	check_bytes_arrays_all_within_dist[16384 elems,s=64,end]	0.0541ms	0.1134ms	109.5% slower
❌	check_bytes_arrays_first_within_dist[16384 elems,s=64,mid]	0.0248ms	0.0516ms	108.5% slower
❌	check_bytes_arrays_all_within_dist[16384 elems,s=64,at 0]	0.0543ms	0.1132ms	108.5% slower
❌	check_bytes_arrays_first_within_dist[16384 elems,s=64,end]	0.0540ms	0.1029ms	90.5% slower
❌	check_bytes_arrays_best_within_dist[ 512 elems,s=16,end]	0.0026ms	0.0039ms	51.1% slower
❌	fixed_width_array_scan_matrix[best-exact_mid-16]	0.0026ms	0.0039ms	50.9% slower
❌	fixed_width_array_scan_matrix[best-random_no_match-16]	0.0050ms	0.0075ms	49.7% slower
❌	fixed_width_array_scan_matrix[best-threshold_d_minus_1-16]	0.0050ms	0.0075ms	49.7% slower
❌	fixed_width_array_scan_matrix[best-exact_late-16]	0.0050ms	0.0075ms	49.6% slower
❌	fixed_width_array_scan_matrix[best-threshold_d-16]	0.0050ms	0.0075ms	49.5% slower
❌	fixed_width_array_scan_matrix[best-threshold_d_plus_1-16]	0.0051ms	0.0075ms	46.9% slower
❌	fixed_width_array_scan_matrix[best-exact_mid-32]	0.0030ms	0.0043ms	45.7% slower
❌	fixed_width_array_scan_matrix[best-threshold_d_plus_1-32]	0.0059ms	0.0085ms	44.0% slower
❌	check_bytes_arrays_best_within_dist[ 1024 elems,s=32,mid]	0.0059ms	0.0085ms	44.0% slower
❌	fixed_width_array_scan_matrix[best-threshold_d-32]	0.0059ms	0.0085ms	44.0% slower
❌	fixed_width_array_scan_matrix[best-random_no_match-32]	0.0059ms	0.0084ms	43.9% slower
❌	check_bytes_arrays_best_within_dist[ 1024 elems,s=32,end]	0.0059ms	0.0085ms	43.8% slower
❌	fixed_width_array_scan_matrix[best-exact_late-32]	0.0059ms	0.0085ms	43.8% slower
❌	check_bytes_arrays_best_within_dist[ 1024 elems,s=32,at 0]	0.0059ms	0.0084ms	43.8% slower
❌	fixed_width_array_scan_matrix[best-threshold_d_minus_1-32]	0.0059ms	0.0084ms	43.7% slower
❌	check_bytes_arrays_all_within_dist[ 512 elems,s=16,mid]	0.0025ms	0.0036ms	43.0% slower
❌	check_bytes_arrays_all_within_dist[ 512 elems,s=16,end]	0.0025ms	0.0036ms	42.8% slower
❌	check_bytes_arrays_all_within_dist[ 512 elems,s=16,at 0]	0.0025ms	0.0036ms	41.4% slower
❌	fixed_width_array_scan_matrix[all-exact_mid-16]	0.0049ms	0.0070ms	41.1% slower
❌	fixed_width_array_scan_matrix[all-random_no_match-16]	0.0049ms	0.0069ms	41.1% slower
❌	fixed_width_array_scan_matrix[all-exact_late-16]	0.0049ms	0.0069ms	40.9% slower
❌	fixed_width_array_scan_matrix[all-threshold_d-16]	0.0049ms	0.0069ms	40.9% slower
❌	fixed_width_array_scan_matrix[all-threshold_d_minus_1-16]	0.0049ms	0.0068ms	40.9% slower
❌	fixed_width_array_scan_matrix[all-exact_early-16]	0.0049ms	0.0069ms	40.9% slower
❌	fixed_width_array_scan_matrix[first-exact_late-16]	0.0044ms	0.0062ms	40.0% slower
❌	fixed_width_array_scan_matrix[first-threshold_d_minus_1-16]	0.0044ms	0.0062ms	39.9% slower
❌	fixed_width_array_scan_matrix[first-random_no_match-16]	0.0044ms	0.0062ms	39.8% slower
❌	check_bytes_arrays_first_within_dist[ 1024 elems,s=32,mid]	0.0027ms	0.0037ms	37.4% slower
❌	fixed_width_array_scan_matrix[first-threshold_d-32]	0.0027ms	0.0037ms	37.4% slower
❌	fixed_width_array_scan_matrix[all-threshold_d-32]	0.0058ms	0.0079ms	36.8% slower
❌	check_bytes_arrays_all_within_dist[ 1024 elems,s=32,mid]	0.0058ms	0.0079ms	36.8% slower
❌	fixed_width_array_scan_matrix[all-threshold_d_plus_1-32]	0.0058ms	0.0079ms	36.8% slower
❌	check_bytes_arrays_all_within_dist[ 1024 elems,s=32,at 0]	0.0058ms	0.0079ms	36.7% slower
❌	fixed_width_array_scan_matrix[all-exact_early-32]	0.0058ms	0.0079ms	36.7% slower
❌	fixed_width_array_scan_matrix[all-random_no_match-32]	0.0057ms	0.0078ms	36.6% slower
❌	fixed_width_array_scan_matrix[all-exact_mid-32]	0.0058ms	0.0079ms	36.5% slower
❌	fixed_width_array_scan_matrix[all-threshold_d_minus_1-32]	0.0057ms	0.0078ms	36.5% slower
❌	check_bytes_arrays_all_within_dist[ 1024 elems,s=32,end]	0.0058ms	0.0079ms	36.4% slower
❌	fixed_width_array_scan_matrix[all-exact_late-32]	0.0058ms	0.0079ms	36.4% slower
❌	fixed_width_array_scan_matrix[first-threshold_d_minus_1-32]	0.0053ms	0.0072ms	35.8% slower
❌	fixed_width_array_scan_matrix[first-random_no_match-32]	0.0053ms	0.0072ms	35.5% slower
❌	fixed_width_array_scan_matrix[first-exact_late-32]	0.0053ms	0.0072ms	35.5% slower
❌	check_bytes_arrays_first_within_dist[ 1024 elems,s=32,end]	0.0053ms	0.0072ms	35.5% slower
❌	fixed_width_array_scan_matrix[all-threshold_d_plus_1-16]	0.0051ms	0.0069ms	34.9% slower

➖ 51 benchmarks within noise (click to expand)

Benchmark	Base	PR	Delta
check_bytes_arrays_best_within_dist[ 512 elems,s=16,at 0]	0.0001ms	0.0001ms	40.8% slower
check_bytes_arrays_best_within_dist[ 512 elems,s=16,mid]	0.0013ms	0.0020ms	49.3% slower
check_bytes_arrays_first_within_dist[ 512 elems,s=16,at 0]	0.0001ms	0.0001ms	37.0% slower
check_bytes_arrays_first_within_dist[ 512 elems,s=16,end]	0.0023ms	0.0032ms	42.5% slower
check_bytes_arrays_first_within_dist[ 512 elems,s=16,mid]	0.0012ms	0.0016ms	41.6% slower
check_bytes_arrays_first_within_dist[ 1024 elems,s=32,at 0]	0.0001ms	0.0001ms	31.7% slower
check_bytes_arrays_first_within_dist[16384 elems,s=64,at 0]	0.0001ms	0.0002ms	48.0% slower
check_bytes_within_dist[127 bytes,d=500]	0.0001ms	0.0001ms	66.7% slower
check_bytes_within_dist[16 bytes,d=0]	0.0001ms	0.0001ms	31.2% slower
check_bytes_within_dist[64 bytes,d=100]	0.0001ms	0.0001ms	47.9% slower
hex_within_dist	0.0001ms	0.0001ms	24.1% slower
fixed_width_array_scan_matrix[best-exact_early-16]	0.0001ms	0.0001ms	41.6% slower
fixed_width_array_scan_matrix[best-exact_early-32]	0.0001ms	0.0001ms	43.1% slower
fixed_width_array_scan_matrix[first-exact_early-16]	0.0001ms	0.0001ms	36.9% slower
fixed_width_array_scan_matrix[first-exact_early-32]	0.0001ms	0.0001ms	34.4% slower
fixed_width_array_scan_matrix[first-exact_mid-16]	0.0023ms	0.0032ms	42.7% slower
fixed_width_array_scan_matrix[first-exact_mid-32]	0.0027ms	0.0037ms	37.1% slower
fixed_width_array_scan_matrix[first-threshold_d-16]	0.0023ms	0.0032ms	42.7% slower
fixed_width_array_scan_matrix[first-threshold_d_plus_1-16]	0.0023ms	0.0032ms	42.7% slower
fixed_width_array_scan_matrix[first-threshold_d_plus_1-32]	0.0027ms	0.0037ms	35.9% slower
hd_bytes[1000-diff]	0.0001ms	0.0001ms	58.4% slower
hd_bytes[1000-same]	0.0001ms	0.0001ms	53.1% slower
hd_bytes[1024-diff]	0.0001ms	0.0001ms	29.5% slower
hd_bytes[1024-same]	0.0001ms	0.0001ms	22.7% slower
hd_bytes[3-diff]	0.0001ms	0.0002ms	172.7% slower
hd_bytes[3-same]	0.0001ms	0.0001ms	22.3% slower
hd_bytes[64-diff]	0.0001ms	0.0001ms	25.9% slower
hd_bytes_buffer[bytearray]	0.0001ms	0.0001ms	28.6% slower
hd_bytes_buffer[memoryview]	0.0001ms	0.0001ms	26.5% slower
hd_bytes_gil_boundary[16376]	0.0003ms	0.0006ms	83.0% slower
hd_bytes_gil_boundary[16384]	0.0004ms	0.0006ms	49.2% slower
hd_bytes_gil_boundary[16392]	0.0004ms	0.0006ms	50.5% slower
hd_bytes_random[1024]	0.0001ms	0.0001ms	50.8% slower
hd_bytes_random[16]	0.0001ms	0.0001ms	22.7% slower
hd_bytes_random[64]	0.0001ms	0.0001ms	27.3% slower
hd_string[1000-diff]	0.0001ms	0.0002ms	73.0% slower
hd_string[1000-same]	0.0001ms	0.0002ms	63.1% slower
hd_string[1024-diff]	0.0001ms	0.0002ms	52.4% slower
hd_string[1024-same]	0.0001ms	0.0002ms	42.2% slower
hd_string[3-diff]	0.0001ms	0.0001ms	31.7% faster
hd_string[3-same]	0.0001ms	0.0001ms	30.0% slower
hd_string[64-diff]	0.0001ms	0.0001ms	7.9% slower
hd_string_random[1024]	0.0001ms	0.0002ms	50.3% slower
hd_string_random[16]	0.0001ms	0.0001ms	43.4% slower
hd_string_random[64]	0.0001ms	0.0001ms	18.4% slower
stdlib_bytes_distance_random[1024]	0.0021ms	0.0028ms	34.6% slower
stdlib_bytes_distance_random[16]	0.0003ms	0.0004ms	53.0% slower
stdlib_bytes_distance_random[64]	0.0003ms	0.0005ms	52.2% slower
stdlib_hex_distance_random[1024]	0.0029ms	0.0036ms	23.2% slower
stdlib_hex_distance_random[16]	0.0002ms	0.0004ms	61.4% slower
stdlib_hex_distance_random[64]	0.0004ms	0.0005ms	47.0% slower

Legend: ✅ Faster (>5%) · ⚠️ Slower (5-30%) · ❌ Regression (>30%) · 🆕 New · 🗑️ Removed

---

⛔ This PR has severe performance regressions (>20% slower). Please investigate before merging.

[mrecachinas]

C4 AVX-512 hardware is now available and the initial three-run benchmark archive has been captured on an Intel Xeon Platinum 8581C (Emerald Rapids) with avx2, avx512bw, and avx512_bitalg exposed.

Initial median findings:

• AVX-512 hex strings are 1.18x-2.40x faster than AVX2 across 16-254 characters.
• AVX-512 byte distance is 1.53x faster at 64 bytes and 3.00x faster at 127 bytes.
• Below 64 bytes, the existing AVX fallback chain is slower than the scalar classic implementation in the forced-algorithm matrix.
• Rust Criterion run-to-run ranges stayed below 2%.

Commit 9067fd5 rebases this PR onto current main, adds native dispatch plus 48/63/65/96-byte threshold cases, and allows targeted Rust-only harness runs. The next step is a targeted baseline on the same C4 VM before changing dispatch.

[mrecachinas]

Masked short-input experiment 6007179 passes the performance gate and is retained. Three-run medians on the same C4 VM:

Bytes	Before	Masked AVX-512	Improvement
8	4.299 ns	4.057 ns	5.9%
16	5.380 ns	4.048 ns	32.9%
32	6.112 ns	4.068 ns	50.2%
48	6.924 ns	4.071 ns	70.1%
63	11.955 ns	4.068 ns	193.9%
64	3.612 ns	3.576 ns	1.0%

The implementation removes the premature AVX2 fallback and uses the AVX-512 BITALG masked-tail code that was already present for residual bytes. The next experiment is cross-record SIMD for 16- and 32-byte catalogs.

[mrecachinas]

Preliminary cross-record scanner results for ef9e31b are strongly positive on the C4 VM. Three-run medians against parent 6007179:

Workload	Before	After	Speedup
512x16 first, random/no-match	1.025 us	311 ns	3.29x
512x16 best, random/no-match	1.224 us	324 ns	3.77x
512x16 all, random/no-match	1.110 us	372 ns	2.98x
1024x16 first	2.030 us	647 ns	3.14x
1024x16 best	2.420 us	628 ns	3.85x
1024x16 all	2.223 us	730 ns	3.05x
1024x32 first	3.606 us	1.315 us	2.74x
1024x32 best	4.140 us	1.478 us	2.80x
1024x32 all	3.886 us	1.427 us	2.72x

All nine cases improved; after-run ranges were generally 1.4%-4.2%, with one 8.3% all case. A full scenario matrix is still required before retention because four-record batching may affect early-match performance.

[mrecachinas]

The hybrid cross-record scanner in 62c8a7e is retained after early-match validation. Compared with parent 6007179:

Scenario	Case	Before	Hybrid	Result
random/no-match	16B first	2.030 us	587 ns	3.46x
random/no-match	16B best	2.420 us	594 ns	4.07x
random/no-match	16B all	2.223 us	686 ns	3.24x
random/no-match	32B first	3.606 us	1.281 us	2.82x
random/no-match	32B best	4.140 us	1.432 us	2.89x
random/no-match	32B all	3.886 us	1.401 us	2.77x
exact at index 0	16B first	4.05 ns	4.54 ns	0.49 ns slower
exact at index 0	16B best	5.63 ns	6.26 ns	0.63 ns slower
exact at index 0	32B first	6.43 ns	5.16 ns	1.25x
exact at index 0	32B best	8.36 ns	7.43 ns	1.13x

The small 16B index-0 trade-off is accepted in exchange for the 3.24x-4.07x full-scan gains. all remains 2.55x-3.00x faster even when the exact record is first. Commit 3494c29 also corrects the direct scanner test to use exact-match threshold 0; the previous -1 correctly made every record eligible.