perf: optimize FFI boundary and SIMD hot paths; fix AVX-512 string overflow

[mrecachinas]

Summary

Performance review across the FFI boundary and SIMD hot paths, plus a correctness fix for AVX-512. Driven by a multi-subagent review of the Python/FFI, NEON, x86 SIMD, and dispatch/batch axes.

Changes

Python FFI (src/python.rs)

• Added GIL_RELEASE_THRESHOLD (4096): short hex strings now compute on borrowed bytes without a redundant to_vec() or GIL release/acquire round-trip.
• Wired check_bytes_arrays_{first,best,all}_within_dist to the rayon api.rs implementations so batch calls use multiple cores, preserving Python-side validation, error messages, and sentinels.

NEON (src/neon_simd.rs)

• Added pack32_xor_neon helper (immediate vshlq_n_u8 shift, dedup parse+pack).
• Batched popcounts into a vector accumulator and OR-combined validation masks, reducing/validating once per batch. _with_max checks invalid-hex before the max_dist sentinel to preserve original semantics.

x86 (src/x86_simd.rs)

• Correctness fix: hamming_distance_string_avx512 accumulated epi8 lanes unbounded, producing silently wrong results for strings >~4032 chars. Now flushes via SAD into a wide accumulator every 32 iterations.
• Added #[inline] to popcnt128_shuffle / popcnt256_shuffle.

Tests (src/tests.rs)

• Added long-string overflow regression tests, including an AVX-512-gated case.

Measured impact (Apple Silicon / NEON host)

Path	Speedup
hamming_distance_string (short)	~1.7-2.1x
hamming_distance_bytes (short)	~1.2-1.3x
NEON large strings	~14-18%
NEON small strings	neutral
check_bytes_arrays_*	multi-core scaling for large arrays

x86 changes (AVX-512 fix, #[inline]) validated by cross-compilation + regression tests; not runtime-benchmarked (no x86 host available).

Validation

• 72 Rust tests pass; 122 pytest pass (2 skipped).
• cargo fmt --check, ruff check, ruff format --check all green.

[github-actions]

📊 Benchmark Comparison Results

Status	Benchmark	Base	PR	Delta
✅	check_bytes_arrays_all_within_dist[16384 elems,s=64,at 0]	0.1285ms	0.0860ms	33.1% faster
✅	check_bytes_arrays_all_within_dist[16384 elems,s=64,mid]	0.1276ms	0.0860ms	32.6% faster
✅	check_bytes_arrays_all_within_dist[16384 elems,s=64,end]	0.1279ms	0.0867ms	32.3% faster
✅	check_bytes_arrays_best_within_dist[16384 elems,s=64,mid]	0.1334ms	0.0980ms	26.6% faster
✅	check_bytes_arrays_best_within_dist[16384 elems,s=64,at 0]	0.1332ms	0.0992ms	25.5% faster

➖ 40 benchmarks within noise (click to expand)

Benchmark	Base	PR	Delta
check_bytes_arrays_all_within_dist[ 512 elems,s=16,at 0]	0.0039ms	0.0038ms	3.8% faster
check_bytes_arrays_all_within_dist[ 512 elems,s=16,end]	0.0039ms	0.0038ms	3.8% faster
check_bytes_arrays_all_within_dist[ 512 elems,s=16,mid]	0.0039ms	0.0038ms	3.8% faster
check_bytes_arrays_all_within_dist[ 1024 elems,s=32,at 0]	0.0085ms	0.0083ms	2.8% faster
check_bytes_arrays_all_within_dist[ 1024 elems,s=32,end]	0.0085ms	0.0083ms	2.8% faster
check_bytes_arrays_all_within_dist[ 1024 elems,s=32,mid]	0.0085ms	0.0083ms	2.7% faster
check_bytes_arrays_best_within_dist[ 512 elems,s=16,at 0]	0.0046ms	0.0051ms	11.5% slower
check_bytes_arrays_best_within_dist[ 512 elems,s=16,end]	0.0040ms	0.0044ms	10.8% slower
check_bytes_arrays_best_within_dist[ 512 elems,s=16,mid]	0.0043ms	0.0048ms	11.2% slower
check_bytes_arrays_best_within_dist[ 1024 elems,s=32,at 0]	0.0087ms	0.0096ms	10.2% slower
check_bytes_arrays_best_within_dist[ 1024 elems,s=32,end]	0.0088ms	0.0096ms	10.1% slower
check_bytes_arrays_best_within_dist[ 1024 elems,s=32,mid]	0.0088ms	0.0097ms	10.4% slower
check_bytes_arrays_best_within_dist[16384 elems,s=64,end]	0.1334ms	0.0999ms	25.1% faster
check_bytes_arrays_first_within_dist[ 512 elems,s=16,at 0]	0.0002ms	0.0002ms	1.0% slower
check_bytes_arrays_first_within_dist[ 512 elems,s=16,end]	0.0039ms	0.0040ms	4.7% slower
check_bytes_arrays_first_within_dist[ 512 elems,s=16,mid]	0.0020ms	0.0021ms	4.7% slower
check_bytes_arrays_first_within_dist[ 1024 elems,s=32,at 0]	0.0002ms	0.0002ms	3.3% slower
check_bytes_arrays_first_within_dist[ 1024 elems,s=32,end]	0.0085ms	0.0089ms	4.4% slower
check_bytes_arrays_first_within_dist[ 1024 elems,s=32,mid]	0.0044ms	0.0046ms	4.1% slower
check_bytes_arrays_first_within_dist[16384 elems,s=64,at 0]	0.0002ms	0.0002ms	5.4% slower
check_bytes_arrays_first_within_dist[16384 elems,s=64,end]	0.1331ms	0.1327ms	0.3% faster
check_bytes_arrays_first_within_dist[16384 elems,s=64,mid]	0.0662ms	0.0632ms	4.6% faster
check_bytes_within_dist[127 bytes,d=500]	0.0002ms	0.0002ms	2.5% faster
check_bytes_within_dist[16 bytes,d=0]	0.0002ms	0.0002ms	1.0% faster
check_bytes_within_dist[64 bytes,d=100]	0.0002ms	0.0002ms	0.3% faster
hex_within_dist	0.0001ms	0.0001ms	1.0% slower
hd_bytes[1000-diff]	0.0002ms	0.0002ms	20.7% faster
hd_bytes[1000-same]	0.0002ms	0.0001ms	21.9% faster
hd_bytes[1024-diff]	0.0002ms	0.0001ms	24.1% faster
hd_bytes[1024-same]	0.0002ms	0.0001ms	25.1% faster
hd_bytes[3-diff]	0.0002ms	0.0001ms	25.9% faster
hd_bytes[3-same]	0.0002ms	0.0001ms	23.7% faster
hd_bytes[64-diff]	0.0002ms	0.0001ms	28.1% faster
hd_string[1000-diff]	0.0003ms	0.0002ms	41.3% faster
hd_string[1000-same]	0.0003ms	0.0002ms	39.9% faster
hd_string[1024-diff]	0.0003ms	0.0001ms	42.8% faster
hd_string[1024-same]	0.0003ms	0.0002ms	40.9% faster
hd_string[3-diff]	0.0002ms	0.0002ms	27.3% faster
hd_string[3-same]	0.0001ms	0.0001ms	42.9% faster
hd_string[64-diff]	0.0001ms	0.0001ms	41.1% faster

Legend: ✅ Faster (>5%) · ⚠️ Slower (5-30%) · ❌ Regression (>30%) · 🆕 New · 🗑️ Removed