Remove remaining array and Python overhead

[mrecachinas]

Why

The first performance pass made the individual Hamming-distance kernels very fast. Follow-up profiling showed that batch scans and Python wrappers were still spending meaningful time on repeated setup, thread scheduling, and buffer handling.

This PR removes that remaining overhead without changing the public API or results.

What changed

• Choose the byte kernel once per batch. Array scans no longer reload and match the selected algorithm for every record.
• Use a fast path for 16-byte records. This common width is handled with two native 64-bit loads and popcounts. Larger fixed-width experiments were slower and were not retained.
• Start Rayon only when it helps. Parallel scanning now begins at the measured 5 MiB crossover, avoiding a severe latency cliff on medium batches.
• Make generic Python buffers cheaper. bytearray, memoryview, and other contiguous buffers use a stack-pinned buffer guard instead of boxed PyBuffer setup.
• Tune GIL release paths. Bytes remain attached until 16 KiB, while immutable strings can release the GIL without copying their contents.
• Broaden performance coverage. Criterion and pytest-benchmark now include random inputs, no-match scans, standard-library comparisons, and dispatch boundaries.

Representative results

These comparisons use a same-state main baseline to avoid CPU-frequency differences between benchmark sessions.

Workload	main	This PR	Improvement
Rust 512×16 all	1,049 ns	449 ns	57.2%
Rust 512×16 first, match last	912 ns	402 ns	55.9%
Rust 16,384×64 best	55.8 µs	11.0 µs	80.3%
Rust 16,384×64 all	65.7 µs	20.4 µs	69.0%
Python 512×16 all	1,125 ns	538 ns	52.2%
Python 16,384×64 best	73.3 µs	32.0 µs	56.3%
Python 16,384×64 all	72.2 µs	30.7 µs	57.5%

Additional isolated measurements show 64-byte bytearray calls improving 38.2%, 1 MiB hex calls improving 26.0% by avoiding copies, and the 4 KiB byte boundary improving 26.5%.

Safety and compatibility

• No public API or result changes.
• Batch ordering and tie behavior remain unchanged.
• Generic buffers are still required to be contiguous and byte-formatted.
• Large calls still release the GIL so other Python threads can run.
• The pinned guard ensures Python’s self-referential Py_buffer is never moved after acquisition.

Testing

• cargo test --no-default-features (74 unit tests, 2 doctests)
• cargo fmt --check
• ruff check .
• ruff format --check .
• python -m pytest -q . (140 passed, 3 skipped)
• Three complete Criterion passes
• Three complete pytest-benchmark passes

Follow-up

• Investigate remaining SIMD performance opportunities #51 tracks the only remaining material opportunities: x86-specific SIMD on real hardware and cross-record SIMD batching.

[github-actions]

📊 Benchmark Comparison Results

Status	Benchmark	Base	PR	Delta
❌	check_bytes_arrays_all_within_dist[16384 elems,s=64,mid]	0.0786ms	0.1041ms	32.4% slower
❌	check_bytes_arrays_all_within_dist[16384 elems,s=64,at 0]	0.0789ms	0.1041ms	31.9% slower
❌	check_bytes_arrays_all_within_dist[16384 elems,s=64,end]	0.0794ms	0.1039ms	30.8% slower
⚠️	check_bytes_arrays_best_within_dist[16384 elems,s=64,end]	0.0853ms	0.1088ms	27.6% slower
⚠️	check_bytes_arrays_best_within_dist[16384 elems,s=64,at 0]	0.0857ms	0.1089ms	27.1% slower
⚠️	check_bytes_arrays_best_within_dist[16384 elems,s=64,mid]	0.0857ms	0.1087ms	26.8% slower
✅	check_bytes_arrays_first_within_dist[ 512 elems,s=16,end]	0.0045ms	0.0032ms	28.3% faster
✅	check_bytes_arrays_best_within_dist[ 512 elems,s=16,end]	0.0047ms	0.0034ms	26.9% faster
✅	check_bytes_arrays_first_within_dist[ 1024 elems,s=32,end]	0.0097ms	0.0072ms	26.1% faster
✅	check_bytes_arrays_first_within_dist[ 1024 elems,s=32,mid]	0.0049ms	0.0037ms	25.7% faster
✅	check_bytes_arrays_best_within_dist[ 1024 elems,s=32,at 0]	0.0100ms	0.0075ms	25.1% faster
✅	check_bytes_arrays_best_within_dist[ 1024 elems,s=32,end]	0.0100ms	0.0075ms	25.1% faster
✅	check_bytes_arrays_best_within_dist[ 1024 elems,s=32,mid]	0.0100ms	0.0075ms	25.0% faster
✅	check_bytes_arrays_first_within_dist[16384 elems,s=64,mid]	0.0717ms	0.0542ms	24.4% faster
✅	check_bytes_arrays_first_within_dist[16384 elems,s=64,end]	0.1440ms	0.1130ms	21.5% faster
✅	check_bytes_arrays_all_within_dist[ 1024 elems,s=32,at 0]	0.0092ms	0.0073ms	20.8% faster
✅	check_bytes_arrays_all_within_dist[ 1024 elems,s=32,mid]	0.0092ms	0.0073ms	20.6% faster
✅	check_bytes_arrays_all_within_dist[ 1024 elems,s=32,end]	0.0092ms	0.0073ms	20.5% faster
🆕	hd_bytes_buffer[bytearray]	N/A	0.0002ms	NEW
🆕	hd_bytes_buffer[memoryview]	N/A	0.0001ms	NEW
🆕	hd_bytes_gil_boundary[16376]	N/A	0.0006ms	NEW
🆕	hd_bytes_gil_boundary[16384]	N/A	0.0006ms	NEW
🆕	hd_bytes_gil_boundary[16392]	N/A	0.0006ms	NEW
🆕	hd_bytes_random[1024]	N/A	0.0001ms	NEW
🆕	hd_bytes_random[16]	N/A	0.0001ms	NEW
🆕	hd_bytes_random[64]	N/A	0.0001ms	NEW
🆕	hd_string_random[1024]	N/A	0.0002ms	NEW
🆕	hd_string_random[16]	N/A	0.0001ms	NEW
🆕	hd_string_random[64]	N/A	0.0001ms	NEW
🆕	stdlib_bytes_distance_random[1024]	N/A	0.0028ms	NEW
🆕	stdlib_bytes_distance_random[16]	N/A	0.0004ms	NEW
🆕	stdlib_bytes_distance_random[64]	N/A	0.0005ms	NEW
🆕	stdlib_hex_distance_random[1024]	N/A	0.0036ms	NEW
🆕	stdlib_hex_distance_random[16]	N/A	0.0004ms	NEW
🆕	stdlib_hex_distance_random[64]	N/A	0.0005ms	NEW

➖ 27 benchmarks within noise (click to expand)

Benchmark	Base	PR	Delta
check_bytes_arrays_all_within_dist[ 512 elems,s=16,at 0]	0.0042ms	0.0033ms	22.2% faster
check_bytes_arrays_all_within_dist[ 512 elems,s=16,end]	0.0043ms	0.0033ms	22.3% faster
check_bytes_arrays_all_within_dist[ 512 elems,s=16,mid]	0.0043ms	0.0033ms	22.2% faster
check_bytes_arrays_best_within_dist[ 512 elems,s=16,at 0]	0.0001ms	0.0001ms	11.1% faster
check_bytes_arrays_best_within_dist[ 512 elems,s=16,mid]	0.0024ms	0.0018ms	27.2% faster
check_bytes_arrays_first_within_dist[ 512 elems,s=16,at 0]	0.0001ms	0.0001ms	2.2% faster
check_bytes_arrays_first_within_dist[ 512 elems,s=16,mid]	0.0023ms	0.0016ms	27.8% faster
check_bytes_arrays_first_within_dist[ 1024 elems,s=32,at 0]	0.0001ms	0.0001ms	1.5% faster
check_bytes_arrays_first_within_dist[16384 elems,s=64,at 0]	0.0002ms	0.0002ms	20.5% faster
check_bytes_within_dist[127 bytes,d=500]	0.0001ms	0.0001ms	5.3% faster
check_bytes_within_dist[16 bytes,d=0]	0.0001ms	0.0001ms	5.8% faster
check_bytes_within_dist[64 bytes,d=100]	0.0001ms	0.0001ms	3.6% slower
hex_within_dist	0.0001ms	0.0001ms	12.6% slower
hd_bytes[1000-diff]	0.0001ms	0.0001ms	19.5% slower
hd_bytes[1000-same]	0.0001ms	0.0002ms	111.7% slower
hd_bytes[1024-diff]	0.0001ms	0.0001ms	6.6% slower
hd_bytes[1024-same]	0.0001ms	0.0002ms	130.6% slower
hd_bytes[3-diff]	0.0002ms	0.0001ms	50.7% faster
hd_bytes[3-same]	0.0001ms	0.0001ms	7.2% slower
hd_bytes[64-diff]	0.0001ms	0.0001ms	9.2% slower
hd_string[1000-diff]	0.0002ms	0.0002ms	12.0% faster
hd_string[1000-same]	0.0002ms	0.0002ms	15.6% faster
hd_string[1024-diff]	0.0002ms	0.0002ms	5.8% faster
hd_string[1024-same]	0.0002ms	0.0002ms	9.5% faster
hd_string[3-diff]	0.0002ms	0.0002ms	0.0% faster
hd_string[3-same]	0.0001ms	0.0001ms	11.9% slower
hd_string[64-diff]	0.0001ms	0.0001ms	2.0% faster

Legend: ✅ Faster (>5%) · ⚠️ Slower (5-30%) · ❌ Regression (>30%) · 🆕 New · 🗑️ Removed

---

⛔ This PR has severe performance regressions (>20% slower). Please investigate before merging.