Skip to content

Speed up AVX-512 byte and fixed-width array scans - #55

Merged
mrecachinas merged 8 commits into
mainfrom
perf/avx-simd-investigation
Aug 3, 2026
Merged

Speed up AVX-512 byte and fixed-width array scans#55
mrecachinas merged 8 commits into
mainfrom
perf/avx-simd-investigation

Conversation

@mrecachinas

@mrecachinas mrecachinas commented Aug 2, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

What

Improves the AVX-512 BITALG paths identified in #51 and adds a repeatable three-run x86 benchmark harness.

  • Uses the existing masked AVX-512 tail implementation for byte inputs below 64 bytes instead of falling back through AVX2/SSE.
  • Adds AVX-512 cross-record scanners for 16- and 32-byte catalogs, processing four records per batch.
  • Preserves first-index ordering, lowest-index best ties, ordered all-results, threshold semantics, exact-match termination, and scalar tails.
  • Keeps explicit classic/SSE/AVX2 selections unchanged; the new scanners engage only for native or avx512 when AVX-512 BITALG is available.
  • Expands the x86 threshold and scanner benchmark matrices.

Advances the AVX-512 portion of #51. AVX2-only tuning remains a separately measured follow-up.

Performance

Three-run medians on a Google Cloud c4-standard-4, Intel Xeon Platinum 8581C (Emerald Rapids), with avx2, avx512bw, and avx512_bitalg exposed.

Python fixed-width scans

Workload Before After Speedup
1024x16 first, random/no-match 3.222 us 0.679 us 4.75x
1024x16 best, random/no-match 3.720 us 0.651 us 5.72x
1024x16 all, random/no-match 3.466 us 0.710 us 4.88x
1024x32 first, random/no-match 3.199 us 1.278 us 2.50x
1024x32 best, random/no-match 3.729 us 1.383 us 2.70x
1024x32 all, random/no-match 3.445 us 1.377 us 2.50x

Short AVX-512 byte inputs

Bytes Before After Improvement
16 5.380 ns 4.048 ns 32.9%
32 6.112 ns 4.068 ns 50.2%
48 6.924 ns 4.071 ns 70.1%
63 11.955 ns 4.068 ns 193.9%

The hybrid scanner scalar-checks the first record before batching. The accepted worst-case trade-off is approximately 0.5-0.6 ns for 16-byte first/best when an exact match is at index 0; full scans improve by 2.77x-4.07x in Rust and 2.50x-5.72x through Python.

Testing

  • cargo test --lib --no-default-features on the AVX-512 C4 VM — 72 tests
  • cargo test --no-default-features on ARM64 — 76 unit and 2 doc tests
  • python -m pytest test/ -k "not bench" — 80 passed, 3 skipped
  • x86 benchmark comparisons use three-run medians on the same VM
  • Independent correctness and safety review found no scanner defects
  • cargo fmt --check
  • git diff --check

@github-actions

github-actions Bot commented Aug 2, 2026 •
edited
Loading

Copy link
Copy Markdown

📊 Benchmark Comparison Results

Status Benchmark Base PR Delta
❌ check_bytes_arrays_best_within_dist[16384 elems,s=64,at 0] 0.0539ms 0.1232ms 128.4% slower
❌ check_bytes_arrays_best_within_dist[16384 elems,s=64,mid] 0.0538ms 0.1229ms 128.3% slower
❌ check_bytes_arrays_best_within_dist[16384 elems,s=64,end] 0.0544ms 0.1230ms 126.2% slower
❌ check_bytes_arrays_all_within_dist[16384 elems,s=64,mid] 0.0539ms 0.1132ms 109.9% slower
❌ check_bytes_arrays_all_within_dist[16384 elems,s=64,end] 0.0541ms 0.1134ms 109.5% slower
❌ check_bytes_arrays_first_within_dist[16384 elems,s=64,mid] 0.0248ms 0.0516ms 108.5% slower
❌ check_bytes_arrays_all_within_dist[16384 elems,s=64,at 0] 0.0543ms 0.1132ms 108.5% slower
❌ check_bytes_arrays_first_within_dist[16384 elems,s=64,end] 0.0540ms 0.1029ms 90.5% slower
❌ check_bytes_arrays_best_within_dist[ 512 elems,s=16,end] 0.0026ms 0.0039ms 51.1% slower
❌ fixed_width_array_scan_matrix[best-exact_mid-16] 0.0026ms 0.0039ms 50.9% slower
❌ fixed_width_array_scan_matrix[best-random_no_match-16] 0.0050ms 0.0075ms 49.7% slower
❌ fixed_width_array_scan_matrix[best-threshold_d_minus_1-16] 0.0050ms 0.0075ms 49.7% slower
❌ fixed_width_array_scan_matrix[best-exact_late-16] 0.0050ms 0.0075ms 49.6% slower
❌ fixed_width_array_scan_matrix[best-threshold_d-16] 0.0050ms 0.0075ms 49.5% slower
❌ fixed_width_array_scan_matrix[best-threshold_d_plus_1-16] 0.0051ms 0.0075ms 46.9% slower
❌ fixed_width_array_scan_matrix[best-exact_mid-32] 0.0030ms 0.0043ms 45.7% slower
❌ fixed_width_array_scan_matrix[best-threshold_d_plus_1-32] 0.0059ms 0.0085ms 44.0% slower
❌ check_bytes_arrays_best_within_dist[ 1024 elems,s=32,mid] 0.0059ms 0.0085ms 44.0% slower
❌ fixed_width_array_scan_matrix[best-threshold_d-32] 0.0059ms 0.0085ms 44.0% slower
❌ fixed_width_array_scan_matrix[best-random_no_match-32] 0.0059ms 0.0084ms 43.9% slower
❌ check_bytes_arrays_best_within_dist[ 1024 elems,s=32,end] 0.0059ms 0.0085ms 43.8% slower
❌ fixed_width_array_scan_matrix[best-exact_late-32] 0.0059ms 0.0085ms 43.8% slower
❌ check_bytes_arrays_best_within_dist[ 1024 elems,s=32,at 0] 0.0059ms 0.0084ms 43.8% slower
❌ fixed_width_array_scan_matrix[best-threshold_d_minus_1-32] 0.0059ms 0.0084ms 43.7% slower
❌ check_bytes_arrays_all_within_dist[ 512 elems,s=16,mid] 0.0025ms 0.0036ms 43.0% slower
❌ check_bytes_arrays_all_within_dist[ 512 elems,s=16,end] 0.0025ms 0.0036ms 42.8% slower
❌ check_bytes_arrays_all_within_dist[ 512 elems,s=16,at 0] 0.0025ms 0.0036ms 41.4% slower
❌ fixed_width_array_scan_matrix[all-exact_mid-16] 0.0049ms 0.0070ms 41.1% slower
❌ fixed_width_array_scan_matrix[all-random_no_match-16] 0.0049ms 0.0069ms 41.1% slower
❌ fixed_width_array_scan_matrix[all-exact_late-16] 0.0049ms 0.0069ms 40.9% slower
❌ fixed_width_array_scan_matrix[all-threshold_d-16] 0.0049ms 0.0069ms 40.9% slower
❌ fixed_width_array_scan_matrix[all-threshold_d_minus_1-16] 0.0049ms 0.0068ms 40.9% slower
❌ fixed_width_array_scan_matrix[all-exact_early-16] 0.0049ms 0.0069ms 40.9% slower
❌ fixed_width_array_scan_matrix[first-exact_late-16] 0.0044ms 0.0062ms 40.0% slower
❌ fixed_width_array_scan_matrix[first-threshold_d_minus_1-16] 0.0044ms 0.0062ms 39.9% slower
❌ fixed_width_array_scan_matrix[first-random_no_match-16] 0.0044ms 0.0062ms 39.8% slower
❌ check_bytes_arrays_first_within_dist[ 1024 elems,s=32,mid] 0.0027ms 0.0037ms 37.4% slower
❌ fixed_width_array_scan_matrix[first-threshold_d-32] 0.0027ms 0.0037ms 37.4% slower
❌ fixed_width_array_scan_matrix[all-threshold_d-32] 0.0058ms 0.0079ms 36.8% slower
❌ check_bytes_arrays_all_within_dist[ 1024 elems,s=32,mid] 0.0058ms 0.0079ms 36.8% slower
❌ fixed_width_array_scan_matrix[all-threshold_d_plus_1-32] 0.0058ms 0.0079ms 36.8% slower
❌ check_bytes_arrays_all_within_dist[ 1024 elems,s=32,at 0] 0.0058ms 0.0079ms 36.7% slower
❌ fixed_width_array_scan_matrix[all-exact_early-32] 0.0058ms 0.0079ms 36.7% slower
❌ fixed_width_array_scan_matrix[all-random_no_match-32] 0.0057ms 0.0078ms 36.6% slower
❌ fixed_width_array_scan_matrix[all-exact_mid-32] 0.0058ms 0.0079ms 36.5% slower
❌ fixed_width_array_scan_matrix[all-threshold_d_minus_1-32] 0.0057ms 0.0078ms 36.5% slower
❌ check_bytes_arrays_all_within_dist[ 1024 elems,s=32,end] 0.0058ms 0.0079ms 36.4% slower
❌ fixed_width_array_scan_matrix[all-exact_late-32] 0.0058ms 0.0079ms 36.4% slower
❌ fixed_width_array_scan_matrix[first-threshold_d_minus_1-32] 0.0053ms 0.0072ms 35.8% slower
❌ fixed_width_array_scan_matrix[first-random_no_match-32] 0.0053ms 0.0072ms 35.5% slower
❌ fixed_width_array_scan_matrix[first-exact_late-32] 0.0053ms 0.0072ms 35.5% slower
❌ check_bytes_arrays_first_within_dist[ 1024 elems,s=32,end] 0.0053ms 0.0072ms 35.5% slower
❌ fixed_width_array_scan_matrix[all-threshold_d_plus_1-16] 0.0051ms 0.0069ms 34.9% slower
➖ 51 benchmarks within noise (click to expand)
Benchmark Base PR Delta
check_bytes_arrays_best_within_dist[ 512 elems,s=16,at 0] 0.0001ms 0.0001ms 40.8% slower
check_bytes_arrays_best_within_dist[ 512 elems,s=16,mid] 0.0013ms 0.0020ms 49.3% slower
check_bytes_arrays_first_within_dist[ 512 elems,s=16,at 0] 0.0001ms 0.0001ms 37.0% slower
check_bytes_arrays_first_within_dist[ 512 elems,s=16,end] 0.0023ms 0.0032ms 42.5% slower
check_bytes_arrays_first_within_dist[ 512 elems,s=16,mid] 0.0012ms 0.0016ms 41.6% slower
check_bytes_arrays_first_within_dist[ 1024 elems,s=32,at 0] 0.0001ms 0.0001ms 31.7% slower
check_bytes_arrays_first_within_dist[16384 elems,s=64,at 0] 0.0001ms 0.0002ms 48.0% slower
check_bytes_within_dist[127 bytes,d=500] 0.0001ms 0.0001ms 66.7% slower
check_bytes_within_dist[16 bytes,d=0] 0.0001ms 0.0001ms 31.2% slower
check_bytes_within_dist[64 bytes,d=100] 0.0001ms 0.0001ms 47.9% slower
hex_within_dist 0.0001ms 0.0001ms 24.1% slower
fixed_width_array_scan_matrix[best-exact_early-16] 0.0001ms 0.0001ms 41.6% slower
fixed_width_array_scan_matrix[best-exact_early-32] 0.0001ms 0.0001ms 43.1% slower
fixed_width_array_scan_matrix[first-exact_early-16] 0.0001ms 0.0001ms 36.9% slower
fixed_width_array_scan_matrix[first-exact_early-32] 0.0001ms 0.0001ms 34.4% slower
fixed_width_array_scan_matrix[first-exact_mid-16] 0.0023ms 0.0032ms 42.7% slower
fixed_width_array_scan_matrix[first-exact_mid-32] 0.0027ms 0.0037ms 37.1% slower
fixed_width_array_scan_matrix[first-threshold_d-16] 0.0023ms 0.0032ms 42.7% slower
fixed_width_array_scan_matrix[first-threshold_d_plus_1-16] 0.0023ms 0.0032ms 42.7% slower
fixed_width_array_scan_matrix[first-threshold_d_plus_1-32] 0.0027ms 0.0037ms 35.9% slower
hd_bytes[1000-diff] 0.0001ms 0.0001ms 58.4% slower
hd_bytes[1000-same] 0.0001ms 0.0001ms 53.1% slower
hd_bytes[1024-diff] 0.0001ms 0.0001ms 29.5% slower
hd_bytes[1024-same] 0.0001ms 0.0001ms 22.7% slower
hd_bytes[3-diff] 0.0001ms 0.0002ms 172.7% slower
hd_bytes[3-same] 0.0001ms 0.0001ms 22.3% slower
hd_bytes[64-diff] 0.0001ms 0.0001ms 25.9% slower
hd_bytes_buffer[bytearray] 0.0001ms 0.0001ms 28.6% slower
hd_bytes_buffer[memoryview] 0.0001ms 0.0001ms 26.5% slower
hd_bytes_gil_boundary[16376] 0.0003ms 0.0006ms 83.0% slower
hd_bytes_gil_boundary[16384] 0.0004ms 0.0006ms 49.2% slower
hd_bytes_gil_boundary[16392] 0.0004ms 0.0006ms 50.5% slower
hd_bytes_random[1024] 0.0001ms 0.0001ms 50.8% slower
hd_bytes_random[16] 0.0001ms 0.0001ms 22.7% slower
hd_bytes_random[64] 0.0001ms 0.0001ms 27.3% slower
hd_string[1000-diff] 0.0001ms 0.0002ms 73.0% slower
hd_string[1000-same] 0.0001ms 0.0002ms 63.1% slower
hd_string[1024-diff] 0.0001ms 0.0002ms 52.4% slower
hd_string[1024-same] 0.0001ms 0.0002ms 42.2% slower
hd_string[3-diff] 0.0001ms 0.0001ms 31.7% faster
hd_string[3-same] 0.0001ms 0.0001ms 30.0% slower
hd_string[64-diff] 0.0001ms 0.0001ms 7.9% slower
hd_string_random[1024] 0.0001ms 0.0002ms 50.3% slower
hd_string_random[16] 0.0001ms 0.0001ms 43.4% slower
hd_string_random[64] 0.0001ms 0.0001ms 18.4% slower
stdlib_bytes_distance_random[1024] 0.0021ms 0.0028ms 34.6% slower
stdlib_bytes_distance_random[16] 0.0003ms 0.0004ms 53.0% slower
stdlib_bytes_distance_random[64] 0.0003ms 0.0005ms 52.2% slower
stdlib_hex_distance_random[1024] 0.0029ms 0.0036ms 23.2% slower
stdlib_hex_distance_random[16] 0.0002ms 0.0004ms 61.4% slower
stdlib_hex_distance_random[64] 0.0004ms 0.0005ms 47.0% slower

Legend: ✅ Faster (>5%) · ⚠️ Slower (5-30%) · ❌ Regression (>30%) · 🆕 New · 🗑️ Removed

⛔ This PR has severe performance regressions (>20% slower). Please investigate before merging.

@mrecachinas
Add repeatable x86 SIMD benchmark harness
3e5f4d5 
Capture CPU metadata and run Criterion plus end-to-end Python benchmarks three times on the same AVX2 or AVX-512 machine.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 01071e57-12fd-4f42-bc67-e2b274d0b007
@mrecachinas
Expand x86 threshold benchmark coverage
9067fd5 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 01071e57-12fd-4f42-bc67-e2b274d0b007
@mrecachinas
mrecachinas force-pushed the perf/avx-simd-investigation branch from 302f001 to 9067fd5 Compare August 2, 2026 20:34
@mrecachinas

mrecachinas commented Aug 2, 2026

Copy link
Copy Markdown
Owner Author

C4 AVX-512 hardware is now available and the initial three-run benchmark archive has been captured on an Intel Xeon Platinum 8581C (Emerald Rapids) with avx2, avx512bw, and avx512_bitalg exposed.

Initial median findings:

  • AVX-512 hex strings are 1.18x-2.40x faster than AVX2 across 16-254 characters.
  • AVX-512 byte distance is 1.53x faster at 64 bytes and 3.00x faster at 127 bytes.
  • Below 64 bytes, the existing AVX fallback chain is slower than the scalar classic implementation in the forced-algorithm matrix.
  • Rust Criterion run-to-run ranges stayed below 2%.

Commit 9067fd5 rebases this PR onto current main, adds native dispatch plus 48/63/65/96-byte threshold cases, and allows targeted Rust-only harness runs. The next step is a targeted baseline on the same C4 VM before changing dispatch.

@mrecachinas
Benchmark native x86 byte dispatch
f607409 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 01071e57-12fd-4f42-bc67-e2b274d0b007
@mrecachinas
Try masked AVX-512 for short byte inputs
6007179 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 01071e57-12fd-4f42-bc67-e2b274d0b007
@mrecachinas

mrecachinas commented Aug 2, 2026

Copy link
Copy Markdown
Owner Author

Masked short-input experiment 6007179 passes the performance gate and is retained. Three-run medians on the same C4 VM:

Bytes Before Masked AVX-512 Improvement
8 4.299 ns 4.057 ns 5.9%
16 5.380 ns 4.048 ns 32.9%
32 6.112 ns 4.068 ns 50.2%
48 6.924 ns 4.071 ns 70.1%
63 11.955 ns 4.068 ns 193.9%
64 3.612 ns 3.576 ns 1.0%

The implementation removes the premature AVX2 fallback and uses the AVX-512 BITALG masked-tail code that was already present for residual bytes. The next experiment is cross-record SIMD for 16- and 32-byte catalogs.

@mrecachinas
Try AVX-512 cross-record array scans
ef9e31b 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 01071e57-12fd-4f42-bc67-e2b274d0b007
@mrecachinas

mrecachinas commented Aug 2, 2026

Copy link
Copy Markdown
Owner Author

Preliminary cross-record scanner results for ef9e31b are strongly positive on the C4 VM. Three-run medians against parent 6007179:

Workload Before After Speedup
512x16 first, random/no-match 1.025 us 311 ns 3.29x
512x16 best, random/no-match 1.224 us 324 ns 3.77x
512x16 all, random/no-match 1.110 us 372 ns 2.98x
1024x16 first 2.030 us 647 ns 3.14x
1024x16 best 2.420 us 628 ns 3.85x
1024x16 all 2.223 us 730 ns 3.05x
1024x32 first 3.606 us 1.315 us 2.74x
1024x32 best 4.140 us 1.478 us 2.80x
1024x32 all 3.886 us 1.427 us 2.72x

All nine cases improved; after-run ranges were generally 1.4%-4.2%, with one 8.3% all case. A full scenario matrix is still required before retention because four-record batching may affect early-match performance.

@mrecachinas
Preserve first-record scanner short circuit
62c8a7e 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 01071e57-12fd-4f42-bc67-e2b274d0b007
@mrecachinas
Fix AVX scanner exact-match test threshold
3494c29 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 01071e57-12fd-4f42-bc67-e2b274d0b007
@mrecachinas

mrecachinas commented Aug 2, 2026

Copy link
Copy Markdown
Owner Author

The hybrid cross-record scanner in 62c8a7e is retained after early-match validation. Compared with parent 6007179:

Scenario Case Before Hybrid Result
random/no-match 16B first 2.030 us 587 ns 3.46x
random/no-match 16B best 2.420 us 594 ns 4.07x
random/no-match 16B all 2.223 us 686 ns 3.24x
random/no-match 32B first 3.606 us 1.281 us 2.82x
random/no-match 32B best 4.140 us 1.432 us 2.89x
random/no-match 32B all 3.886 us 1.401 us 2.77x
exact at index 0 16B first 4.05 ns 4.54 ns 0.49 ns slower
exact at index 0 16B best 5.63 ns 6.26 ns 0.63 ns slower
exact at index 0 32B first 6.43 ns 5.16 ns 1.25x
exact at index 0 32B best 8.36 ns 7.43 ns 1.13x

The small 16B index-0 trade-off is accepted in exchange for the 3.24x-4.07x full-scan gains. all remains 2.55x-3.00x faster even when the exact record is first. Commit 3494c29 also corrects the direct scanner test to use exact-match threshold 0; the previous -1 correctly made every record eligible.

@mrecachinas
Document measured AVX-512 speedups
c8c9c4d 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 01071e57-12fd-4f42-bc67-e2b274d0b007
@mrecachinas mrecachinas changed the title Draft: benchmark and tune remaining AVX SIMD paths Speed up AVX-512 byte and fixed-width array scans Aug 2, 2026
@mrecachinas
mrecachinas marked this pull request as ready for review August 2, 2026 23:09
@mrecachinas
mrecachinas merged commit 18ac42b into main Aug 3, 2026
23 checks passed
@mrecachinas
mrecachinas deleted the perf/avx-simd-investigation branch August 3, 2026 00:56
@mrecachinas mrecachinas mentioned this pull request Aug 3, 2026
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@mrecachinas