Skip to content

Add #[inline] hints to hot path functions#8260

Merged
joseph-isaacs merged 3 commits into
developfrom
claude/vortex-array-inlining-perf-HV9B8
Jun 5, 2026
Merged

Add #[inline] hints to hot path functions#8260
joseph-isaacs merged 3 commits into
developfrom
claude/vortex-array-inlining-perf-HV9B8

Conversation

@joseph-isaacs

@joseph-isaacs joseph-isaacs commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

Summary

This PR adds #[inline] hints to a collection of small, frequently-called functions across the codebase to improve performance. These are primarily simple wrapper methods, trait implementations, and utility functions that benefit from inlining to reduce function call overhead.

These are all candidates for inlining because they are:

  1. Small wrapper functions with minimal logic
  2. Called frequently in hot paths (e.g., binary search, array access)
  3. Generic or trait methods where inlining enables better monomorphization
  4. Simple accessors and type checks

…units = 16`

#8257 set `[profile.bench]` `codegen-units = 16` (cargo's release default) to
speed up benchmark/CI builds, but `[profile.release]` uses `codegen-units = 1`.
With 16 CGUs the compiler no longer inlines small hot functions across
codegen-unit / crate boundaries unless they are explicitly `#[inline]`, so a
number of benchmarks regressed (CodSpeed reported ~16% overall, 147 benches).

This adds `#[inline]` to the small, hot, per-element / per-dispatch functions
that were previously inlined implicitly at `codegen-units = 1`, so the bench
profile once again reflects release performance.

Measured locally (divan `fastest`, cu=16 baseline -> cu=16 + these inlines,
cu=1 shown as the target):

  search_sorted::binary_search_vortex     27.4ns -> 24.1ns  (cu=1 23.4ns)
  listview_rebuild::varbinview_large      24.2us -> 12.0us  (cu=1 11.7us)
  chunk_array::varbinview_canonical_into  50.4us -> 28.7us  (cu=1 24.1us)  [1000,10]
  vortex_bitbuffer::from_iter[65536]      58.1us -> 32.4us  (cu=1 32.4us)
  vortex_bitbuffer::from_iter[16384]      14.7us ->  8.2us  (cu=1  8.2us)

Functions:
- search_sorted: the whole `SearchSorted` / `IndexOrd` chain is co-dependent;
  ablation shows inlining only the entry method is *worse* than none (29.4ns),
  because the per-iteration `index_cmp` / `search_sorted_side_idx` calls then go
  out of line. All links are required.
- builders::varbinview: `adjust_view` / `push_view` run once per element in the
  `VarBinViewBuilder` extend loop (~24% of self-time when out of line).
- array dispatch (`Matcher`/`is`/`as_opt`, `AnyCanonical`): folded per chunk in
  the canonical execute path.
- vortex_buffer `BitBuffer`/`BitBufferMut` `FromIterator`: inlining into the
  caller lets the optimizer specialise the fill loop for the concrete iterator.

`as_any` was evaluated and intentionally left un-annotated: ablation showed its
`#[inline]` had no measurable effect (LLVM already inlines the trivial body).
fastlanes `bitpack_compare_sweep` was also checked but showed no local cu=1 vs
cu=16 difference (the SIMD unpack kernel dominates), so it is left unchanged.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

https://claude.ai/code/session_019w6k1b8tcA9FohTse8PNrh
Continuing the `codegen-units = 16` inlining sweep. A cu=1 vs cu=16 timing
diff across all vortex-array / vortex-buffer / vortex-mask benches surfaced a
further set of regressions; this fixes the one with a clean, verified cause.

`varbinview_zip`: `push_view` is called once per element from `push_range`'s
loops, and `BufferHandle::as_host` is a per-buffer accessor. Both were inlined
implicitly at cu=1 but went out of line at cu=16 (samply self-time: push_view
17.7%, as_host 5.2% at cu=16, both absent from the cu=1 profile).

Measured (divan fastest, cu=16 baseline -> cu=16 + these inlines, cu=1 target):

  varbinview_zip::fragmented_mask   1.358ms -> 1.222ms  (cu=1 1.085ms)

`block_mask` (slice-copy dominated) is unchanged and not regressed. The residual
gap is a per-call `DeduplicatedBuffers` collect, which has no clean inline target.

Investigated but deliberately NOT changed (proven not real inline wins):
- `scalar_at_struct::execute_scalar_struct_simple` (cu=1 125us, cu=16 154us):
  inlining the `execute_scalar` / `OperationsVTable::scalar_at` chain made it
  *worse* (171-180us) -- partial inlining across the dyn-dispatch boundary
  backfires, like inlining a search_sorted entry without its helpers. Reverted.
- `dict_compare` (~1.5x): the hot `bool::take::take_valid_indices` is out of
  line in both cu=1 and cu=16 with its inner `collect_bool` loop already fully
  inlined; the slowdown is CGU codegen variance, not a missing `#[inline]`.
- `filter_bool` (~1.3x): the bench profile is dominated by per-sample RNG/mask
  setup; the mask `BitBuffer::from_iter` is already inlined (prior commit).

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

https://claude.ai/code/session_019w6k1b8tcA9FohTse8PNrh
…udit

Audited every `#[inline]` added in this branch by removing it and re-measuring
(or static call-graph analysis). Three are not justified and are removed:

- `IndexOrd::index_gt` / `index_ge`: zero call sites anywhere in the workspace
  (only `index_lt` (Left) and `index_le` (Right) are ever invoked by
  `search_sorted`). Dead trait defaults -- the annotation can never fire.
- `BitBuffer::from_iter` (the outer forwarder to `BitBufferMut::from_iter`):
  removing it left `from_iter_bit_buffer[65536]` unchanged (27.79us vs 27.82us);
  LLVM already inlines the trivial `.freeze()` delegation. Redundant.

Everything else was confirmed load-bearing by ablation, e.g. dropping
`BufferHandle::as_host`/`as_host_opt` regressed `varbinview_zip` 996us -> 1136us
(+14%), and dropping `is`/`as_`/`as_opt` regressed `chunk_array_builder` ~3%.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

https://claude.ai/code/session_019w6k1b8tcA9FohTse8PNrh
@joseph-isaacs joseph-isaacs added the changelog/performance A performance improvement label Jun 5, 2026 — with Claude
@joseph-isaacs joseph-isaacs enabled auto-merge (squash) June 5, 2026 09:01
@codspeed-hq

codspeed-hq Bot commented Jun 5, 2026

Copy link
Copy Markdown

Merging this PR will improve performance by 19.92%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚡ 47 improved benchmarks
❌ 5 regressed benchmarks
✅ 1455 untouched benchmarks

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation chunked_bool_canonical_into[(1000, 10)] 31.7 µs 46.9 µs -32.36%
Simulation varbinview_zip_block_mask 2.9 ms 3.7 ms -21.49%
Simulation compare[15] 120.6 µs 146.5 µs -17.66%
Simulation compare[14] 118.1 µs 142.1 µs -16.88%
Simulation compare[13] 116.3 µs 138.4 µs -15.99%
Simulation from_iter_bit_buffer[65536] 241.4 µs 113.5 µs ×2.1
Simulation from_iter_bit_buffer[16384] 64.7 µs 32.5 µs +99.25%
Simulation from_iter_bit_buffer[2048] 12.2 µs 7.9 µs +55.53%
Simulation chunked_varbinview_canonical_into[(1000, 10)] 284.5 µs 198.6 µs +43.23%
Simulation chunked_varbinview_into_canonical[(1000, 10)] 300.6 µs 212.7 µs +41.32%
Simulation from_iter_bit_buffer[1024] 8.6 µs 6.2 µs +38.3%
Simulation binary_search_vortex 669.4 ns 486.7 ns +37.56%
Simulation chunked_varbinview_opt_canonical_into[(1000, 10)] 308.3 µs 226.5 µs +36.08%
Simulation chunked_varbinview_opt_into_canonical[(1000, 10)] 323.9 µs 241.1 µs +34.33%
Simulation varbinview_large 175.3 µs 132.7 µs +32.07%
Simulation take_search[(0.005, 1.0)] 3.1 ms 2.3 ms +31.1%
Simulation take_search[(0.005, 0.5)] 1.5 ms 1.2 ms +30.89%
Simulation take_search[(0.005, 0.1)] 321.6 µs 248.6 µs +29.39%
Simulation chunked_varbinview_canonical_into[(100, 100)] 398.8 µs 309.3 µs +28.93%
Simulation take_search[(0.01, 1.0)] 3.3 ms 2.5 ms +28.66%
... ... ... ... ... ...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing claude/vortex-array-inlining-perf-HV9B8 (5d9bb1f) with develop (e484daf)

Open in CodSpeed

@joseph-isaacs joseph-isaacs merged commit bfe88b8 into develop Jun 5, 2026
70 of 72 checks passed
@joseph-isaacs joseph-isaacs deleted the claude/vortex-array-inlining-perf-HV9B8 branch June 5, 2026 09:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/performance A performance improvement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants