Skip to content

Commit 2831822

Browse files
ahravahrav
committed
docs: refresh upstream performance comparison
1 parent 1e0b467 commit 2831822

29 files changed

Lines changed: 3138 additions & 114 deletions

‎README.md‎

Lines changed: 11 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# Aho-Corasick
22

3-
[![Build Status](https://travis-ci.com/BobuSumisu/aho-corasick.svg?token=eGRFn5xdQ7p9yby3GVvc&branch=master)](https://travis-ci.com/BobuSumisu/aho-corasick)
3+
[![CI](https://github.com/ahrav/aho-corasick/actions/workflows/ci.yml/badge.svg?branch=master)](https://github.com/ahrav/aho-corasick/actions/workflows/ci.yml)
44
![Go Version](https://img.shields.io/github/go-mod/go-version/BobuSumisu/aho-corasick)
55
![Latest Tag](https://img.shields.io/github/v/tag/BobuSumisu/aho-corasick)
66

@@ -15,8 +15,8 @@ This implementation does not use a [Double-Array Trie](https://linux.thai.net/~t
1515

1616
This reduces the build time drastically, but at the cost of higher memory consumption.
1717

18-
The search time is still fast, and comparable to other Go implementations I have found on github that claims to be fast
19-
(see [performance](#Performance)).
18+
In the linked historical 512k-pattern benchmark, search time was comparable
19+
to other Go implementations (see [performance](#performance)).
2020

2121
## Documentation
2222

@@ -51,7 +51,7 @@ for _, match := range matches {
5151

5252
// => Matched pattern 0 "or" at position 1.
5353
// => Matched pattern 0 "or" at position 15.
54-
// => Matched patterh 1 "amet" at position 22.
54+
// => Matched pattern 1 "amet" at position 22.
5555
```
5656

5757
## Building
@@ -85,24 +85,11 @@ trie, err := Decode(f)
8585

8686
## Performance
8787

88-
Some simple benchmarking on my machine (Intel(R) Core(TM) i7-8665U CPU @ 1.90GHz, 32 GiB RAM).
89-
90-
Build and search time grows quite linearly with regards to number of patterns and input text length.
91-
92-
### Building
93-
94-
BenchmarkTrieBuild/100-4 7886 154786 ns/op
95-
BenchmarkTrieBuild/1000-4 739 1647419 ns/op
96-
BenchmarkTrieBuild/10000-4 91 13331713 ns/op
97-
BenchmarkTrieBuild/100000-4 9 123886615 ns/op
98-
99-
100-
### Searching
101-
102-
BenchmarkMatchIbsen/100-4 1471089 819 ns/op
103-
BenchmarkMatchIbsen/1000-4 202288 5667 ns/op
104-
BenchmarkMatchIbsen/10000-4 19957 59680 ns/op
105-
BenchmarkMatchIbsen/100000-4 2012 595086 ns/op
88+
Against upstream commit `b4b5728`, this fork at `1e0b467` reduced
89+
single-core time by 30% to 98% across six preselected workloads on AWS
90+
Graviton3. The results use 31 paired process executions per revision. See
91+
[STOCK-COMPARISON.md](STOCK-COMPARISON.md) for the full results, confidence
92+
intervals, raw samples, and reproduction protocol.
10693

10794
### Compared to Other Implementation
10895

@@ -111,6 +98,5 @@ See
11198

11299
### Memory Usage
113100

114-
As mentioned, the memory consumption will be quite high compared to a double-array trie
115-
implementation. Especially during the build phase (which currently contains a lot of object
116-
allocations).
101+
Memory consumption is higher than a double-array trie implementation,
102+
especially during the build phase.

‎STOCK-COMPARISON.md‎

Lines changed: 112 additions & 89 deletions
Original file line numberDiff line numberDiff line change
@@ -1,92 +1,115 @@
1-
# Stock vs optimized: full before/after comparison
2-
3-
**Before:** stock upstream `BobuSumisu/aho-corasick` @ b4b5728 (v1.0.3, no
4-
fork optimizations of any kind).
5-
**After:** this fork's full chain tip (`perf/26-builder-index-fusion`,
6-
master + PRs #3–#10 + the 19-branch stack #11–#29).
7-
8-
**Machine:** AMD EPYC 9R14 (Zen 4), 48 cores, Go 1.25.11, linux/amd64.
9-
**Stability:** load average 0.9–1.0 at start (idle but for the session
10-
agent), no other benchmark processes, no cpufreq scaling exposed (fixed-
11-
frequency VM). Interleaved A/B binaries; benchstat n=8; hyperfine 10+ runs
12-
with warmup. Dispersion: benchstat CIs mostly ≤3%, hyperfine σ ≤3.5% —
13-
consistent with a quiet machine.
14-
15-
**Fairness:** identical public-API-only benchmark source compiled against
16-
both trees (`bench_public_test.go`, pub-prefixed; no internals touched).
17-
Cross-version semantics verified: both binaries report identical match
18-
counts on every workload (e.g. 6,178 on Ibsen/10k-dict; 110,249 on the
19-
400MB spread scan).
20-
21-
## benchstat (Go microbenchmarks, n=8 interleaved)
22-
23-
**geomean: −90.2% (10.2x)**
24-
25-
| Benchmark | stock | tip | Δ |
26-
|---|---|---|---|
27-
| Text sorted-dict 1KB | 2.87µs | 435ns | **−84.9%** |
28-
| Text sorted-dict 4KB | 11.7µs | 1.78µs | **−84.8%** |
29-
| Text sorted-dict 100KB | 322µs | 45.2µs | **−86.0%** |
30-
| Text spread-dict 4KB | 12.8µs | 6.27µs | **−51.1%** |
31-
| Text spread-dict 100KB | 348µs | 88.4µs | **−74.6%** |
32-
| Text 100k-pattern dict 100KB | 728µs | 160µs | **−78.0%** |
33-
| Large input 512KB | 1.65ms | 189µs | **−88.5%** |
34-
| Large input 2MB | 6.62ms | 429µs | **−93.5%** |
35-
| Large input 8MB | 28.3ms | 1.34ms | **−95.3% (21x)** |
36-
| No-match 100KB (sorted) | 198µs | 1.20µs | **−99.4% (165x)** |
37-
| No-match 1MB (sorted) | 2.02ms | 30.8µs | **−98.5%** |
38-
| No-match 100KB (spread) | 197µs | 25.4µs | **−87.1%** |
39-
| No-match 1MB (spread) | 2.02ms | 79.0µs | **−96.1%** |
40-
| Dense overlaps 64KB | 6.49ms | 1.15ms | **−82.3%** |
41-
| Concatenated words 64KB | 1.18ms | 181µs | **−84.7%** |
42-
| Walk sorted-dict 100KB | 233µs | 38.1µs | **−83.7%** |
43-
| Walk spread-dict 100KB | 266µs | 141µs | **−46.8%** |
44-
| MatchFirst (late needle) | 201µs | 28.4µs | **−85.9%** |
45-
| Build 1k patterns | 15.9ms | 970µs | **−93.9%** |
46-
| Build 10k patterns | 163ms | 8.2ms | **−95.0%** |
47-
| Build 100k patterns | 2.27s | 92.6ms | **−95.9% (24.5x)** |
48-
49-
Allocations per Match call: **−94…−100%** on every match workload (the
50-
pool + arena machinery; e.g. Dense 64KB: 2.24MB/op → 13KB/op, sorted-text
51-
scans: zero allocations at steady state). Build allocations −26…−44%.
52-
The only rows with allocation increases are no-match inputs above the
53-
parallel threshold (+3.7KB/op of worker scratch on 1MB inputs — noise
54-
against the 25x speed win there).
55-
56-
## hyperfine (whole-process wall clock, warmup + 10 runs)
57-
58-
| Scenario | stock | tip | Speedup |
59-
|---|---|---|---|
60-
| Build 100k-pattern trie | 2.477s ± 34ms | 312.7ms ± 8.7ms | **7.9x** |
61-
| Scan 400MB prose, sorted dict | 1.554s ± 5ms | 306.0ms ± 10.6ms | **5.1x** |
62-
| Scan 400MB prose, spread dict | 2.26s | 513.8ms ± 2.4ms | **4.4x** |
63-
| Cold end-to-end (build 10k + scan Ibsen once) | 276ms | 113.8ms ± 5.7ms | **2.4x** |
64-
65-
(Whole-process numbers include Go runtime startup and pattern-file
66-
loading, which is why the cold e2e ratio is smaller than the library-only
67-
ratios.)
68-
69-
## Where the wins come from (chain attribution)
70-
71-
- **No-match / long-gap 165x:** SWAR + vectorized IndexByte root skipping,
72-
windowed multi-stop escape, density-aware parallel dispatch.
73-
- **Build 24.5x:** row-copy DP construction, index-based value-struct
74-
states with inline flags/failTrans16 fusion (vs stock's per-(state,byte)
75-
fail-chain walks through per-node maps).
76-
- **Large inputs 21x:** parallel scan with per-worker segment materialize
77-
and size/liveliness-scaled worker caps (stock is single-threaded).
78-
- **Text scans 4–7x:** devirtualized specialized loops, 16-bit half-width
79-
tables, stop-entry constant, dual-cursor scans, branchless skip locate.
80-
- **Dense/overlap-heavy 5.6x:** dual-cursor + byte-class-compressed table
81-
+ pooled zero-allocation buffers.
82-
- **Allocation elimination:** pooled match buffers and arena
83-
materialization (stock allocates every Match struct and slice per call).
1+
# Upstream performance comparison
2+
3+
This compares upstream `BobuSumisu/aho-corasick` at `b4b5728` with this fork
4+
at `1e0b467`. The upstream commit is 12 commits after v1.0.3 (`58861e9`).
5+
Both revisions used the same public-API benchmark source and byte-identical
6+
corpora.
7+
8+
## Results
9+
10+
All primary results use one pinned AWS Graviton3 core. Times are medians
11+
across 31 separate process executions per revision. Reductions are geometric
12+
means of paired time ratios. The table reports Bonferroni-adjusted 99.1667%
13+
percentile-bootstrap intervals, giving nominal 95% simultaneous coverage
14+
across the six endpoints.
15+
16+
| Workload | Upstream median | Fork median | Time reduction (99.1667% CI) |
17+
|---|---:|---:|---:|
18+
| Natural text, spread 10k dictionary, 100 KiB | 464.921 us | 325.286 us | 30.02% (29.95% to 30.09%) |
19+
| No match, spread 10k dictionary, 1 MiB | 2.983 ms | 336.482 us | 88.72% (88.71% to 88.73%) |
20+
| Dense overlapping matches, 64 KiB | 10.626 ms | 897.377 us | 91.51% (91.45% to 91.57%) |
21+
| `MatchFirst`, late match in 100 KiB | 282.567 us | 5.177 us | 98.17% (98.16% to 98.17%) |
22+
| Build 10k-pattern trie | 111.702 ms | 13.124 ms | 88.26% (88.11% to 88.41%) |
23+
| Natural text, sorted 10k dictionary, 8 MiB | 37.780 ms | 8.724 ms | 76.87% (76.81% to 76.92%) |
24+
25+
The four `Match` scan endpoints reported zero allocations per operation in
26+
the fork and one to four upstream. `MatchFirst` reported one 48-byte
27+
allocation in the fork in a separate `-benchmem` check. The 10k-pattern
28+
build reported 32 allocations in the fork and about 54,261 upstream.
29+
30+
These numbers describe these workloads on this machine. They are not a suite
31+
geomean or an end-to-end application claim.
32+
33+
## Experiment design
34+
35+
- Machine: AWS `m7g.8xlarge` (Graviton3), Linux arm64.
36+
- Toolchain: Go 1.25.11.
37+
- Experimental unit: one benchmark process execution.
38+
- Pairing: upstream and fork ran in adjacent blocks, alternating order
39+
`upstream/fork` then `fork/upstream`.
40+
- CPU control: `taskset -c 19`, `GOMAXPROCS=1`, and `-test.cpu=1`.
41+
- Final timing: 1 second for scan benchmarks; 3 seconds for build and the
42+
8 MiB scan. The excluded pilot used 500 milliseconds for scans.
43+
- Warmup: Go's benchmark calibration ran before each reported measurement.
44+
- Stopping: fixed at 31 executions per revision before final collection.
45+
- Exclusions: none.
46+
47+
A separate 10-pair pilot estimated per-execution CV. Sizing assumed a more
48+
conservative 5% CV, a 5% minimum detectable effect, 90% power, and
49+
`alpha=0.05/6` for six primary endpoints:
50+
51+
```text
52+
n = ceil(2 * (z_(1-alpha/2) + z_(1-beta))^2 * (CV/MDE)^2) = 31
53+
```
54+
55+
Final primary-endpoint sample CV was at most 2.74%. For analysis, each pair
56+
produced `log(fork_time / upstream_time)`. The point estimate is the
57+
exponentiated mean log ratio. Intervals use 500,000 paired percentile
58+
bootstrap resamples from NumPy's `Generator(PCG64)`, seed `20260717`, and
59+
linear quantiles. The recorded environment used Python 3.9.25, NumPy 2.0.2,
60+
and SciPy 1.13.1.
61+
62+
At `alpha=0.05`, a Welch test comparing log ratios by first arm did not
63+
detect a statistically significant order effect (`p=0.185` to `0.985`).
64+
Spearman tests did not detect a statistically significant monotonic trend
65+
across pair number (`p=0.308` to `0.952`).
66+
67+
## Integrity
68+
69+
Both revisions passed `go test -count=1 ./...`. The benchmark source, corpus,
70+
build flags, and toolchain were identical. Both binaries used
71+
`-trimpath -buildvcs=false`. The
72+
[raw pilot and primary samples](benchmarks/upstream-20260717/) are included
73+
with the report, including the environment, every command timestamp, and
74+
fresh test output from both revisions.
75+
76+
| Input | SHA-256 |
77+
|---|---|
78+
| `bench_public_test.go` | `e936d64744524c24b1e9bfaecebadb1ba416d4491b8ea4638b2fe59790ba42f3` |
79+
| `test_data/NSF-ordlisten.cleaned.txt` | `2d9ad4e5838dc03b438d1881ba52dbb8b6702d9aaf78a979e6a412068e712ae5` |
80+
| `test_data/Ibsen.txt` | `d5fb85f811c2954ff7bb47d90b72e9140585c99fd2b4e333181de1e2c5a48200` |
8481

8582
## Reproduction
8683

87-
- `bench_public_test.go` (this directory) compiles unmodified against
88-
both trees; run with `-bench Pub -count 8` per tree and compare with
89-
benchstat.
90-
- `cmd/acbench` builds against both trees for the hyperfine scenarios:
91-
`hyperfine -N --warmup 2 './acbench-stock build 100000 1' './acbench-tip build 100000 1'`
92-
etc. `ACBENCH_DATA` points at `test_data/`.
84+
Prepare checkouts at the two revisions above. Add the fork's
85+
`bench_public_test.go` unchanged to the upstream checkout, then run:
86+
87+
```bash
88+
tools/run_upstream_benchmark.sh \
89+
/path/to/fork-at-1e0b467 \
90+
/path/to/upstream-at-b4b5728 \
91+
/tmp/aho-upstream-comparison
92+
```
93+
94+
The runner verifies revisions and input hashes, records the environment and
95+
commands, executes the excluded 10-pair pilot, then collects 31 final pairs
96+
without invoking the analyzer. Analyze the completed output with:
97+
98+
```bash
99+
python3 tools/analyze_upstream_benchmark.py \
100+
/tmp/aho-upstream-comparison
101+
```
102+
103+
The committed analyzer defines endpoint ordering, sample CV, paired
104+
percentile bootstrap, PRNG, quantile method, and diagnostic tests.
105+
Its Python dependencies are pinned in `tools/benchmark-requirements.txt`.
106+
107+
`benchstat` provides an independent nonparametric summary:
108+
109+
```bash
110+
benchstat \
111+
-alpha 0.008333333333333333 \
112+
-confidence 0.9916666666666667 \
113+
/tmp/aho-upstream-comparison/final-scan-upstream.txt \
114+
/tmp/aho-upstream-comparison/final-scan-fork.txt
115+
```

‎benchmarks/upstream-20260717/README.md‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
# Benchmark evidence
2+
3+
These files support `STOCK-COMPARISON.md`.
4+
5+
- `pilot-*`: 10 exploratory executions per revision, excluded from inference.
6+
- `final-*`: 31 primary executions per revision.
7+
- `pilot-order.tsv` and `pilot-large1-order.tsv`: first arm and timestamp for
8+
each excluded pilot block.
9+
- `final-order.tsv` and `final-large1-order.tsv`: first arm and timestamp for
10+
each paired block.
11+
- `analysis.txt`: output from the committed analyzer.
12+
- `commands.log`: timestamp and parameters for every test, build, and
13+
benchmark process.
14+
- `environment.txt` and `environment-end.txt`: host, toolchain, affinity,
15+
hashes, and start/end load.
16+
- `matchfirst-benchmem-fork.txt`: separate allocation check for `MatchFirst`.
17+
- `test-fork.txt` and `test-upstream.txt`: fresh `go test -count=1 ./...`
18+
output from the measured revisions.
19+
20+
Recompute the report with:
21+
22+
```bash
23+
python3 tools/analyze_upstream_benchmark.py \
24+
benchmarks/upstream-20260717
25+
```

‎benchmarks/upstream-20260717/analysis.txt‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
python=3.9.25 numpy=2.0.2 scipy=1.13.1
2+
planned_samples=31 resamples=500000 seed=20260717
3+
4+
| Workload | Upstream | Fork | Reduction (99.1667% CI) |
5+
|---|---:|---:|---:|
6+
| Natural text, spread 10k, 100 KiB | 464.921 us | 325.286 us | 30.023% (29.945% to 30.093%) |
7+
| No match, spread 10k, 1 MiB | 2.983 ms | 336.482 us | 88.716% (88.705% to 88.728%) |
8+
| Dense overlaps, 64 KiB | 10.626 ms | 897.377 us | 91.511% (91.452% to 91.568%) |
9+
| MatchFirst, late match, 100 KiB | 282.567 us | 5.177 us | 98.167% (98.161% to 98.173%) |
10+
| Build 10k-pattern trie | 111.702 ms | 13.124 ms | 88.262% (88.107% to 88.408%) |
11+
| Natural text, sorted 10k, 8 MiB | 37.780 ms | 8.724 ms | 76.871% (76.811% to 76.918%) |
12+
13+
| Workload | Upstream CV | Fork CV | Order p | Trend p |
14+
|---|---:|---:|---:|---:|
15+
| Natural text, spread 10k, 100 KiB | 0.127% | 0.157% | 0.185 | 0.474 |
16+
| No match, spread 10k, 1 MiB | 0.218% | 0.135% | 0.985 | 0.952 |
17+
| Dense overlaps, 64 KiB | 1.432% | 0.240% | 0.818 | 0.855 |
18+
| MatchFirst, late match, 100 KiB | 0.683% | 0.077% | 0.401 | 0.696 |
19+
| Build 10k-pattern trie | 0.354% | 2.733% | 0.363 | 0.551 |
20+
| Natural text, sorted 10k, 8 MiB | 0.457% | 0.100% | 0.906 | 0.308 |

0 commit comments

Comments
 (0)