[Feature Request] Add a substring / grep (`LIKE '%...%'`) benchmark case, backed by the Pizza&Chili corpus

[xiaofan-luan]

Summary

I'd like to propose a substring / "grep" search benchmark case — text LIKE '%pattern%' over a text column — plus adding the Pizza&Chili corpus as a new data source to back it.

Today VDBBench has no lexical coverage at all: every case ultimately drives search_embedding, and there is no substring, wildcard, or full-text case type. #687 asks for match_phrase + kNN hybrid, which is adjacent but a different problem — phrase matching runs on a tokenized inverted index, while arbitrary substring matching cannot use one (the pattern may start or end mid-token). Both are worth having; this issue is about the substring half.

Motivation

Substring filtering over text is a first-class production workload, not a niche one:

• Log / observability search — find every line containing a request ID, an error fragment, a stack frame.
• Code search — identifier and snippet lookup.
• RAG metadata filtering — narrow a corpus by a substring on a text attribute before or during vector retrieval.

Multiple systems in VDBBench's comparison set already ship dedicated index structures for exactly this, so the case is meaningfully cross-DB rather than vendor-specific:

System	Mechanism
Milvus	NGRAM index; FM-index (suffix-array based) landing now
Elasticsearch / OpenSearch	wildcard field type
PostgreSQL / pgvector	pg_trgm GIN index
ClickHouse	ngrambf_v1 / tokenbf_v1 skip indexes

There is currently no shared, reproducible benchmark for any of it. The practical consequence is that every vendor publishes numbers on a self-chosen corpus with self-chosen patterns, and none of them are falsifiable or comparable.

Why this cannot be a single number

This is the part that shapes the whole design. Substring-index performance is not one value — it moves by orders of magnitude along axes that a naive benchmark holds fixed:

1. Pattern length. Suffix-array / FM-index backward search costs scale with pattern length. n-gram indexes go the other way: patterns shorter than n degenerate to a full scan. The crossover between the two families lives entirely in this axis, so a benchmark with one fixed pattern length cannot show it.
2. Selectivity (occurrence count). Counting matches is cheap; locating them costs time proportional to the number of hits. High-frequency patterns are the worst case for a compressed index and the best case for a bitmap. A single hand-picked pattern hides this cliff completely.
3. Build time and index size. This is the adoption gate — an index that is 3× the raw text is a different product than one that is 0.3×. It belongs in the report, not a footnote.
4. Corpus repetitiveness. Determines how much a compressed index can exploit; separates run-length-compressed indexes (r-index) from plain FM-index by a large factor.

So the deliverable is a matrix — selectivity × pattern length — not a headline number.

Proposed data source: Pizza&Chili corpus

pizzachili.dcc.uchile.cl has been the standard corpus for compressed full-text indexes since ~2005 (Ferragina @ Pisa + Navarro @ Chile — hence the name). Using it means results are directly comparable to twenty years of published numbers, and reviewers in this area recognize the setup immediately.

Concretely I'd propose these, matching the axes above. δ is substring complexity, the standard repetitiveness measure — lower = more repetitive:

File	Size	Alphabet	δ	Why include it
english (200MB prefix)	200 MB of 2.1 GB	239	—	Natural-language baseline
sources (200MB prefix)	200 MB of 201 MiB	230	—	Code search; C/Java from Linux 2.6.11.6 + GCC 4.0.0
einstein.en.txt	446 MiB	139	42,884	Extreme repetition — all revisions of one Wikipedia article. 7z's to 320 KB. Separates run-length-compressed indexes from plain ones
HDFS_1	100 MiB	73	1,173,611	Hadoop DFS logs
BGL	100 MiB	78	1,301,165	Blue Gene/L supercomputer logs
Thunderbird	100 MiB	92	1,048,593	Thunderbird supercomputer logs
Android	100 MiB	176	881,571	Android system logs
EDGAR	100 MiB	71	1,116,985	SEC EDGAR financial system logs
Windows	100 MiB	91	61,711	Windows system logs — notably more regular than the other five, useful as a contrast

The six log files are the highest-value group: real system logs are the actual target workload, and they're already normalized to 100 MiB each, which conveniently sits at about one segment/shard worth of text for most engines — per-segment index build and size can be measured without any truncation.

Pizza&Chili also ships three execution traces (Horspool 546 MiB, NQueens 289 MiB, Quicksort 364 MiB) with δ in the tens of thousands. They produce very flattering numbers precisely because they're so repetitive, so I'd either leave them out or label them clearly as a corner case rather than a representative workload.

Query set design

This is where most of the real work is, and where a shared benchmark adds the most value over everyone rolling their own:

• Selectivity buckets: ~1e-6, 1e-5, 1e-4, 1e-3, 1e-2, plus a high-frequency bucket.
• Pattern length buckets: 2, 4, 8, 16, 32 characters.
• N patterns per cell, sampled deterministically (fixed seed) from the corpus itself so every pattern is guaranteed to have the intended hit count.
• Ship the generator script plus the resulting pattern file — the patterns are kilobytes, so no large artifact needs hosting for this part.

Report each cell separately. Averaging across selectivity is what makes published substring-search numbers uninformative.

Ground truth is exact — and that changes the design

Unlike ANN, substring-match ground truth is exact and cheap to compute: it's just the set of matching row IDs from a linear scan. Two useful consequences:

1. Correctness becomes a pass/fail assertion, not a tuning dial. Recall must be exactly 1.0. Anything else is a bug in the index, not a speed/accuracy trade-off. That's a stronger guarantee than any ANN case can offer and it should be checked, since a wrong-but-fast substring index is easy to ship accidentally.
2. No large GT artifact is needed. A linear scan regenerates it locally in seconds, so GT can be produced at prepare-time rather than downloaded — sidestepping the artifact-size problem that dominates [Feature Request] Add a Large-TopK benchmark case: LAION-100M with top-1M ground truth #826.

Metrics to report

• Index build time and index size (absolute, and as a ratio to raw text)
• p50 / p95 / p99 latency per (selectivity, length) cell
• QPS under concurrency
• Resident memory
• Correctness assertion (exact match against GT)

Two query shapes

1. Scalar-only grep — SELECT id WHERE text LIKE '%pat%' LIMIT n, no vector involved. This isolates the index and is the number that's comparable to the full-text-index literature.
2. Filtered ANN — vector search carrying a substring predicate. This is the shape that actually shows up in RAG, and it stresses a different path (pre-filter vs. post-filter, bitmap handoff to the vector index) that shape 1 never touches.

Shape 2 fits the existing runner almost as-is. Shape 1 does not, which brings us to:

What's missing in VDBBench today

Mapping the gap concretely, since this touches more than one extension point:

• FilterOp has no substring op — backend/filter.py:6-9 has only NumGE, StrEqual, NonFilter. Needs a StrContains plus a Filter subclass, and each client extends supported_filter_types / prepare_filter. The rejection path for non-supporting clients already exists (FilterNotSupportedError in assembler.py), so unsupported DBs degrade cleanly.
• There is no non-vector query path — VectorDB.search_embedding is the sole query entry point, and SerialSearchRunner unconditionally passes an embedding. Shape 1 needs a new abstract method (something like scalar_query(expr, limit)) and a runner path that doesn't require a query vector.
• Metrics are selected by CaseLabel, not by the case class — task_runner.py:241-258 dispatches on it. A new metric shape means a new CaseLabel, a new _run_*_case branch, and new Metric fields.
• Datasets assume a vector column — BaseDataset is built around emb + neighbors_id. A text dataset needs an id + text variant, registered in both the Dataset enum and DatasetWithSizeType/DatasetWithSizeMap.
• Hosting: converting the selected files to parquet and mirroring them under the existing assets.zilliz.com/benchmark/ root would need no new download machinery — AwsS3Reader / AliyunOSSReader already handle it.
• Worth noting there's a documented recipe for adding a client (README "Adding New Clients") but none for adding a dataset or a case. If this lands, that gap is probably worth closing at the same time.

Open questions for maintainers

1. Is a non-vector (scalar-only) case type something the project wants to host at all, or should this be scoped to filtered-ANN only to avoid introducing a second query path?
2. Licensing — Pizza&Chili is derived from Gutenberg, Linux/GCC sources, Wikipedia, and public log datasets. I'd want to confirm redistribution terms before mirroring to assets.zilliz.com; alternatively the case could download from the upstream mirror directly. Preference?
3. Full matrix, or a trimmed default? The full selectivity × length grid across 9 corpora is a long run — a --quick subset (say the six log files at three selectivity levels) is probably what most people should run by default.

Happy to contribute the corpus conversion, the pattern generator, and the Milvus-side client implementation if there's interest in the direction.

[xiaofan-luan]

Design update: settled corpus plan + query specification

Narrowing the proposal above. The issue body listed nine corpora (three Pizza&Chili texts + six log files); after digging into what's actually available at scale, that set is both too wide and missing the one that matters most. Concrete plan below, with every figure verified against the source rather than quoted from memory.

Corpus plan — four runs, one primary

Run	Corpus	Text volume	Download	Purpose
Primary	CLP hadoop-14TB-part1	100 GB slice	~5 GB	Scale inflection — does the index still fit?
Fast regression	same	1 GB slice	(same tars, truncated)	Full query matrix, dev loop
Calibration	einstein.en.txt	446 MiB	320 KB	Does the index exploit long-range repetition at all?
Control	Common Crawl WET	1 GB slice	~420 MB	Non-repetitive pessimistic bound

Only the first is a serious investment. The other three together are under 1 GB of download and run in minutes, so they're effectively free to keep in every run.

Sources and licenses

• CLP hadoop-14TB-part1 — Zenodo 7114847, CC-BY-4.0. YScope's dataset index states 428.94 GB uncompressed / 20.33 GB compressed, Text format. Five tar files, ~4.3 GB compressed each (~86 GB uncompressed each). Content is Hadoop daemon logs from three clusters of 48 data nodes running the HiBench suite over one month — real production shape, single grammar.
• einstein.en.txt — Pizza&Chili repetitive corpus, 446 MiB, alphabet 139, δ = 42,884. The .7z is 320,696 bytes, i.e. it compresses to 0.07% of its size.
• Common Crawl WET, CC-MAIN-2026-25 — free on AWS Open Data, not requester-pays. 100,000 independent files; I measured one at 62,943,693 B gz → 171,101,086 B raw (2.718×), so ~162 MiB of text per file. File list is crawl-data/CC-MAIN-2026-25/wet.paths.gz (200,835 B, 100,000 lines). A 1 GB slice is ~7 files; 100 GB would be ~588.

Why real logs as the primary rather than web text

It is the workload. Substring search is a log-search, observability and code-search feature. Nobody greps a web crawl. Common Crawl is a good control but the wrong thing to make the headline corpus.

Download efficiency differs by 7×. The Hadoop logs compress ~21×, so 100 GB of text costs a ~5 GB download (1.2 tar files, streamable and truncatable). The same 100 GB from Common Crawl WET needs ~588 files ≈ 35 GB.

Repetitiveness sits where real workloads sit. einstein.en.txt and Common Crawl are the two endpoints of the repetitiveness axis. Whether a compressed index compresses down to something that fits is only meaningfully answered in the middle, and logs are the middle.

The two tiers deliberately share a corpus. If the 100 GB run used one corpus and the 1 GB run another, a difference between them couldn't be attributed to scale rather than content — and scale is the only reason the 100 GB tier exists. Its single question is whether the index still fits: a plain FM-index typically lands near the size of the raw text, so 100 GB of text implies ~100 GB of index, which pushes the workload from resident memory to mmap/disk. That is a different performance regime, and the 1 GB tier cannot see it.

A note on the six log files in the issue body: they're third-hand. Line counts confirm it — LogHub's Thunderbird (211,212,192 lines) and BGL (4,747,963 lines) are byte-identical repackagings of USENIX CFDR HPC4's tbird2 and bgl2, which Pizza&Chili then truncated to 100 MiB each. Nothing wrong with them, but at 100 MiB they can't answer the scale question, and CFDR's whole four-file set is 104,404,005,900 B (97.23 GiB) — still short of the primary tier.

Query specification

The important structural point: "a query" is three orthogonal choices, and collapsing them produces numbers that prove nothing.

Layer 1 — operation (what you ask for)

This layer discriminates most, because these hit different parts of an index.

	Query	What it isolates
Q1	EXISTS — ... where text like "%p%" limit 1	Backward search only, no suffix-array sampling touched. Must be measured, because it is the one case brute-force scan can win — an early-exit scan on a frequent pattern may return in milliseconds. Omitting it overstates index benefit.
Q2	COUNT — count(*) where text like "%p%"	An FM-index gets the occurrence count directly from backward search at O(|P|), independent of hit count; n-gram and scan must walk every hit. Largest expected gap between families.
Q3	LOCATE full — select id where text like "%p%"	Cost ∝ occurrences × SA sample rate. This is the cliff. Q2 being cheap does not imply Q3 is; separating them is what quantifies the sampling-rate space/time knob.
Q4	LOCATE + LIMIT 10/100/1000	Early termination. If Q4 latency equals Q3, the engine is computing the full set then truncating — a common defect, and most real queries carry a LIMIT.
Q5	Filtered ANN — vector search + substring predicate	The RAG shape. Exercises predicate→bitmap handoff and pre- vs post-filter planning. Q1–Q4 never touch this path.
Q6	Snippet / context extract	Tests the self-index property. If the implementation falls back to re-reading the original column, the raw text must be counted in the storage cost. This pins down the definition of "index size".

Layer 2 — pattern shape (what you match)

Core four, all required:

• Infix %p% — the main event; a token-based inverted index cannot serve it, since the pattern may straddle token boundaries.
• Prefix p% — servable by an ordinary sorted index. Acts as a planner check: if prefix queries are slower than infix, the engine is blindly routing to the substring index.
• Multi-substring AND — like "%a%" and like "%b%". Two 1%-selectivity patterns may intersect to 1e-4. Whether the engine materializes both full sets before intersecting is a common performance trap.
• NOT LIKE — inverts selectivity: a 1e-6 pattern negated matches 99.9999%. Tests whether the planner degenerates to a full scan and whether the cost model handles negation.

Secondary: OR/alternation, case-insensitive (which inflates or defeats n-gram indexes), internal single-char wildcard %a_b%.

Separate and non-negotiable: multi-byte / CJK, as a correctness assertion rather than a performance measurement. The classic substring-index bug is treating byte boundaries as character boundaries, which on UTF-8 yields invalid sequences or missed matches. Without this in the suite, a fast-but-wrong implementation passes. Common Crawl WET carries plenty of non-English text for it.

Layer 3 — parameter grid

Selectivity buckets ~1e-6 / 1e-5 / 1e-4 / 1e-3 / 1e-2 / high-frequency, crossed with pattern lengths 2 / 4 / 8 / 16 / 32. Report per cell; never average across cells.

The length axis matters specifically because n-gram indexes degenerate to a full scan for patterns shorter than n, so lengths 2 and 4 are where the family boundary actually shows up.

Expected discrimination

If the measured shape disagrees with this, the implementation is suspect rather than the theory:

Query	FM-index	NGRAM	Inverted (token)	Brute scan
Q2, long pattern	O(|P|), independent of hits	verify all candidates	cannot serve	O(N)
Q2, pattern shorter than n	unchanged	degenerates to scan	cannot serve	O(N)
Q3, low occurrence	fast	fast	—	O(N)
Q3, high occurrence	cliff, ∝ occurrences	cliff, different shape	—	O(N), flat
Q1 exists	instant	fast	—	early exit — may tie

Beyond query latency

Latency alone doesn't decide adoption. Also report: build time; index size, absolute and as a ratio to raw text; incremental ingest — in engines where newly written data is unindexed until sealed, indexed and unindexed latency must be reported separately or the number just reflects when you started measuring; concurrent QPS; resident memory.

Default subset

The full grid across four runs is long. Suggested default: the 1 GB Hadoop slice plus the two small corpora, operations Q1–Q4, infix patterns, three selectivity levels (1e-5 / 1e-3 / high-frequency) × three lengths (2 / 8 / 32). That already exposes both headline effects — n-gram degradation on short patterns, and the FM-index locate cliff at high occurrence. Everything else goes behind a full flag.

Pattern generation

No corpus at this scale ships a query set — Common Crawl publishes WARC/WAT/WET plus a URL-keyed index (you can ask which crawl captured a URL, not which pages contain a string), and the LM-training corpora were never built for retrieval evaluation. So patterns must be generated. Two passes:

1. Sample candidate substrings at random offsets for each target length.
2. Run all candidates through a single Aho-Corasick pass over the corpus to get exact occurrence counts. One scan, all patterns — minutes at 100 GB.
3. Bucket by measured count into the selectivity bins, keep N per (length, selectivity) cell.

The count table produced in step 2 is the ground truth for Q2, so the query set and its ground truth fall out of the same scan.

For the log corpora, don't use uniform random substrings. People grep logs for specific token classes — request IDs, UUIDs, IPs, file paths, error codes — which are extractable by regex from the corpus itself, and naturally span the length axis. Uniform random sampling mostly yields cross-boundary fragments nobody would ever search for.

Two methodology traps worth stating explicitly

Pizza&Chili's own genpatterns does not control selectivity. It samples by length only, uniformly at random, and most random substrings of natural text occur once or twice. So the twenty-year standard methodology systematically under-samples the high-occurrence region — precisely where locate() cliffs and compressed indexes fail. This is why the selectivity axis above is not a restatement of existing practice.

Never scale a corpus by duplication. The common tooling for reaching large sizes (OpenSearch's expand-data-corpus.py, and Elastic's logs track, which concedes data "will inevitably be reused") works by copying documents and adjusting timestamps. For a compressed text index this is fatal: duplicating N times drives the repetitiveness measures (BWT runs r, LZ phrases z, δ) down roughly N-fold, so a compressed index's space improves nearly linearly in the duplication factor while a plain scanner is unaffected. The published number would describe the data generator, not the index. Pizza&Chili's own figures show the gap: real 100 MiB log slices p7zip to 0.54%–8.49%, while synthetic execution traces reach 0.38%–0.47%. Scale must come from concatenating genuinely distinct data — which is exactly why the primary corpus is a 428.94 GB real one that can be sliced rather than a small one that gets multiplied.

Open items

1. Licensing is now clean for all three sources (CC-BY-4.0, Pizza&Chili, Common Crawl ToU), so mirroring the converted parquet under the existing assets.zilliz.com/benchmark/ root looks viable — but I'd like a maintainer call on whether to mirror or have the case download from upstream.
2. Still open from the issue body: whether a scalar-only (non-vector) case type is acceptable, or whether this should be scoped to filtered-ANN only to avoid introducing a second query path into VectorDB.

[xiaofan-luan]

Query specification v2 — supersedes the "Query specification" section of the previous comment

The corpus plan in that comment stands unchanged. Only the query spec is revised, after surveying what the mature text-search, regex and string-matching benchmarks actually test. Two findings drove the rewrite: the gap this issue targets is wider than I assumed, and a selectivity × pattern-length matrix — which is what I proposed — turns out to hold roughly nine other variables constant, most of which are the primary axis of some authoritative suite.

The existing suites barely test substring search

Worth stating, because it establishes there is nothing to adopt wholesale:

• luceneutil (the Lucene nightly suite) constructs every wildcard query as term[0:2] + "*" + term[-1], so every wildcard carries a two-character literal prefix and is always FST-prefix-accelerated. grep -c "^Wildcard: \*" over its task files returns 0 — leading wildcards have never been measured. There is no RegexpQuery task anywhere in TaskParser.java, CreateQueries.java, or any .tasks file. And while Term/Phrase/And/Or are rigorously frequency-bucketed, wildcard selectivity is uncontrolled — patterns are drawn in docFreq order and neither the expanded-term count nor the matched-doc count is recorded.
• elastic/rally-tracks contains no wildcard query and no regexp query at all; prefix appears 9 times in a single track.
• opensearch-benchmark-workloads: wildcard appears in exactly one file (clickbench/operations/dsl.json), and those four queries all search the single literal google.
• ClickBench: 5 of 43 queries touch text matching. Every one is %…%; there is not a single prefix-only 'foo%' LIKE in the entire benchmark; only the literals google / Google / .google. are ever searched, fixing pattern length at 6–8 characters. No selectivity sweep, no length sweep, no ILIKE.
• Pizza&Chili and SMART cannot express a zero-match query. Both generate patterns by copying a random substring out of the text (genpatterns.c picks a random offset in [0, n−m]), so by construction every pattern matches at least once.

TPC-H is the one standard benchmark that covers pattern shape well — Q2 suffix '%:2', Q9 infix, Q13 negated multi-segment not like '%:1%:2%', Q14 prefix, Q16 negated prefix plus '%Customer%Complaints%', Q20 prefix — and that taxonomy is reflected below.

Layer 1 — operation / return shape

The single most useful idea borrowed here is rebar's: vary what the engine is asked to return, not just what it matches. rebar's MODELS.md puts it bluntly — "Most regex benchmarks either define only one model, or even worse, combine multiple models together."

	Operation	Isolates	Priority
Q1	EXISTS — ... where text like "%p%" limit 1	Backward search only; no suffix-array sampling. The one case an early-exit brute-force scan can win — omitting it overstates index benefit.	core
Q2	COUNT — count(*) where text like "%p%"	FM-index gets this from backward search at O(|P|), independent of hit count; n-gram and scan must walk every hit. Largest expected inter-family gap.	core
Q2b	COUNT requiring match offsets, not just row IDs	rebar splits count from count-spans precisely because "Hyperscan can have quite different performance characteristics depending on whether the start of the match is requested from it or not." Same distinction applies to any index that can count without locating.	recommended
Q3	LOCATE full — select id where text like "%p%"	Cost ∝ occurrences × SA sample rate. The cliff. Q2 being cheap does not imply Q3 is.	core
Q4	LOCATE + LIMIT 10/100/1000	Early termination. If Q4 ≈ Q3, the engine computes the full set then truncates — and most real queries carry a LIMIT.	core
Q4b	Top-K ordered by another column among matches	A third collection mode distinct from count-only and unordered top-K (ClickBench Q23 is exactly this: text filter + wide projection + sort + LIMIT).	recommended
Q5	Filtered ANN — vector search + substring predicate	The RAG shape; exercises predicate→bitmap handoff and pre- vs post-filter planning. See the note on why this is the highest-value gap, below.	core
Q6	Snippet / context extraction	Tests the self-index property. If the implementation re-reads the original column, raw text must be counted in storage cost. Pins down the definition of "index size".	recommended
Q7	Document-level: DISTINCT doc_id, COUNT(DISTINCT doc_id), top-k documents by match frequency	These are separate operations in the compressed-index literature (document listing, document counting, top-k most frequent), not variants of one query.	optional

On Q5 being the highest-value cell. Elastic's own wildcard field type has never been benchmarked in combination with a selective filter: jpountz asked for exactly that on day one of elastic/elasticsearch#49993, the implementer confirmed it wasn't run, and #70612 — still open — notes that real security rules use wildcards "in combination with a term query on a keyword field". Compounding it, BinaryDvConfirmedQuery.matchCost() returns a hard-coded 1000f next to // TODO: how can we compute this?, so Lucene's two-phase iterator ordering is flying blind whenever a wildcard is ANDed with anything. The common case is the untested case.

Layer 2 — pattern shape

By anchoring — and note this axis flips which implementation wins, which is why it is no longer a footnote. In Elastic's published numbers, 80.40.134.103* runs 1 ms on plain keyword versus 5 ms on the wildcard field, because the FST prefix-seeks.

Shape	Form	Note
Infix	%p%	Core case; a token-based inverted index cannot serve it
Prefix	p%	Servable by an ordinary sorted index — planner check: if prefix is slower than infix, the engine is blindly routing to the substring index
Suffix	%p	TPC-H Q2
Multi-segment	%a%b%	TPC-H Q13/Q16. Requires conjunctive n-gram intersection plus order verification — strictly harder than ANDing two independent LIKEs
Internal wildcard	%a_b%	Lucene's only wildcard shape (re*f)

By combination: multi-substring AND; OR/alternation; NOT LIKE; negated multi-segment not like '%a%b%'; negated prefix not like 'p%'. Negation matters because a candidate-generating index is useless for a complement — ripgrep's nothing benchmark (-v on .*) exists for the same reason.

By semantics: case-insensitive; word-boundary matching — LIKE '%cat%' matching "concatenate" is a semantic difference, and this is the fault line between LIKE and a full-text index, invisible to a pure LIKE matrix; overlapping vs non-overlapping match semantics.

Special constructions, all new and all cheap:

• Graded zero-match patterns. Not one non-matching string but three, ranked by how common their constituent bytes are — rebar uses zqj / aqj / aei, all with count 0, which differ by an order of magnitude on anything with a byte-frequency prefilter. This is the single largest index win in Elastic's data: *shell_exec* (0 hits) is 1042 ms on keyword vs 1 ms on wildcard.
• Adversarial patterns: a pattern whose n-grams are all high-frequency, so the index returns near-100% candidates and verification dominates. This makes index miss rate a deliberate axis rather than an accident.
• Multi-byte / CJK — a correctness assertion, not a performance measurement. The classic substring-index bug is treating byte boundaries as character boundaries. Without it a fast-but-wrong implementation passes.

Layer 3 — parameter axes

• Selectivity, bucketed by relative anchoring rather than absolute rates. luceneutil defines High ≥ maxDF/10, Med ≥ maxDF/100, Low below, with the AndHighLow pool at df < maxDF/1000 — anchored to the corpus's hottest term, so bucket definitions stay stable as the corpus grows, unlike fixed 1e-6…1e-2 thresholds. A useful empirical note from their 10M-doc run: maxDF = 8,217,362 (the, in 82% of docs), and the entire High bucket contains only 84 terms — the high-frequency region is intrinsically narrow.
• Pattern length 2 / 4 / 8 / 16 / 32, extensible to 128. (SMART sweeps 2^k up to 4096.)
• Pattern-set size — 1 → 10 → 100 → 1k → 10k, run both as one alternation and as N separate patterns. Every serious suite sweeps this: rebar 96 → 2,663 → 43,029 → 123,115; Hyperscan 1k → 5k → 10k → 26k; aho-corasick benchmarks 100,000 Wikipedia titles; Suricata ships ~32,800 signatures. Hyperscan's headline is one-at-a-time vs all-in-parallel on 1,300 regexes: 173 s vs 2.15 s.
• Haystack shape — at equal total bytes, many short rows vs few long ones. rebar runs the same pattern set at 61 KB and at 108 bytes because "throughput workloads tend to correspond to regexes that rarely or never match, whereas latency workloads tend to correspond to regexes that match very frequently." ripgrep's 9.3 GB single-file benchmark is what surfaced a ucg bug on files over 2 GB.
• Match density, independent of row selectivity — "1% of rows match" and "matching rows contain 50 hits each" are orthogonal.
• Alphabet / encoding — ASCII, 2-byte UTF-8, 3-byte UTF-8. rebar runs every literal benchmark in English, Russian and Chinese; SMART sweeps alphabet size 2…256 explicitly.

Reporting

• Build time on its own leaderboard. rebar publishes two rankings that disagree sharply — Hyperscan is first on search and last on compile, at 582× the best. Suricata 8 added on-disk caching of compiled Hyperscan databases, cutting startup by up to 50%, because compile cost is an operational problem at production rule counts.
• Index size: absolute, ratio to raw text, and per-field worst case. The ECS 1.8 wildcard migration was reverted before release (elastic/ecs#1233). Aggregate numbers looked benign — indexing −5%, disk +5% — while per field, url.original went 645 MB → 6.2 GB (9.5×) and error.stack_trace went 27 KB → 628 MB (~23,000×). Aggregates hide exactly the failure mode that matters.
• Report the tail over a pattern population, not per-pattern means. elastic/elasticsearch#62817 proposed dropping ~50% of n-grams by hashCode() % 2, saving ~35% disk with sample queries "not noticeably slower" — but *ERROR*'s n-grams err/rro/ror all hashed away, falling back to a full doc-values scan. It was rejected with: "why should *.exe* take hours to find 10 results but *.foo* take milliseconds? Having this distinction be on the roll of a dice…" No published wildcard benchmark reports p99 over a pattern population; this one should.
• Find and report the hit-count crossover. Elastic's data shows keyword flat at ~1040 ms regardless of hits while wildcard scales with matches, reaching parity at * 200 * with 1,121,055 hits. The crossover point is more informative than any single measurement.
• Normalize the way the compressed-index literature does: counting reported as µs per pattern symbol, locating as time per occurrence. Otherwise results at different pattern lengths and hit counts aren't comparable. Plot against index size as a space/time tradeoff.
• Correctness built in. luceneutil cross-checks hitCount between baseline and candidate and fails the run if they differ; every rebar benchmark carries an expected count, making the benchmark a correctness test. Substring ground truth is exact, so we can hold a stricter line: recall must be exactly 1.0 and any deviation is a bug, not a tuning trade-off.
• Also: latency p50/p95/p99, concurrent QPS, resident memory, and separate reporting for indexed vs not-yet-indexed data in engines where newly written rows are unindexed until sealed.

Default subset

The full matrix is a long run. Default: the 1 GB Hadoop slice plus the two small corpora; operations Q1–Q4; infix patterns; three selectivity levels (1e-5 / 1e-3 / high-frequency) × three lengths (2 / 8 / 32) — plus two additions that cost almost nothing: the three graded zero-match patterns, and the prefix and suffix anchorings. Zero-match is where an index wins by three orders of magnitude, and anchoring is where the winner flips; leaving either out means assuming the conclusion. Everything else goes behind a full flag.

Deliberately out of scope

Approximate / fuzzy matching gets its own track, not a cell in this matrix. It is absent from Pizza&Chili's API, the r-index, and aho-corasick — but it is first-class in Lucene nightly (Fuzzy1/Fuzzy2/Respell, 500 queries each), it is 3 of 14 benchmarks in ANMLZoo, Hyperscan exposes HS_EXT_FLAG_EDIT_DISTANCE, and BWA/Bowtie implement it as bounded backtracking over the FM-index prefix trie — i.e. it is native to this index family, and those tools benchmark speed and accuracy jointly. Notably, no established benchmark exists for large-scale fuzzy multi-pattern matching: ANMLZoo's Mesh class tops out at 93 patterns, three orders of magnitude below exact-match scales. That is an opening rather than something to copy, and it deserves separate treatment with the caveat that there is no baseline to align against.

Compressed-suffix-tree operations (Parent/SLink/LCA/SDepth) are structural internals rather than user queries, and matching statistics / MEMs are standard only in the pangenomics r-index branch, with no database analogue. Both excluded.