Skip to content

[AIMOD-1354] Add "delete" index for AMP keywords#19

Open
vazish wants to merge 3 commits into
mainfrom
aimod-1354-add-fuzzy-index
Open

[AIMOD-1354] Add "delete" index for AMP keywords#19
vazish wants to merge 3 commits into
mainfrom
aimod-1354-add-fuzzy-index

Conversation

@vazish

@vazish vazish commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator
  • This PR adds a "delete" index for the "full_keywords" field from AMP json.
  • This will be used as part of the treatment branch for an A/B AMP experiment in Merino with the fuzzy matching portion as a treatment branch.
  • The best place to start review is SymIndex.build() function that builds the delete index over the full keywords, i.e. generates for the keyword example "world cup" -> "orld cup", "wrld cup", "word cup, ...
  • Then query_fuzzy() for the how it's called.
  • Once merged, I might need some help figuring out how to release!
  • (I did use Claude's help with the Rust side but reviewed it to make sure the approach was what I had in mind)

Some stats / tests done for latency, RSS:

AMP fuzzy index — perf & memory (before/after)

Measured on real US-desktop MARS (3,959 suggestions). Before = v0.3.1 (no fuzzy), after = v0.3.2 (with fuzzy).

Build time (per index — runs once per MARS cron resync)

median min max
0.3.1 (before) 7.7 ms 7.4 ms 13.5 ms
0.3.2 (after) 17.3 ms 17.0 ms 21.3 ms
Δ fuzzy +9.6 ms

Query latency (µs)

path version mean p50 p95 p99
exact 0.3.1 (before) 0.34 0.25 0.96 1.46
exact 0.3.2 (after) 0.35 0.25 1.00 1.46
fuzzy 0.3.2 (after) 4.27 2.29 11.7 27.8

Exact path is unchanged before/after; fuzzy is a new fallback, only on an exact miss.

RSS footprint (per built index)

footprint
0.3.1 (before) 16.2 MB
0.3.2 (after) 25.0 MB
Δ fuzzy +8.8 MB (~7 MB by peak-RSS method)

Memory-leak check (0.3.2)

scenario iterations RSS
rebuild same index (cron resync path) 400× flat ~85 MB
fuzzy query hammering 300,000× flat ~85 MB
build + delete distinct indexes 300× flat ~85 MB

Summary: ~+10 ms build and ~+8 MB per segment index; zero impact on the exact hot path;back; no leaks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant