Commit 90fb7da
authored
Add exact-match mode to /lookup and /bulk-lookup, and run bulk lookups in parallel (#273)
Adds an `exact` parameter to `/lookup` and `/bulk-lookup`, so that
callers who already know the string they are looking for can ask for
whole-string matching instead of the default tokenized search, and sends
the individual lookups behind `/bulk-lookup` concurrently instead of one
at a time. The motivating case is named entity recognition pipelines,
which resolve large numbers of exact strings in bulk and pay for
relevance ranking they never use.
Closes #258.
## What's here
**Exact matching.** When `exact` is set, the eDisMax query is bypassed
entirely in favour of a Solr filter query against the `*_exactish`
fields. Those are indexed with a keyword tokenizer and a lowercase
filter, so the whole string must match, case-insensitively — `parkinson`
does not match the label `parkinsonian disorder`, but `Parkinsonian
Disorder` does.
| `exact` | Matches against |
| ----------- | ---------------------------------------------------- |
| `label` | The preferred name only (`preferred_name_exactish`) |
| `synonyms` | The synonyms only (`names_exactish`) |
| `any` | Either |
| _(omitted)_ | Nothing changes — the usual tokenized eDisMax search |
`label` and `synonyms` can genuinely disagree, because a concept's
preferred name is not necessarily one of its synonyms — HP:0001300 is
`preferred_name="parkinsonian disorder"` with `names=["Parkinsonian
disease"]`, and neither string appears in the other field. That is why
the two modes are separate rather than one boolean, and it is also a
data problem in its own right:
[NCATSTranslator/Babel#1073](NCATSTranslator/Babel#1073)
proposes putting each clique's preferred name into its synonyms.
**Interaction with the other parameters.** Three of them behaved badly
with `exact`, all silently, and all now do something defensible:
- `autocomplete=true` was ignored. It treats the final word as an
incomplete prefix, which contradicts matching the whole string, so the
combination is now a `400` rather than a request that quietly does
something else. The rejection lives in `lookup()`, so it applies to
`/bulk-lookup` too.
- The smart-quote rewrite (#176) was applied to the exact query. The
`*_exactish` fields do no punctuation folding, so the indexed value
keeps whichever quote characters Babel emitted; folding the query
searched for a string the caller had not typed, and put any label
containing a typographic quote out of reach. Exact mode no longer folds.
The default path is unaffected, because StandardTokenizer discards the
punctuation anyway.
- `highlighting=true` returned empty arrays, because the query is `*:*`
and the exactish fields are not stored, so Solr had nothing to mark up.
Since every exact match is a whole-value match, the highlighting is now
synthesized from the returned document — the entire matching name, in
its own capitalisation, wrapped in the same `<strong>` tags the default
path asks Solr for. Callers no longer have to care which mode produced
the field.
**Parallel bulk lookup.** `bulk_lookup()` issues its Solr queries with
`asyncio.gather()` rather than awaiting them one at a time, bounded by a
semaphore of `SOLR_MAX_CONCURRENT_LOOKUPS` (default 100). The bound is
not optional: `NameResQuery.strings` has no upper limit, so an unbounded
gather would open one socket per string and could exhaust the process's
file descriptors and stampede Solr. Sequential execution used to make
that impossible, so the parallelization is what introduces the hazard
and the semaphore is what contains it.
The limit applies per request rather than per process, so the queries in
flight against Solr is that number multiplied by the number of bulk
lookups being served at once. 100 is set deliberately high on the
assumption that Solr can take the strain; sizing it against real traffic
is being worked out in
[TranslatorSRI/babel-validation#107](TranslatorSRI/babel-validation#107),
and it is an environment variable so it can be retuned per deployment
without a code change.
Two hazards that parallelism sharpened are closed off with it. The limit
is clamped to at least 1, since `SOLR_MAX_CONCURRENT_LOOKUPS=0` would
build a semaphore nobody can acquire and hang every bulk request with no
error and no log line. And Solr queries now have a timeout
(`SOLR_TIMEOUT_SECONDS`, default 60) instead of `timeout=None`:
sequentially a stalled connection held up one query, but concurrently it
can pin an otherwise-complete bulk request indefinitely while holding a
semaphore slot.
**The default search is not "fuzzy".** Documentation and parameter
descriptions called it that; it never was. `LowerTextField` is a
`StandardTokenizer` and a `LowerCaseFilter` with no stemming, stopwords,
synonyms or ngrams, and `lookup()` escapes Solr's `~` operator out of
the query string, so no caller can request edit-distance matching
either. The default search is _tokenized_: it matches the query's tokens
in any order, rewarding order and adjacency with score rather than
requiring them, and treats the final token as a prefix when
`autocomplete=true`. Calling it fuzzy invited callers to expect typo
tolerance the service has never had.
**Documentation.** `documentation/API.md` gained an "Exact matching"
section, the `exact` parameter and the notes above — that file
enumerates every `/lookup` parameter and shows a complete `/bulk-lookup`
body, so omitting `exact` would have made it wrong rather than merely
incomplete. `documentation/Deployment.md` gained the two new environment
variables, the OpenAPI service blurb now mentions exact mode alongside
autocomplete, and `CLAUDE.md` gained a Gotchas section for the three
things here that a reader cannot infer from the code.
## What it produces
Exact matching is cheaper than the equivalent tokenized search because
there is nothing to score: no eDisMax parsing, no phrase or field
boosts, just a term lookup against a field holding each name as a single
token. Bulk requests of ordinary size now complete in roughly the time
of their slowest lookup rather than the sum of all of them.
The exactish filter is marked `{!cache=false}` on purpose. Solr's
filterCache is bounded by entry count (512 in `solrconfig.xml`), and
what it holds are the shared, reusable filters — `types:`, `taxa:`,
`curie:` — that nearly every search benefits from. An exactish clause is
one distinct entry per distinct search string, so caching it would let a
single bulk NER request evict the whole cache and slow down the
_ordinary_ search path as collateral damage, for a hit rate near zero on
its own entries. Repeated identical exact lookups are still served from
the queryResultCache, which caches the whole result and is bounded by
RAM.
Because there is no relevance score to rank by in exact mode, results
are sorted by `clique_identifier_count DESC, curie_suffix ASC` instead
of the usual `score DESC, …`. The `score` field is still present in each
result but carries no ranking information.
## What it deliberately does not do
- **No stemming, normalisation, or punctuation folding beyond
lowercasing.** Exact means exact; a caller who wants looser matching
already has the default search.
- **No cap on `strings`.** The semaphore throttles a large bulk request
rather than rejecting it, so existing clients sending long lists keep
working and simply queue.
- **No change to which cliques the default search returns.** With
`exact` omitted the generated Solr query is what it was before; the only
behavioural difference anywhere is the Solr client's timeout, which used
to be unbounded.
## Tests
44, all against a live Solr in CI. Beyond the exact-match modes
themselves, several encode promises that a change elsewhere could
quietly break: that the default search ignores word order but does not
tolerate misspellings, that the exactish filter still reaches Solr
marked uncached, that exact matching composes with the `biolink_type`
and prefix filters, that the `autocomplete`/`exact` rejection survives
being raised inside `asyncio.gather()`, and that a bulk lookup carrying
more strings than the concurrency limit still returns every one of them
correctly keyed.
The exact-match cases live in `tests/test_exact_mode.py`, following the
repository's existing habit of splitting tests by topic;
`tests/test_service.py` keeps the ones that are about the default search
or about bulk lookup generally.
## Before merging
Nothing blocking.
Two things to pick up afterwards, both tracked outside this PR because
neither can be settled from inside it:
-
[TranslatorSRI/babel-validation#107](TranslatorSRI/babel-validation#107)
— confirm `SOLR_MAX_CONCURRENT_LOOKUPS = 100` against the real request
rate, specifically the peak number of concurrent `/bulk-lookup` requests
and the tail of the `strings`-length distribution, whose product is what
Solr actually sees. Needs log analysis, and the value is an environment
variable, so it does not gate this.
-
[NCATSTranslator/Babel#1073](NCATSTranslator/Babel#1073)
— put each clique's preferred name into its synonyms. If that lands,
`exact=synonyms` and `exact=any` converge for most cliques, and the
three-way split is worth revisiting _then_ rather than pre-emptively
now.7 files changed
Lines changed: 566 additions & 62 deletions
File tree
- api
- resources
- documentation
- tests
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
59 | 59 | | |
60 | 60 | | |
61 | 61 | | |
62 | | - | |
| 62 | + | |
63 | 63 | | |
64 | | - | |
| 64 | + | |
65 | 65 | | |
66 | 66 | | |
| 67 | + | |
67 | 68 | | |
68 | 69 | | |
69 | 70 | | |
70 | 71 | | |
| 72 | + | |
71 | 73 | | |
72 | 74 | | |
73 | 75 | | |
| |||
89 | 91 | | |
90 | 92 | | |
91 | 93 | | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
92 | 100 | | |
93 | 101 | | |
94 | 102 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
11 | | - | |
| 11 | + | |
| 12 | + | |
12 | 13 | | |
13 | 14 | | |
14 | 15 | | |
| |||
0 commit comments