The NameRes log-analysis notebook (#107) emits a replayable benchmark at log-analysis/nameres/benchmark/nameres_autocomplete_solr_benchmark_*.json: one case per unique (query, params) combination — 416 cases from the current 607-row autocomplete export — each carrying everything needed to reissue the lookup plus the observed Solr baseline latency (n_observed, solr_wait_ms_p50, solr_wait_ms_p95, took_ms_p50, ever_slow, first_seen, last_seen).
Nothing consumes it yet. That is the whole point of the exercise: comparing Solr against the ElasticSearch-backed NameRes on real production queries.
- Issue each case against a live NameRes endpoint, once per backend, recording measured latency per case.
CachedNameRes (src/babel_validation/services/nameres.py) already wraps lookup; caching must be off or per-run for a latency measurement to mean anything.
- Every case has
autocomplete=True. A harness that drops that flag measures a different workload entirely.
- Join the replayed latencies back to
baseline_solr for per-case and aggregate speedup/regression tables. Start with the pathological cases: short prefixes at limit=100.
- Compare against
ci and ci-es, not dev/exp — per the root CLAUDE.md, those two run the same Babel data with different backends, which is what separates "the backend ranks this differently" from "the data changed".
Latency is only half of it; result-set parity is the other half, and is worth its own issue if this one gets large.
The NameRes log-analysis notebook (#107) emits a replayable benchmark at
log-analysis/nameres/benchmark/nameres_autocomplete_solr_benchmark_*.json: one case per unique (query, params) combination — 416 cases from the current 607-row autocomplete export — each carrying everything needed to reissue the lookup plus the observed Solr baseline latency (n_observed,solr_wait_ms_p50,solr_wait_ms_p95,took_ms_p50,ever_slow,first_seen,last_seen).Nothing consumes it yet. That is the whole point of the exercise: comparing Solr against the ElasticSearch-backed NameRes on real production queries.
CachedNameRes(src/babel_validation/services/nameres.py) already wrapslookup; caching must be off or per-run for a latency measurement to mean anything.autocomplete=True. A harness that drops that flag measures a different workload entirely.baseline_solrfor per-case and aggregate speedup/regression tables. Start with the pathological cases: short prefixes atlimit=100.ciandci-es, notdev/exp— per the root CLAUDE.md, those two run the same Babel data with different backends, which is what separates "the backend ranks this differently" from "the data changed".Latency is only half of it; result-set parity is the other half, and is worth its own issue if this one gets large.