Add a NameRes autocomplete log-analysis notebook and replayable Solr benchmark

[gaurav]

NameRes autocomplete queries are a small slice of traffic — ~3% of lookups — and by far the worst-behaved slice. In a mixed 16,461-row CloudWatch export, autocomplete p95 Solr wait was 2,803 ms against 43 ms for exact-match lookups, with a p99 of 264 seconds. They are also the most latency-sensitive requests NameRes serves, because a user is waiting on them between keystrokes.

This PR adds a marimo notebook that characterizes that workload from production logs and distils it into a benchmark we can replay against a Solr backend and an ElasticSearch one, so the two can be compared on real queries rather than synthetic ones. It also moves the existing NodeNorm notebook into log-analysis/nodenorm/ and writes down how to work on marimo notebooks in this repo.

What's here

log-analysis/nameres/analyze_nameres_logs.py — a marimo notebook, autocomplete-only. It loads every data/log-analysis/nameres-autocomplete-only-*.json export, parses each Solr lookup log line into a QueryLogEntry, and reports latency by result limit and query-length bucket, slow-query rates, the slowest individual lookups, and inferred typing chains. It emits two artifacts: a replayable JSON benchmark (one case per unique query+params, carrying the observed Solr baseline) and a per-lookup CSV.

Multi-export loading with an overlap guard. Logs Insights exports are plain time-window dumps with no per-record ID, so two exports covering the same window would silently double-count the same lookups and bias every statistic with no visible symptom. The loader computes each file's span from the raw @timestamp values in a first pass and refuses to proceed if two spans intersect, naming the offending pairs and their windows. Spans are treated as closed intervals, so exports that merely touch at an endpoint are also rejected.

Autocomplete-only, by deliberate choice. The notebook started out analysing all lookups. Mixing the two modes made every aggregate a weighted average of two populations that behave nothing alike, so exact-match analysis was split out (#139) and the filter moved upstream into the CloudWatch query itself:

SOURCE "[application logs]" START=-10w END=now
| fields @timestamp, @message, @logStream, @log
| filter @message like /Lookup query to Solr.*autocomplete=True/
| sort @timestamp desc
| limit 10000

That filter is selective enough that a 10-week window returns 607 rows — well under the 10,000-row cap — so unlike the general exports, this is the complete autocomplete population for its window rather than a sample of it. The loader's glob is correspondingly narrow (nameres-autocomplete-only-*), because the general exports share a directory and would both dilute the analysis and trip the overlap guard.

Typing chains. An autocomplete log line is one keystroke's worth of a query, so diab alone says little. Consecutive lookups are joined into a chain when they share a request shape (limit plus every filter), one query is a case-insensitive prefix of the other, and they fall within a tunable gap — reconstructing diab → diabe → diabetes type 2. The logs carry no session or user ID, so this is a heuristic, and it is documented as one: it merges concurrent identical requests from different callers, and splits a user who pauses. The result is not sensitive to the threshold — between a 30 s and a 60 s gap the chain count moves under 3%, because the median gap between prefix-compatible queries is ~2 s.

Two "final term" columns are emitted because they answer different questions and disagree whenever a user backspaces at the end: final_query (chronologically last — what they were left looking at) and longest_query (the fullest term they ever typed). On the real data, salis → salisylic → … → salic gives salic and salisylic respectively.

What it produces

From the current 607-row export (2026-08-03 → 2026-09-01, 399 inferred chains):

• log-analysis/nameres/output/nameres_autocomplete_terms_<span>.csv — one row per lookup: query, time, solr_wait_ms, final_query, longest_query, plus chain position and request parameters.
• log-analysis/nameres/benchmark/nameres_autocomplete_solr_benchmark_<span>.json — 416 replayable cases, each with baseline_solr percentiles observed in production. Nothing consumes it yet; the replay harness is Build the Solr-vs-ElasticSearch replay harness for the NameRes benchmark #141.

Both are gitignored, along with exports/. Generated artifacts are named after the log window they cover so two runs over different exports cannot overwrite each other.

One finding worth surfacing on its own: the six slowest lookups in the whole export were the same query, vitamin a, issued 35 ms apart — concurrent, not sequential — each blocking on Solr for 4.4 to 5.4 minutes at limit=100 with a single Biolink type filter. That is 6 of the only 7 lookups exceeding 60 s, against a median of 48.5 ms. Filed upstream as NCATSTranslator/NameResolution#306.

Supporting changes

• Tests (tests/log_analysis/, 30 tests, marked unit). marimo notebooks are not importable as modules — the directory name has a hyphen and cells are @app.cell functions — but they load by path, and Cell.run() executes a cell plus its ancestors and hands back its definitions. The notebook's pure helpers live in their own data-free cells (parser, span_helpers, chain_helpers) specifically so this works without the uncommitted log exports. Covers field extraction, SLOW QUERY detection, loud failure on unparseable lookup lines, filter normalization, span computation, all four overlap shapes, and chain assignment including backspacing, case-insensitivity, gap splits, request-shape isolation and index alignment.
• log-analysis/CLAUDE.md — how to drive a live marimo notebook via marimo._code_mode, the graph rules, cm.get_context() gotchas, path handling, how to test notebook logic, and how to export for sharing.
• NodeNorm notebook moved to log-analysis/nodenorm/, so the two services sit side by side.
• pandas/numpy added as dependencies; marimo[recommended] as a dev dependency.

Decisions a reader would otherwise have to reverse-engineer

• Paths anchor on mo.notebook_dir(), never the process cwd. marimo inherits the cwd of whatever launched it, so a cwd-relative Path("benchmark") resolves next to the notebook when opened from its own directory and at the repo root when opened from there — which is how generated artifacts ended up outside the .gitignore written to catch them, silently, because both spellings appear to work.
• pod_name and image_tag are parsed but kept off df. df is the one frame rendered whole, so it is the only place a parsed field reaches the shared HTML export; every other displayed table names its columns explicitly. Nothing here analyses either field. They stay on entries for a future notebook comparing NameRes releases (Report NameRes latency per typing chain, not just per lookup #142).
• Exports are shared out of band, not through this repo. marimo export html produces a self-contained file that opens with nothing installed and cannot be forked by accident, which is why it is preferred over .ipynb. Its rendered outputs embed every query string real users typed, so exports/ is gitignored.
• The root .gitignore needs both data/ and data — the first matches a directory, the second also matches a worktree's symlink to a shared one.

What this deliberately does not do

• No exact-match (autocomplete=False) analysis — split out to Split exact-match NameRes lookups into their own log-analysis notebook #139.
• No replay against either backend, and no result-quality comparison; the benchmark is emitted but unconsumed (Build the Solr-vs-ElasticSearch replay harness for the NameRes benchmark #141).
• Overlapping exports are rejected rather than de-duplicated (De-duplicate overlapping NameRes log exports instead of rejecting them #140).
• Chain data is descriptive only; it is not yet a metric either backend is judged on (Report NameRes latency per typing chain, not just per lookup #142).

Follow-on issues

• Split exact-match NameRes lookups into their own log-analysis notebook #139 — split exact-match NameRes lookups into their own notebook
• De-duplicate overlapping NameRes log exports instead of rejecting them #140 — de-duplicate overlapping log exports instead of rejecting them
• Build the Solr-vs-ElasticSearch replay harness for the NameRes benchmark #141 — build the Solr-vs-ElasticSearch replay harness
• Report NameRes latency per typing chain, not just per lookup #142 — report NameRes latency per typing chain, not just per lookup
• Make the marimo notebook's provenance survive into its shared exports #143 — make the notebook's provenance survive into its shared exports
• NCATSTranslator/NameResolution#306 — concurrent vitamin a autocomplete lookups blocking on Solr for minutes

Before merging

• This branch is 138 commits behind main. Merge or rebase before landing — main has since added pytest-xdist, the github_issues test source and a reworked tests.yaml, none of which this branch has seen. The pyproject.toml [tool.pytest.ini_options] block is touched on both sides and will want attention.
• pandas and numpy are in [project] dependencies, but nothing in src/babel_validation/ imports either — only the notebook and its tests do. They probably belong in the dev dependency group alongside marimo[recommended], so installing the package doesn't drag in pandas for nothing.

TODO: size NameRes's bulk-lookup concurrency limit

Not addressed by this PR — it needs /bulk-lookup request logs, which this notebook does not read. Worth splitting into its own issue if it isn't going to happen here.

NCATSTranslator/NameResolution#273 makes /bulk-lookup run its per-string lookups against Solr concurrently instead of one at a time, bounded by SOLR_MAX_CONCURRENT_LOOKUPS, currently defaulted to 100. That number is a guess: it was picked to be high enough to help large requests while still capping the fan-out, without knowing what NameRes is actually asked to do.

The bound applies per request, so what Solr sees at peak is roughly SOLR_MAX_CONCURRENT_LOOKUPS × (concurrent /bulk-lookup requests). At 100, ten simultaneous bulk lookups is already ~1,000 queries in flight. That is the product this log analysis needs to pin down.

• From the NameRes logs, establish (a) the peak number of /bulk-lookup requests in flight at once, and (b) the distribution of strings lengths per request, particularly the tail. Together those give the peak concurrent Solr query count implied by any given limit, and show whether the limit binds on real traffic at all — if the p99 request carries 20 strings, a limit of 100 never engages and the value doesn't matter. Then set SOLR_MAX_CONCURRENT_LOOKUPS in NameResolution accordingly (it is settable by environment variable, so it can be tuned per deployment without a code change).

TODO: make chain inference cheap enough for a wider log window

• assign_chains walks the frame with iterrows() and rebuilds a request_shape tuple from the list-valued filter columns on every row. Imperceptible at 607 rows, but the chain gap is a mo.ui.slider, so every drag re-runs the chain build and the CSV export downstream of it. Before widening the export window, precompute the shape column once and iterate over itertuples/NumPy arrays. If the slider still lags at ~10k rows, debounce it or move the chain build behind an mo.ui.run_button.

How this PR got here — kept for anyone tracing why a particular line looks the way it does; the durable conclusions are above and in the code.

• The notebook was first built against a single hard-coded log file (logs/nameres-log-analytics-results-2026-07-06.json) that did not exist in the worktree. It then grew multi-file loading over data/log-analysis/nameres-*, and only afterwards was narrowed to autocomplete-only — which is why the overlap guard, written for the general case, is what now forces the autocomplete exports into their own glob.
• Analysis was originally split by mode (autocomplete vs exact) throughout. Every one of those splits became a no-op when exact lookups left, and was replaced by a split on result limit and query-length bucket. box_by_mode became box_by_qlen.
• Generated artifacts initially landed in benchmark/ and output/ at the repo root, because the paths were cwd-relative and marimo had been launched from there — past log-analysis/nameres/.gitignore, which existed to catch exactly those files. Fixed by anchoring on mo.notebook_dir(); the trap is recorded in log-analysis/CLAUDE.md.
• The first read of the vitamin a chain described it as ~29 minutes of Solr wait "across the chain", by summing six waits. Checking the timestamps showed the six requests were 35 ms apart and therefore concurrent, so the sum is of overlapping intervals and the elapsed time is ~5 minutes, not 29. The upstream issue describes the concurrent reading.
• The export was generated once with pod_name and image_tag still in df, embedding the Kubernetes pod name and image tag in a file meant to be shared outside the project. Dropped from df and re-exported; the .gitignore and CLAUDE.md notes that had justified gitignoring exports/ because of those fields were corrected, since a stale reason is worse than none — someone could read it and conclude the export is now safe to commit.
• The test file was added without pytestmark = pytest.mark.unit. main's only pytest job is pytest -m unit, so all 30 tests would have been silently deselected on merge — the same failure Publish a daily cross-repository Babel milestones page as part of the dashboard site #112 shipped. Marker added, and unit registered in pyproject.toml verbatim from main so the two merge cleanly.

[Copilot]

Pull request overview

Adds a new log-analysis/ workflow for analyzing NameRes production query logs and exporting a replayable benchmark dataset, aligning with the existing NodeNorm log-analysis efforts.

Changes:

• Add a marimo-based NameRes log analysis notebook that parses CloudWatch exports, computes summary stats, and exports benchmark cases.
• Add log-analysis documentation and .gitignore rules to keep raw logs and generated artifacts out of git.
• Update Python dependencies to support the notebook tooling (marimo/Altair + pandas/numpy).

Reviewed changes

Copilot reviewed 5 out of 8 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
pyproject.toml	Adds data/analysis dependencies and a dependency-groups.dev section for marimo tooling.
log-analysis/nodenorm/logs/README.md	Documents how to export NodeNorm logs from CloudWatch Logs Insights.
log-analysis/nameres/logs/.gitignore	Ignores raw NameRes log exports (*.json).
log-analysis/nameres/analyze_nameres_logs.py	New marimo notebook for parsing NameRes lookup logs, visualizing latency, and exporting a benchmark JSON.
log-analysis/nameres/.gitignore	Ignores marimo cache and generated benchmark artifacts.
log-analysis/CLAUDE.md	Adds contributor guidance for working on the log-analysis notebooks and dependency management via uv.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.