Skip to content

CLAUDE.md

Latest commit

 

History

History
548 lines (488 loc) · 41.7 KB

CLAUDE.md

File metadata and controls

548 lines (488 loc) · 41.7 KB
Raw

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Repository Purpose

Local pipeline that ingests OHBM 2026 accepted abstracts from Oxford Abstracts (GraphQL), enriches them (figures, claims, references), embeds and clusters them, and exports a static search UI plus organizer-facing poster-layout/sequencing experiments. The README is the operational runbook; docs/reproducibility-vision.md is the project charter.

There are two coupled but distinct tracks:

  • Track A — canonical corpus pipeline: driven by aacli (src/abstractatlas/cli.py). Produces the authoritative artifacts under data/primary/, data/cache/, data/outputs/experiments/, and data/outputs/exported-sites/.
  • Track B — exploratory layout/sequencing: driven by standalone scripts in scripts/ and recorded runs under data/outputs/proposals/. These produce comparative evidence, not silent replacements for canonical outputs.

Non-negotiables (from .specify/memory/constitution.md)

The canonical constitution lives at .specify/memory/constitution.md (the root CONSTITUTION.md is a pointer). It applies on every turn, not just when Spec Kit slash commands run — see "Constitution applies to every turn" below. Short-name summary:

  • I. Venv-only Python. Always run through .venv/bin/python or uv explicitly targeting .venv/bin/python. Same rule for tests, scripts, one-offs, and dependency installs.
  • II. Immutable evidence, no committed data. Recorded experiment outputs are append-only (new runs → fresh directories). data/primary/abstracts.json is canonical raw corpus; cleanup belongs in explicit derivative artifacts. Data, caches, exports, downloaded assets MUST NOT be committed — data/, export/, tmp/, archive/, memory/archive/, .claude/ are gitignored; new artifact roots must be gitignored before any write.
  • III. Resumable, auditable pipelines. Long-running API/LLM jobs checkpoint incrementally; caches under data/cache/ are keyed by <state-key> so reruns skip completed records.
  • IV. Plan-first, test-first. Update the nearest plan/spec doc and add or identify failing tests before implementing. When canonical defaults change, docs in the same change.
  • V. Secret-safe, commit early and often. Secrets stay in .env; never commit, echo into logs, or paste into docs/transcripts. Commit each verified slice as it lands; do not accumulate hours of unrecorded work.
  • VI. Fail loudly, no shortcuts. No bare except, no silent fallbacks, no --no-verify or skipped tests/hooks to make CI green. Temporary workarounds must be labeled with root cause and follow-up.
  • VII. Discover external state, don't hardcode it. Upstream checkpoints, vendor enumerations, API schemas, and external file layouts MUST be discovered at runtime from metadata; mismatches surface as precise errors, never silent skips.
  • VIII. Provenance for organizer-facing outputs. Every artifact that reaches organizers, reviewers, or downstream consumers ships with machine-readable provenance (inputs, bundle, config, code revision, command, seed) alongside it — no absolute or user-home paths.

Constitution applies to every turn

Spec Kit slash commands run a Constitution Check automatically, but the constitution applies to every action in this repo — direct edits, ad-hoc prompts, bash-only work, debugging sessions, and unattended jobs all included. Before reporting work complete, self-check against the I–VIII short names above. If any check fails, the change is not complete.

The pattern-detectable subset of these checks is automated by the local lint:

.specify/scripts/bash/constitution-check.sh --staged   # what the pre-commit hook runs
.specify/scripts/bash/constitution-check.sh --full     # CI / manual sweep

The lint catches Principle II (tracked files under gitignored roots), Principle V (token-shaped strings in the staged diff), and Principle VI (bare except: in src/, --no-verify usage in committed code). It is necessary but not sufficient — Principles I, III, IV, VII, VIII still require judgment.

Install the lint as a pre-commit hook once per clone:

git config core.hooksPath .githooks

The hook lives at .githooks/pre-commit and delegates to the lint script.

Setup and validation

UV_CACHE_DIR=.uv-cache uv venv --python 3.14 .venv
PYTHONPATH=src .venv/bin/python -m unittest discover -s tests -v

Run a single test module:

PYTHONPATH=src .venv/bin/python -m unittest tests.test_neuroscape -v

Run a single test:

PYTHONPATH=src .venv/bin/python -m unittest tests.test_neuroscape.TestClass.test_method

Optional dependency groups (only install what a workflow needs):

  • Embeddings via HF/MiniLM: uv pip install --python .venv/bin/python sentence-transformers
  • Projections/UMAP: uv pip install --python .venv/bin/python plotly umap-learn
  • Stage 2 enrichment (figures + claims + references via OpenAI Responses API): uv pip install --python .venv/bin/python ".[enrich]"
  • Headless layout review: uv pip install --python .venv/bin/python ".[review]" then playwright install chromium

CLI entrypoint

The canonical interface is aacli (mapped in pyproject.toml):

PYTHONPATH=src .venv/bin/python -m abstractatlas.cli <subcommand>

Subcommands group into stages (see README for full options):

  • ingest/refresh: fetch-abstracts, fetch-withdrawn, refresh-assets (note: ingest and authors removed; authors are fetched inline by fetch-abstracts)
  • enrichment: enrich-abstracts, title-audit (note: enrich, analyze-figures, extract-claims, reference-metadata REMOVED in Stage 2 rewire — FR-014. Operators run enrich-abstracts and use --invalidate <component> for targeted refresh.)
  • embeddings: embed-matrix (Stage 3 canonical — per-component bundles for one or more (model, component) pairs; supports voyage / minilm / openai / pubmedbert; NeuroScape is a derivation step). Per-model debugging subcommands kept for backward compat: embed-minilm, embed-hf, embed-openai, embed-voyage, embed-stage2, apply-published-stage2. Multi-component recipes (full-manuscript, methods+results, title+results+conclusion) are composed downstream via neuroscape.compose_recipe([...], model_key=...).
  • analysis: semantic-analysis, cluster-benchmark, umap-plot, compare-projections, optimize-projections
  • UI data package: scripts/build_ui_data.py (Stage 6 canonical; emits site/static/data/ consumed by the SvelteKit site under site/).

Poster layout/sequencing is not in aacli AND is parked as of Stage 5 (see abstractatlas.layout parked package and scripts/layout/ for the 15 companion scripts: scripts/layout/optimize_poster_layout.py, scripts/layout/analyze_poster_layout.py, scripts/layout/benchmark_poster_sequencing.py, etc.). Always pass explicit input paths and a fresh --output-root/--output-dir; do not rely on stale baked-in defaults.

Code architecture

All library code lives in src/abstractatlas/:

  • graphql_api.py — Oxford Abstracts GraphQL client (env loading, batching, exponential-backoff retries). Defines ABSTRACT_IDS_QUERY (accepted), WITHDRAWN_IDS_QUERY (withdrawn), ABSTRACT_CONTENTS_QUERY (incl. program_code + program_sessions_submissions chain), INTROSPECTION_QUERY (canonical), and fetch_schema_introspection.
  • assets.py — figure asset download/refresh (reuse-aware via asset_stem matching), normalize_abstract (maps program_code → poster_id, flattens program_sessions_submissions → program_sessions), fetch_content_batches generator with per-batch + per-record callback hooks, advance_record_state state-machine validator.
  • fetch_stage.py — Stage 1 orchestrator for aacli fetch-abstracts / fetch-withdrawn; drives introspection → tiered schema diff → checkpoint lifecycle → batched fetch with figure download → atomic-write corpus + schema + provenance.
  • schema_diff.py — tiered HARD/SOFT/INFORMATIONAL field-level diff classifier; pure functions, no I/O.
  • exceptions.py — typed cross-stage exception hierarchy rooted at OhbmStageError(RuntimeError). Stage 1 subtree: Stage1Error → SchemaContractError, CheckpointError, FigureFailureError. Stage 2 subtree: Stage2Error → EnrichmentError, CacheVersionError, ComponentFailureThresholdError. ProvenanceError is shared (both stages enforce the no-absolute-/-no-~ path-boundary rule). Re-exports GraphQLAPIError.
  • enrich_stage.py — Stage 2 orchestrator for aacli enrich-abstracts; reads the accepted corpus, runs figures + claims + references components with per-component caching keyed by sha256(input || model_id), writes the enriched corpus as SQLite + zlib(json) per row, writes provenance with model identifiers and cache hit/miss counts. Optional Parquet export via --export-parquet PATH (lazy-imports pyarrow).
  • enrich_storage.py — EnrichedCorpusWriter SQLite I/O helper for Stage 2 (atomic temp→rename) plus read_one_by_id / iter_enriched / corpus_metadata. Stdlib only.
  • enrichment.py — Stage 2 building blocks (markdown conversion, legacy figure-analysis helpers). Wrapped (not refactored) by enrich_stage.py. Stage 2.1 replaces the cllm-based claim-extraction path with the agentic OpenAI Responses API call in stage2_claims.py.
  • stage2_figures.py, stage2_claims.py, stage2_references.py — Stage 2.1 per-component production runners. Figures: per-abstract grouped vision call with local JPEG-q85 compression + a four-field quality probe (image_quality.py). Claims: agentic Responses API call with three function tools (verify_source_quote, lookup_eco_code, dedupe_check) returning Pydantic-validated, ECO-annotated claims. References: thin adapter to the existing openalex.collect_reference_metadata pipeline.
  • flex_tier.py — OpenAI flex-tier retry/fallback helper used by figures + claims (1 flex attempt + 1 standard retry; default timeouts 120s figures / 180s claims).
  • image_quality.py — pure Pillow helpers for the local quality probe.
  • data/eco_top_codes.json — committed-source ECO v1 controlled vocabulary (9 top-level codes from ECO:0000000).
  • openalex.py — reference parsing pipeline: markdown normalization → LLM-assisted splitting (validated lexically against source) → DOI/PMID lookup → OpenAlex title search → Semantic Scholar fallback. Wrapped by Stage 2's references component.
  • neuroscape.py — embeddings (MiniLM/HF/OpenAI/Voyage), stage-2 projection (apply published NeuroScape model or train local), semantic community detection, k-sweep clustering benchmarks, UMAP, projection comparison/optimization.
  • titles.py — title normalization rules (used by title-audit).
  • artifacts.py — shared artifact-naming/state-key helpers used across stages.
  • category_evaluation.py, category_rollup.py — compare learned cluster families against submitter taxonomies.
  • layout/ (parked as of Stage 5 — specs/007-package-reorg/) — poster_layout, poster_sequencing, nocd_experiments. Preserved verbatim for future revival; not actively maintained. Tests under tests/test_poster_*.py + tests/test_nocd_experiments.py still run with import-paths updated to abstractatlas.layout.*. The 15 companion scripts live under scripts/layout/. Revive when a new organizer cycle needs poster-layout work.
  • ui_data/ — Stage 6 UI data-package builders. manifest.py, abstracts.py, authors.py, cells.py, topics.py, neighbors.py, enrichment.py, vectors.py produce per-shard envelopes (every shard carries a top-level build_info block — FR-019 + CA-008). state_key.py discovers the corpus + Stage 4 rollup state-keys at build time (CA-007). builder.py orchestrates + enforces the 8 cross-shard invariants from specs/008-ui-rewrite/data-model.md §8. link_check.py HEAD-validates specs/008-ui-rewrite/contracts/references.yaml (every external citation from the About page; non-zero exit blocks the deploy — FR-017). CLI entry: scripts/build_ui_data.py. Schema: every emitted JSON shard validates against specs/008-ui-rewrite/contracts/ui_data.linkml.yaml via scripts/validate_ui_data.sh. The SvelteKit site lives at site/ (self-contained pnpm project; gh-pages deploy via .github/workflows/{deploy-ui,pr-preview,pr-preview-cleanup}.yml; runtime data fetched from the Dropbox tarball at vars.OHBM2026_UI_DATA_PACKAGE_URL).
  • atlas_hosting/ — Stage 20 (spec 020-cloudflare-r2-migration) data-bundle publishing to Cloudflare R2 + Dropbox-vs-R2 comparison. content_hash.py (streamed sha256 + <sha256>/<filename> content-addressed keys), r2_client.py (boto3 S3 client from .env; head_object existence; multipart upload; creds by name only), uploader.py (discover the four required parquets across two locations — ohbm2026.parquet via --ohbm2026-parquet, neuroscape/atlas/neuroscape_vectors from --package-dir; idempotent skip-if-present + size-guard; immutable, never overwrites), manifest.py (UploadManifest provenance under data/provenance/atlas_upload_provenance__<key>.json), compare.py (Range/CORS/byte-parity probes → data/outputs/data-hosting-comparison__<ts>.json). CLI: aacli upload-atlas-package + compare-data-hosting; typed Stage20Error subtree. R2 public base = https://aadata.cirrusscience.org; the site loader + resolve-data-channel.sh are UNCHANGED (R2 URLs are opaque). Dropbox stays the production default — cutover deferred.
  • cli.py — single dispatch entrypoint that wires the above into subcommands.

Tests in tests/ mirror the module names and use unittest.

Artifact layout contract

The directory hierarchy is part of the contract — don't write to other roots:

  • data/inputs/ — fetched GraphQL snapshots, API-derived inputs, operator-supplied inputs (e.g. authors, poster layout geometry, manual CSVs).
  • data/primary/ — canonical normalized datasets consumed downstream (abstracts.json, abstracts_withdrawn.json, authors.json, abstracts_enriched.sqlite — Stage 2 canonical SQLite+zlib; the legacy abstracts_enriched.json is retained until downstream consumers migrate).
  • data/cache/ — resumable caches. Stage 1's fetch_abstracts/checkpoint__<state-key>.json uses the legacy state-key naming. Stage 2's per-component caches under figure_analysis/, claim_analysis/, reference_metadata/ are keyed by sha256(input || model_id) and named <cache-key>.json.
  • data/outputs/experiments/ — clustering, embeddings, projections, audit outputs.
  • data/outputs/proposals/ — poster-layout proposal bundles and analyses.
  • data/outputs/exported-sites/ui-site__<state-key>/ — legacy local UI bundle root (the export-ui / build-ui CLI commands were retired with the Stage 6 rewrite; left in the contract for any remaining legacy bundles).
  • export/ui-site/ — legacy publish mirror of the retired UI bundle.
  • site/static/data/ — Stage 6 static-JSON shards produced by scripts/build_ui_data.py.
  • archive/ — local pre-migration backups; preserves legacy paths.

data/, export/, tmp/, archive/, and memory/archive/ are gitignored.

Default pipeline state

Current canonical defaults (the UI consumes these):

  • Stage 2 single entry: aacli enrich-abstracts (scripts/run_enrich_abstracts.py). Reads data/primary/abstracts.json, writes data/primary/abstracts_enriched.sqlite + per-component caches + data/provenance/abstracts_enrich_provenance__<state-key>.json. Optional --export-parquet PATH.
  • figure-interpretation model: OpenAI gpt-5.4-mini (flex tier on by default), per-abstract grouped Responses API call with manuscript-text context + in-memory JPEG-q85@1024px compression + a four-field local quality probe. Cached under data/cache/figure_analysis/<cache-key>.json.
  • claims-extraction: agentic OpenAI Responses API call with gpt-5.4-mini (flex tier on by default) — three function tools (verify_source_quote, lookup_eco_code, dedupe_check); Pydantic-validated structured output annotated with ECO v1 codes. Cached under data/cache/claim_analysis/<cache-key>.json (key = sha256(manuscript || model_id || vocabulary_version)). The legacy cllm zero-shot path was removed in Stage 2.1.
  • reference-resolution strategy: refs.v1+openai-gpt-5-nano (multi-stage: LLM-assisted splitting → DOI/PMID → OpenAlex title search → Semantic Scholar fallback), cached under data/cache/reference_metadata/<cache-key>.json.
  • Stage 3 single entry: aacli embed-matrix (scripts/run_embed_matrix.py). Per-component bundles for voyage / minilm / openai / pubmedbert × {title, introduction, methods, results, conclusion, claims}. Output: data/outputs/embeddings/<model_key>/<component>__<state-key>/{vectors.npy,ids.npy,metadata.json,provenance.json}. State-key suffix on the bundle dir lets multiple historical versions coexist; clean stale corpora with rm -rf data/outputs/embeddings/*/*__<old_state_key>. Run-level provenance at data/provenance/embeddings_matrix_provenance__<state-key>.json. Per-abstract cache under data/cache/embeddings/<model_key>/<cache-key>.json (key = sha256(text || model_id || model_version)).
  • embedding bundles in use by the UI (recipes composed at read time via neuroscape.compose_recipe): voyage_stage2_published (mean of voyage title+introduction+methods+results+conclusion, then optional NeuroScape Stage-2 transform) and minilm_claims (the per-component minilm_claims bundle directly).
  • UI projection: composed at consumption time from the per-component minilm bundles using compose_recipe(["title", "introduction", "methods", "results", "conclusion"], model_key="minilm").

Reading order for unfamiliar context

  1. docs/reproducibility-vision.md — project charter, what is canonical vs exploratory.
  2. README.md — operational runbook with every subcommand example.
  3. .specify/memory/constitution.md — hard rules (root CONSTITUTION.md is a pointer).
  4. memory/summary.md — reconstructed history of major design moves.
  5. The plan doc under docs/ closest to the area you're touching (e.g. static-ui-plan.md, poster-layout-optimizer-plan.md).
  6. The Spec Kit plan under specs/<NNN>-<topic>/plan.md for the most recent or most relevant stage if you're touching its area.

For additional context about technologies to be used, project structure, shell commands, and other important information, read the current plan at specs/027-abstractatlas-rename-ingestors/plan.md. Stage 27 (Track A, Python package + CLI + docs; a few site test strings) does two sequenced things: (1) renames the component ohbm2026→abstractatlas (package src/ohbm2026/→src/abstractatlas/ via git mv + import rewrite across ~86 files) and the CLI ohbmcli→**aacli** (module form python -m abstractatlas.cli; legacy ohbm-* scripts→aa-* or dropped), with ohbmcli/import ohbm2026 kept as labeled deprecation shims for one cycle; and (2) generalizes ingestion into a pluggable architecture — a NEW src/abstractatlas/ingest/ subpackage with an Ingestor ABC (pull→normalize→validate), a runtime-discoverable registry (never a hardcoded source list — CA-007), and a standardized LinkML ingest schema contracts/ingest-schema.linkml.yaml (IngestedDocument core + ConferenceDocument/LiteratureDocument extensions). The two existing sources are wrapped (not rewritten) as the first ingestors: ohbm-2026 (conference, wraps fetch/stage.py+assets.normalize_abstract) and neuroscape-pubmed (literature, wraps the NeuroScape normalization) — byte-identical outputs, zero downstream-stage change (SC-003/SC-004). CRITICAL constraint: data is preserved — all on-disk paths, state-keys, and published data-package names (incl. the historical ohbm2026.parquet, the /ohbm2026/ route, SITE_MODE='ohbm', OHBM2026_* CI vars) are NOT renamed (source-data identity ≠ component identity), so no data regen and no site re-publish (byte-identical, FR-004). Scope is FOUNDATION ONLY: arXiv/bioRxiv/medRxiv + new-conference ingestors are explicit non-goals (follow-on specs the architecture enables). Verified: full unittest suite green under new names (SC-002) + fixture artifacts identical to pre-rename (SC-001) + new failing-first tests for registry/schema-validation/port- fidelity. Companions: research.md, data-model.md, contracts/{rename-map,ingestor-interface,ingest-schema.linkml.yaml,cli-aacli}.md, quickstart.md. NOTE for future sessions: after this ships, the package is abstractatlas and the CLI is aacli — update muscle memory (ohbmcli still works via a deprecation shim slated for removal).

The immediately-prior Stage 26 plan is at specs/026-neuroscape-year-density/plan.md. Stage 26 (Track A, UI-only, site/, neuroscape mode only) makes the NeuroScape scatter backdrop's density year-aware when a year filter is active: each year in the window contributes dots ∝ √(that year's count in the filtered set) — compressed-proportional, tempering the 1999→2023 volume growth without flattening it — with each year's dots picked by ascending lod_level so the selection stays a shape-preserving spatial cover (reuses the existing per-point LOD rank; no new data). Fixes the reported problem: sliding a fixed-width year window showed 10×+ dot-density swings (sparse early years, dense recent). Client-side only: the sole change is a NEW pure module site/src/lib/atlas/year_density.ts (calibrate / yearQuota / yearAwareSample, vitest-tested) wired into the existing scatterBackdropForMap derivation (+page.svelte:840) — full-span (no year filter) keeps today's lod_level ≤ cap path UNCHANGED (FR-004), so the default landing view, atlas-root, and /ohbm2026/ are untouched. Calibration constant k = targetBudget / Σ√count_y (targetBudget ≈ the current full-span base-sample size) is computed once from the loaded corpus (runtime-discovered, never a hardcoded 1999–2023 table). No change to filtering semantics (result list/counts still report true filtered totals — FR-006); UmapPanel unchanged; backdropFull viewport detail is already year-filtered so zoom respects the window (FR-008). No corpus/ pipeline rerun and no data-package re-publish (byte-identical preserved). Verified failing-first via vitest run (9 sampler cases in contracts/year-density-sampler.md) + a neuroscape_year_density e2e (dot-count-band B1–B6 in contracts/render-integration.md). Companions: research.md, data-model.md, contracts/{year-density-sampler,render-integration}.md, quickstart.md.

The immediately-prior Stage 25 plan is at specs/025-neuroscape-year-range-slider/plan.md. Stage 25 (Track A, UI-only, site/, neuroscape mode only) replaces the NeuroScape atlas "Years" facet's two free-text number boxes (From/To in NeuroscapeFacets.svelte:126–151) with a dual-handle range slider that (a) sets the start/end endpoints by dragging each handle and (b) shifts the whole selected window — width preserved — by dragging the band between the handles (FR-004). No new npm dependency: the slider is built in-house from a <div> track + two role="slider" thumbs + a draggable fill, driven by Pointer Events (mouse+touch one path) + arrow/Home/End keyboard (research R1). The window math (clamp, setStart/setEnd, moveWindow with width-preserving clamp, resolveSpan for crossed handles, isFullSpan, toFilter/fromFilter) is extracted into a pure, vitest-tested module NEW site/src/lib/filter/year_range.ts (co-located with the existing $lib/filter normalize); the NEW site/src/lib/components/YearRangeSlider.svelte owns only DOM/gesture concerns. The existing state plumbing is REUSED UNCHANGED: +page.svelte keeps filterMinYear/filterMaxYear: number|null (null ⇒ unbounded / full-span = inactive, FR-007) and the derived yearBounds {lo,hi} (min/max year over the loaded backdrop — runtime-discovered, never hardcoded 1999–2023, CA-007); NeuroscapeFacets keeps emitting the same update payload {cluster_ids,min_year,max_year}, so the downstream year filter (yLo = filterMinYear ?? yearBounds.lo etc.) and the activeCount/Clear logic are untouched (no filtering-semantics regression, SC-003). Verified failing-first via vitest run unit tests (9 helper cases in contracts/year-range-helper.md) + a Playwright neuroscape_year_slider e2e (U1–U6 in contracts/slider-ui.md). Client-side only; no corpus/ pipeline rerun and no data-package re-publish (byte-identical preserved). Companions: research.md, data-model.md, contracts/{year-range-helper,slider-ui}.md, quickstart.md.

The immediately-prior Stage 24 plan is at specs/024-fix-ios-safari-load/plan.md. Stage 24 (Track A, UI-only, site/) fixes the /ohbm2026/ atlas failing to load on iPhone Safari (blank screen / endless spinner). Root cause is NOT a JS build-target issue (the SvelteKit default modules target parses on modern iOS) but a non-resilient bootstrap that hides WebKit-specific failures: (R1) there is no +error.svelte and +page.svelte only sets loaded=true AFTER its awaits (:278,308), so any thrown await leaves the render stuck on {#if !loaded}Loading… (:2589) — the amplifier; (R2) OHBM mode mounts ~3 simultaneous auto-rotating WebGL contexts (2D scattergl + 3D scatter3d + HUD) via show3dPane = mode==='ohbm' || hasWebGL (UmapPanel.svelte:232)

  • autoRotate = mode==='ohbm' (:247), which iOS WebKit's tight GL-context cap kills — the trigger; (R3) the ONNX/WASM semantic worker is warmed eagerly on every load (+page.svelte:317 warmSemantic()) adding memory pressure; (R4) the 25 MB parquet is buffered ~2x and decoded on the main thread (loader.ts). Fix order = visibility first (add +error.svelte + try/catch → explicit failed(reason) state), then a runtime DeviceCapability gate (NEW site/src/lib/device/capability.ts, detected at runtime per CA-007 — NOT a UA/iOS allow-list) that on mobile/low-budget mounts 2D-only, disables auto-rotate, and defers 3D behind an explicit toggle, then lazy/on-demand semantic warm, then (contingent) off-main-thread decode. Verified via a failing-first Playwright WebKit/iPhone load check + vitest run unit tests for the capability gate and load-state machine; physical-iPhone Web-Inspector pass is the final sign-off. Client-side only; no corpus/pipeline rerun and no data-package re-publish expected (byte-identical guarantee preserved). Companions: research.md, data-model.md, contracts/{error-visibility,mobile-rendering,lazy-semantic-warm,load-verification}.md, quickstart.md.

The immediately-prior Stage 23 plan is at specs/023-atlas-research-dimensions/plan.md. Stage 23 (Track A, src/ohbm2026/ui_data/) ingests four externally-computed research- classification dimensions (focus, research_modality, theory_scope, epistemic_basis) from an operator-supplied abstracts.detail.json (Mario's NeuroScape dimension analysis, keyed by Oxford submission id) and surfaces them in the /ohbm2026/ atlas as new per-abstract computed insights and filterable facets. Integration is a left-join inside the Stage 6 UI-data builder: ui_data/abstracts.iter_abstracts already yields each exported record with its abstract_id (the join key) + a facets dict, so the 4 dimensions become 4 more list-of-string entries in facets and ride the existing parquet STRUCT / manifest catalog / generic site facet loader. Clarified: the export is authoritative — a left-join that NEVER pulls in abstracts absent from the export (unmatched file entries are counted/logged via unmatched_in_file); surfaced as BOTH facets AND detail chips, no scatter color-overlay. New module ui_data/dimensions.py (load_research_dimensions, compute_dimension_coverage, typed DimensionInputError(Stage6BuildError)); new optional --dimensions PATH build flag (opt-in — absent ⇒ 4 empty facets); provenance recorded in the manifest's research_dimensions block (basename+sha256+per-dim coverage, no abs paths). Site edits are narrow: 4 keys in facets.ts (FACETS_FROM_BLOCK / FACET_KEYS_ORDERED / FACET_LABELS — sidebar + valuesFor are generic) and 4 chip blocks in DetailPanel.svelte's computed-insights zone. facets LinkML slot is already range: Any (doc note only); atlas- root/neuroscape untouched (they don't consume ohbm2026.parquet facets). Companions: research.md, data-model.md, contracts/{dimension-input,facets-and-detail}.md, quickstart.md.

The immediately-prior Stage 22 plan is at specs/022-r2-edge-caching/plan.md (shipped via PR #62). Stage 22 (Track A, src/ohbm2026/atlas_hosting/) makes the R2 data host (aadata.cirrusscience.org) actually edge-cached — the dashboard showed a 0% cache rate. Key finding: the uploader ALREADY sets Cache-Control: public, max-age=31536000, immutable on every object via boto3 upload_file ExtraArgs (single + multipart, r2_client.py:185), so the real fix is the Cloudflare host cache rule (R2 custom domains aren't edge-cached by default; Range/206 often bypasses) plus verification. Repo deliverables: extend compare.py/compare-data-hosting to capture cf-cache-status/age/cache-control for full + range (cold→warm double request, flag BYPASS, assert range byte-parity); record the cache policy in the upload manifest (provenance, FR-010); a regression test for the upload ExtraArgs. The host cache rule is documented config (contracts/cloudflare-cache-rule.md) — manual by default, optional Cloudflare-API automation behind a .env token. Production channel stays Dropbox (no change). Companions: research.md, data-model.md, contracts/{cache-evidence-report,cloudflare-cache-rule}.md, quickstart.md.

The immediately-prior Stage 21 plan is at specs/021-atlas-cart-lasso/plan.md (shipped via PR #61). Stage 21 was a UI-only change in site/ (no Python, no parquet rebuild): rename "Saved only" → "Cart only" on all three sibling builds and add it to atlas-root + neuroscape; switch filter composition from cart-dominant to a true intersection (search ∩ lasso ∩ facets ∩ cart-only), changing the OHBM home behavior so all three sites match; add a facet-style cross-site warning when saved items aren't present in the current site's loaded corpus (hidden-count discovered by corpus-index membership, not a hardcoded kind→site table); and fix the scatter highlight visibility (per the 2026-06-01 clarification): in 2D keep the existing selectedpoints mechanism and just cap the unselected opacity below the selected opacity when a selection is active (the highlight currently washes out at zoom because applyAtlasZoomOpacity ties unselected opacity to the brightening base); in 3D add the selection highlight (today absent) via an in-place Plotly.restyle of a per-point opacity/colour array — no Plotly.react, no trace-count change (avoids the WebGL-context leak plotly.js#6365), reusing the cheap-restyle pattern from the 3D-focus fix (commit dba3d7cf). Core touchpoints: site/src/routes/+page.svelte (composition + per-mode Cart-only state + highlight-set feed), site/src/lib/components/UmapPanel.svelte + site/src/lib/atlas/opacity.ts (2D unselected-opacity gap + 3D restyle highlight), the two browse panels (expose matched-id set + warning), and NEW pure helpers site/src/lib/selection/{compose,cart_scope}.ts. Companions: research.md, data-model.md, contracts/{selection-composition,cart-only-filter,selection-highlight}.md, quickstart.md.

The immediately-prior Stage 20 plan is at specs/020-cloudflare-r2-migration/plan.md. Stage 20 publishes the UI data bundle (the four atlas-package parquets) to Cloudflare R2 (S3-compatible) under content-addressed, immutable keys (<sha256>/<filename>) via a new local ohbmcli upload-atlas-package command, registers a new R2 channel in the existing OHBM2026_UI_DATA_PACKAGE_URLS registry (Dropbox stays the production default — cutover is deferred), and adds ohbmcli compare-data-hosting for byte-parity / CORS / Range evidence. Companion artefacts under the same directory: research.md, data-model.md, contracts/cli-upload-atlas-package.md, contracts/cli-compare-data-hosting.md, contracts/r2-storage-layout.md, contracts/upload-manifest.schema.json, contracts/comparison-report.schema.json, and quickstart.md. The site loader and .github/scripts/resolve-data-channel.sh are UNCHANGED — R2 URLs flow through normaliseDropboxUrl untouched (only Dropbox hosts are rewritten); new code lives in src/ohbm2026/atlas_hosting/ with a Stage20Error exception subtree and an optional r2 = ["boto3"] extra.

The immediately-prior Stage 19 (specs/019-neuroscape-semantic-search/plan.md; companions research.md, data-model.md, contracts/{parquet-schemas,cli-build-atlas-package,search-ranking-pipeline,atlas-root-search-ui}.md, quickstart.md) added the deferred semantic-search lane for /neuroscape/ plus a cross-conference search bar on atlas-root that ranks OHBM 2026 + NeuroScape together, reusing the existing /ohbm2026/ Xenova/MiniLM-L6-v2 worker with a cluster-routed + KNN-expansion pipeline that bounds per-query cost (~4 MB cold-cache range fetch instead of the full 50 MB sidecar).

Per-table range fetch — never download a whole envelope parquet. The nested-envelope parquets (ohbm2026.parquet, neuroscape.parquet, atlas.parquet) are written with row_group_size=1 (_OUTER_PARQUET_KWARGS in atlas_package/parquet_writer.py) specifically so a browser can fetch ONE inner table via hyparquet predicate pushdown: a { table_name: { $eq: '<table>' } } filter skips every other row group via row-group stats, so only that table's blob crosses the network (e.g. ~268 KB for cluster_centroids, not the 97 MB file). The flat neuroscape_vectors.parquet sidecar uses the same trick on cluster_id. When a browser surface needs one table from a sibling parquet, range-fetch it — do not download or duplicate the whole file.

Geometry / identity split (spec 019 follow-up). neuroscape.parquet keeps geometry OUT of the articles table: articles carries only identity/search columns (pubmed_id, title, year, cluster_id), while a standalone coords table holds (pubmed_id, cluster_id, umap_2d, umap_3d). A self-contained backdrop_decimated table (pubmed_id, cluster_id, umap_2d, umap_3d, title, year) is the default landing scatter sample. The TS loader folds coords → articles after the neuroscape full-GET so existing render code (a.umap_2d / a.umap_3d) is unchanged. atlas.parquet carries ONLY manifest + ohbm_overlay (the OHBM→NeuroScape projection — the one thing impossible to derive from either sibling alone). It deliberately carries NO cluster_centroids, NO clusters, NO backdrop, and NO cross_pointers table: atlas-root range-fetches the cluster legend (loadClustersFromNeuroscape), the landing backdrop (loadBackdropDecimatedFromNeuroscape), and the centroids (loadClusterCentroidsFromNeuroscape) from the sibling neuroscape.parquet (the sibling URL is already known for cache-prefetch

  • drift detection); permalinks are derived from (kind, id) in the browser. The atlas-root backdrop ships no KNN neighbour graph, so the ranker uses an adaptive seed count (max(topK, TOP_K_SEEDS) when knnIndex is empty). NeuroScape minilm vectors embed title+abstract reusing the OHBM seq-length window + chunk_mean_pool. NeuroScape v1.0.1 build inputs live at data/inputs/neuroscape-source/v101/ + data/inputs/neuroscape/; ohbmcli build-atlas-package runs locally end-to-end (built package under data/outputs/atlas-package__*/).

The earlier Stage-15 baseline (the three-sibling-deployment architecture

  • three-parquet data layout this spec extends) is documented in specs/015-neuroscape-context/plan.md — that plan remains the canonical reference for atlas-root, /ohbm2026/, and /neuroscape/ structurally; spec 019 only adds the semantic search lane to the existing surfaces.

Architecture — three sibling deployments on one gh-pages host: abstractatlas.brainkb.org/ (atlas-root mode; binary "Show OHBM 2026 overlay" toggle on a NeuroScape PubMed backdrop colour-coded by cluster); /ohbm2026/ (unchanged); /neuroscape/ (new; full ~600K-article PubMed corpus with search + detail). One SvelteKit project, three build modes via SITE_MODE env + BASE_PATH.

Three-parquet data layout: ohbm2026.parquet (renamed from data.parquet, content-identical), neuroscape.parquet (full NeuroScape 1999–2023 corpus — articles identity/search + coords geometry + self-contained backdrop_decimated + cluster table + k=20 neighbours + lexical search index + cluster_centroids), atlas.parquet (slimmed to ONLY manifest + ohbm_overlay, the OHBM→NeuroScape projection; cluster legend / backdrop / centroids are range-fetched from the neuroscape sibling, never duplicated). atlas.parquet's build_info embeds the two sibling state-keys for drift detection — the browser-side loader surfaces a visible error banner on mismatch, never a silent partial scatter.

New Python orchestrator ohbmcli build-atlas-package reads the NeuroScape v1.0.1 release (HDF5 shards + CSVs + checkpoint, same inputs as scripts/derive_neuroscape_centroids.py), fits a deterministic 2D + 3D UMAP on Stage-2 vectors (seed=0, n_neighbors=30, min_dist=0.10, metric=cosine), projects OHBM 2026 abstracts via umap.transform using the existing voyage_stage2_published recipe, and emits neuroscape.parquet + atlas.parquet. Two caches (UMAP fit, per-abstract projection) make a second invocation byte-identical and <60s.

Constitution-critical guarantees: byte-identical /ohbm2026/ build output before vs after this change (FR-022 / SC-008, CI- enforced); precise typed exceptions across the new Stage15Error subtree for every error path (FR-026); link-checked PubMed/DOI/citation URLs at build time block the deploy (FR-024 / SC-006); no new credentials needed; all new artefact roots are gitignored.

Previous-stage plans:

  • Stage 14 poster-id navigator: specs/014-poster-id-nav/plan.md (extended the existing <SearchBar> with an id: operator and autocomplete dropdown over the in-memory abstractsByPosterId map; pure client-side, no parquet rebuild). Shipped in PR #35.
  • Stage 12 book layout polish + acknowledgments + permalink UX: specs/013-book-layout-polish/plan.md. Six bundled stories: acknowledgments on the permalink page, brief-preview UX with show-more, JPEG-q90 @ 150 DPI figure normalisation, 3-column TOC longtable, author-index letter buckets, tight book margins. Shipped in PR #34 (Stage 12.2 also closed 12 real-corpus LaTeX failures + wired KaTeX math + Unicode super/subscript in section bodies).
  • Stage 11.1 book PDF + standby + DOCX retire + CI telemetry: specs/012-stage11-followups/plan.md (per-abstract parallel + cached PDF; standby INT8 schema; DOCX retirement; CI telemetry on the deploy-ui PR-association retry loop; Stage 1's state_key renamed to fetch_state_key). Shipped in PR #31.
  • Stage 11 book of abstracts: specs/011-abstracts-book/plan.md (deterministic ohbmcli book CLI; markdown-canonical intermediate via pandoc → xelatex/Tectonic for PDF + pandoc-native DOCX writer; optional --style tufte; figure pre-resize at --max-image-width; authoritative standby times from FINAL OHBM 2026 listing CSV with UI display + facet + cart-restore deep links). Shipped via PRs #26-#30. Real-corpus PDF on monolithic compile didn't ship; Stage 11.1 supersedes that path with per-abstract parallel + caching.
  • Stage 10 data export redesign: specs/010-export-redesign/plan.md (single-file Parquet with row-group-per-table layout; LinkML-tight schema eliminating every range: Any; magic-byte sniff dispatch via the in-browser hyparquet decoder; poster_id int16 as the sole user-facing identifier replacing Oxford submission_id; cross-conference Phase 5 deferred — conference outputs frozen post-build, cross-conf linking ships as a UI-side artefact when a second conference is ingested). Shipped in PR #20.
  • Stage 9 conference subpath rework: specs/009-conference-subpath/plan.md (every OHBM-2026 surface under /ohbm2026/; static meta-refresh root-redirect island via <meta http-equiv="refresh"> + JS location.replace because gh-pages can't serve real 301s; legacy URLs not preserved per Q2; PR previews mirror production at /pr-<N>/ohbm2026/; primary mechanism is SvelteKit kit.paths.base with a BASE_PATH env override). Shipped in PR #19.
  • Stage 6 UI rewrite (US1–US8): specs/008-ui-rewrite/plan.md (static SvelteKit site on gh-pages; site/ + src/ohbm2026/ui_data/; typo-tolerant lexical + semantic search; 3D UMAP + lasso; cart + email; guided tour; About + link-checked references; 10+ PRs #9–#18 across spec ship-out + post-spec UX (search operators + badge clarification) + wrap-up).
  • Stage 5 package reorganization: specs/007-package-reorg/plan.md (collapsed enrichment.py into enrich/{text, markdown_render}.py; parked layout/; split ui.py into ui/{payload, cli}.py; consolidated load_json/write_json into ohbm2026/util/json_io.py).
  • Stage 4 analysis & annotation: specs/006-analysis-annotation/plan.md (canonical ohbmcli analyze-matrix producing 48 bundles + canonical rollup annotations__<state-key>.{parquet,sqlite}; joblib-parallel orchestrator; hybrid spaCy + c-TF-IDF + LLM-grouping topics).
  • Stage 3 embeddings matrix: specs/005-embeddings-matrix/plan.md (per-component embeddings × 5 models with token-level chunking; state-key keyed bundle directories; canonical compose_recipe(...) composer).
  • Stage 2.1 production wiring: specs/004-enrich-production-wiring/plan.md (gpt-5.4-mini figures+claims, agentic Responses API, ECO v1 annotation, OpenAlex async references; T058 corpus state_key f0c51e80dc0e).
  • Stage 2 enrichment scaffolding: specs/003-enrich-abstracts/plan.md (SQLite+zlib storage; the orchestrator surface Stage 2.1 wires production runners into).
  • Stage 1 fetch-abstracts rewire: specs/002-rewire-pipeline/plan.md (canonical reference for the per-stage contract pattern).