Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/scripts/resolve-data-channel.sh
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@
# Dropbox shared links OR Cloudflare R2 (e.g. https://aadata.cirrusscience.org/
# <sha256>/<file>.parquet). This script and the site loader are unchanged — the
# URL is opaque to both; loader.ts only rewrites Dropbox hosts, so R2 URLs pass
# through verbatim. `ohbmcli upload-atlas-package` emits a ready-to-paste
# through verbatim. `aacli upload-atlas-package` emits a ready-to-paste
# channel entry in exactly the shape above.

set -euo pipefail
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/deploy-ui.yml
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ on:
- 'site/**'
- '.github/workflows/deploy-ui.yml'
- 'specs/008-ui-rewrite/contracts/references.yaml'
- 'src/ohbm2026/ui_data/link_check.py'
- 'src/abstractatlas/ui_data/link_check.py'
workflow_dispatch:
inputs:
target:
Expand Down Expand Up @@ -191,7 +191,7 @@ jobs:
# the deploy when a citation 4xx's or times out (T086 / T088).
# Runs on both targets — a broken reference shouldn't reach
# sandbox either.
run: PYTHONPATH=src python -m ohbm2026.ui_data.link_check specs/008-ui-rewrite/contracts/references.yaml
run: PYTHONPATH=src python -m abstractatlas.ui_data.link_check specs/008-ui-rewrite/contracts/references.yaml

# Resolve the parquet URLs from the keyed registry variable, using the
# channel this ref declares in site/data-channel.json. The same
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/pr-preview.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ on:
- 'site/**'
- '.github/workflows/pr-preview.yml'
- 'specs/008-ui-rewrite/contracts/references.yaml'
- 'src/ohbm2026/ui_data/link_check.py'
- 'src/abstractatlas/ui_data/link_check.py'
workflow_dispatch:

permissions:
Expand Down Expand Up @@ -71,7 +71,7 @@ jobs:
# Same check that gates the production deploy in deploy-ui.yml —
# mirrored here so PR previews surface a broken reference BEFORE
# the merge instead of failing at deploy time.
run: PYTHONPATH=src python -m ohbm2026.ui_data.link_check specs/008-ui-rewrite/contracts/references.yaml
run: PYTHONPATH=src python -m abstractatlas.ui_data.link_check specs/008-ui-rewrite/contracts/references.yaml

# Resolve the parquet URLs from the keyed registry variable, using the
# channel this PR branch declares in site/data-channel.json. A PR can
Expand Down
2 changes: 1 addition & 1 deletion .specify/feature.json
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
{
"feature_directory": "specs/026-neuroscape-year-density"
"feature_directory": "specs/027-abstractatlas-rename-ingestors"
}
59 changes: 47 additions & 12 deletions CLAUDE.md

Large diffs are not rendered by default.

146 changes: 73 additions & 73 deletions README.md

Large diffs are not rendered by default.

12 changes: 6 additions & 6 deletions docs/abstracts-book-plan.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ the per-abstract pipeline introduced in Stage 11.1 — each abstract
pandoc-renders in parallel with content+toolchain caching, then a
two-pass assembly attaches a page-numbered author index.

DOCX export was **retired in Stage 11.1 US3** (`ohbmcli book --format docx`
DOCX export was **retired in Stage 11.1 US3** (`aacli book --format docx`
exits 2 with a pointer at `--format md` and `--format pdf`). The
real-corpus DOCX hit 2.8 GB even after figure resizing — too large
for Word to open — and the markdown bundle + PDF cover every use
Expand All @@ -39,7 +39,7 @@ live under `specs/011-abstracts-book/`:
optional Tufte styling).
- `data-model.md` — three layers (Stage-1 inputs → markdown-bearing
in-memory model → outputs with figure-filename contract).
- `contracts/cli.md` — `ohbmcli book` flags, error path table,
- `contracts/cli.md` — `aacli book` flags, error path table,
known limitations.
- `quickstart.md` — operator runbook for install + first run.

Expand All @@ -53,13 +53,13 @@ uv pip install --python .venv/bin/python ".[abstracts_book]"
brew install pandoc tectonic

# Markdown-only first run (no system deps needed):
PYTHONPATH=src .venv/bin/python -m ohbm2026.cli book --format md
PYTHONPATH=src .venv/bin/python -m abstractatlas.cli book --format md

# Full PDF run:
PYTHONPATH=src .venv/bin/python -m ohbm2026.cli book --format pdf --sort poster_id
PYTHONPATH=src .venv/bin/python -m abstractatlas.cli book --format pdf --sort poster_id

# All three formats + Tufte typography for the PDF:
PYTHONPATH=src .venv/bin/python -m ohbm2026.cli book --format all --sort poster_id --style tufte
PYTHONPATH=src .venv/bin/python -m abstractatlas.cli book --format all --sort poster_id --style tufte
```

Outputs land at `data/outputs/book/book__<state-key>/`. Figure
Expand Down Expand Up @@ -108,7 +108,7 @@ Operator flags:
Single-abstract debug (re-render one chunk in isolation, populating
the cache):
```bash
PYTHONPATH=src .venv/bin/python -m ohbm2026.book.render_per_abstract \
PYTHONPATH=src .venv/bin/python -m abstractatlas.book.render_per_abstract \
--poster-id 0042 --style plain
```

Expand Down
8 changes: 4 additions & 4 deletions docs/cllm-claims-plan.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,8 @@ experience overwhelming.

## Execution

1. Run `ohbmcli analyze-figures` first if you want cached figure notes included in
1. Run `aacli analyze-figures` first if you want cached figure notes included in
the claim manuscript.
2. Run `ohbmcli extract-claims` to populate or resume the claim cache.
3. Run `ohbmcli enrich` to merge claim extraction into the enriched abstract data.
4. Run `scripts/build_ui_data.py` (Stage 6) so the SvelteKit site's data package picks up the new fields. (The legacy `ohbmcli export-ui` / `build-ui` commands have been removed; this plan predates Stage 2.1 + Stage 6 and is kept for historical reference only.)
2. Run `aacli extract-claims` to populate or resume the claim cache.
3. Run `aacli enrich` to merge claim extraction into the enriched abstract data.
4. Run `scripts/build_ui_data.py` (Stage 6) so the SvelteKit site's data package picks up the new fields. (The legacy `aacli export-ui` / `build-ui` commands have been removed; this plan predates Stage 2.1 + Stage 6 and is kept for historical reference only.)
2 changes: 1 addition & 1 deletion docs/clustering-analysis-plan.md
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,7 @@ Runtime checks on real data:

## Status

- [x] Added `ohbmcli cluster-benchmark`
- [x] Added `aacli cluster-benchmark`
- [x] Implemented method sweep across `kmeans`, `agglomerative-ward`, `agglomerative-average`, `gaussian-mixture`, and `birch`
- [x] Added label-independent ranking using silhouette, Davies-Bouldin, Calinski-Harabasz, inter/intracluster distance ratio, and cluster-balance terms
- [x] Added test coverage for method execution, metric computation, ranking, CLI wiring, and output writing
Expand Down
32 changes: 16 additions & 16 deletions docs/ohbm26-plan.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,36 +37,36 @@ This is the operational planning document for the repository.

## Architecture

- `src/ohbm2026/graphql_api.py`
- `src/abstractatlas/graphql_api.py`
- GraphQL requests, batching, retries, env loading, author/submission fetches
- `src/ohbm2026/assets.py`
- `src/abstractatlas/assets.py`
- local asset indexing, figure URL extraction, reuse-aware downloads, JSON-only asset refresh, and ingest CLI
- `src/ohbm2026/enrichment.py`
- `src/abstractatlas/enrichment.py`
- author export, HTML-to-markdown conversion, section mapping, local Qwen figure analysis, enriched abstract assembly, and phase-2 CLI
- `src/ohbm2026/neuroscape.py`
- `src/abstractatlas/neuroscape.py`
- stage-one embeddings, semantic graph generation, clustering, and NeuroScape handoff manifest
- `src/ohbm2026/cli.py`
- unified `ohbmcli` command
- `src/abstractatlas/cli.py`
- unified `aacli` command

## Unified CLI

- `ohbmcli ingest`
- `aacli ingest`
- fetch abstracts and figure assets from the API
- `ohbmcli refresh-assets`
- `aacli refresh-assets`
- rebuild `local_assets` from the local JSON database without rerunning abstract extraction
- `ohbmcli authors`
- `aacli authors`
- export author metadata
- `ohbmcli enrich`
- `aacli enrich`
- build enriched abstracts from local databases
- `ohbmcli analyze-figures`
- `aacli analyze-figures`
- analyze local figures with Ollama
- `ohbmcli embed-minilm`
- `aacli embed-minilm`
- generate local MiniLM embeddings
- `ohbmcli embed-voyage`
- `aacli embed-voyage`
- generate Voyage embeddings
- `ohbmcli semantic-analysis`
- `aacli semantic-analysis`
- build the semantic graph and semantic cluster summaries from any local embedding bundle
- `ohbmcli write-manifest`
- `aacli write-manifest`
- write the NeuroScape handoff manifest

## Execution Order
Expand Down Expand Up @@ -226,7 +226,7 @@ Checks:
- [x] Create a local MiniLM embedding pipeline and neighbor database.
- [x] Create a purely semantic graph-and-cluster pipeline for the single-year corpus.
- [x] Split the codebase into API, asset, enrichment, and embedding modules.
- [x] Add a unified `ohbmcli` entrypoint.
- [x] Add a unified `aacli` entrypoint.
- [ ] Complete a full cached figure-analysis sweep with local `qwen3.5:35b`.
- [ ] Validate the NeuroScape retraining path against the PR-based workflow and artifact requirements.

Expand Down
28 changes: 14 additions & 14 deletions docs/per-stage-pattern.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@
Every pipeline stage in this repo (Stage 1 fetch, Stage 2 enrich,
Stage 3+ analyses, …) satisfies the same six contracts. This doc
defines them and points at two canonical reference implementations
— **Stage 1 (`src/ohbm2026/fetch_stage.py`)** for a single-fetch
stage, and **Stage 2 (`src/ohbm2026/enrich_stage.py`)** for a
— **Stage 1 (`src/abstractatlas/fetch_stage.py`)** for a single-fetch
stage, and **Stage 2 (`src/abstractatlas/enrich_stage.py`)** for a
multi-component stage — for each.

A new stage author should be able to read this page plus
Expand Down Expand Up @@ -111,7 +111,7 @@ Stage 2 reference:
What failures the stage surfaces loudly, with what typed cause,
and at what exit code.

- Typed exception hierarchy in `ohbm2026.exceptions` rooted at a
- Typed exception hierarchy in `abstractatlas.exceptions` rooted at a
cross-stage `OhbmStageError(RuntimeError)`. Stage 1 subclass tree:
`Stage1Error` → `SchemaContractError`, `CheckpointError`,
`FigureFailureError`. Stage 2 subclass tree: `Stage2Error` →
Expand Down Expand Up @@ -219,36 +219,36 @@ Stage 2 shipped the per-stage pattern scaffolding with
(`specs/004-enrich-production-wiring/`) wires production runners
through those seams and keeps the orchestrator surface unchanged:

- **Figures**: `src/ohbm2026/stage2_figures.py` —
- **Figures**: `src/abstractatlas/stage2_figures.py` —
`run_figure_component(abstract, model_id, flex_enabled, client, cwd)`
groups all figures of one abstract into a single OpenAI Responses
API call with manuscript text as context. In-memory JPEG-q85
compression at 1024 px (`compress_image`) + a four-field local
quality probe (`src/ohbm2026/image_quality.py`:
quality probe (`src/abstractatlas/image_quality.py`:
`laplacian_variance`, `mean_brightness`, `native_max_dim`,
`compression_ratio`) populate `local_quality_estimate` on every
record. Pydantic `FigureInterpretationResponse` + a fixed
`model_quality_estimate` enum validate the response server-side.
- **Claims**: `src/ohbm2026/stage2_claims.py` —
- **Claims**: `src/abstractatlas/stage2_claims.py` —
`run_claims_component(abstract, ..., figure_interpretations)`
issues a single agentic Responses API call with three function
tools registered (`verify_source_quote`, `lookup_eco_code`,
`dedupe_check`). The model orchestrates extract → verify →
annotate → dedupe internally. Post-response validation drops
claims whose source quote isn't a substring of the manuscript
OR whose `evidence_eco_codes` include off-vocabulary ECO IDs.
- **References**: `src/ohbm2026/stage2_references.py` is a thin
- **References**: `src/abstractatlas/stage2_references.py` is a thin
adapter to the existing `openalex.collect_reference_metadata`
async pipeline.
- **Flex tier**: `src/ohbm2026/flex_tier.py` —
- **Flex tier**: `src/abstractatlas/flex_tier.py` —
`call_with_flex_fallback(...)` wraps each LLM call with a 1-flex +
1-standard retry budget; defaults to flex ON; per-component
disable flags `--no-flex-figures` / `--no-flex-claims`.
- **Concurrency**: `enrich_stage._run_component_concurrent`
fans out per-abstract work across a `ThreadPoolExecutor`
(default 30 in flight). Per-component caps via
`--concurrency-figures` / `--concurrency-claims`.
- **ECO v1 vocabulary** lives at `src/ohbm2026/data/eco_top_codes.json`
- **ECO v1 vocabulary** lives at `src/abstractatlas/data/eco_top_codes.json`
(committed source). The claims component's `lookup_eco_code`
tool draws from it without network access.
- **Provenance extensions**: per-component `flex_tier_enabled`,
Expand All @@ -269,7 +269,7 @@ When you write Stage N:
2. **Author the test file first** (Principle IV). One test per
contract element, organized into a class per contract.
3. **Implement the orchestrator** in
`src/ohbm2026/<stage_name>.py`. Use Stage 1's `fetch_stage.py`
`src/abstractatlas/<stage_name>.py`. Use Stage 1's `fetch_stage.py`
as the layout reference. Share helpers via `artifacts.py`,
`exceptions.py`, and `schema_diff.py` where possible.
4. **Add a CLI subcommand** in `cli.py` that delegates to your
Expand All @@ -280,12 +280,12 @@ When you write Stage N:

## Common Helpers

- `ohbm2026.artifacts.build_*_path(state_key)` — path derivation
- `abstractatlas.artifacts.build_*_path(state_key)` — path derivation
under gitignored roots.
- `ohbm2026.artifacts.build_dependency_basis` + `build_state_key` —
- `abstractatlas.artifacts.build_dependency_basis` + `build_state_key` —
deterministic state key from input fingerprint.
- `ohbm2026.exceptions.*` — typed exception hierarchy.
- `ohbm2026.schema_diff.*` — schema-diff / discovery primitives
- `abstractatlas.exceptions.*` — typed exception hierarchy.
- `abstractatlas.schema_diff.*` — schema-diff / discovery primitives
(reusable across stages that talk to GraphQL).
- The constitution lint at
`.specify/scripts/bash/constitution-check.sh` catches the
Expand Down
6 changes: 3 additions & 3 deletions docs/poster-layout-optimizer-plan.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ Confirmed decisions for the first pass:

## Proposed Deliverables

- `src/ohbm2026/poster_layout.py`
- `src/abstractatlas/poster_layout.py`
- reusable loading, feature extraction, optimization, numbering, and scoring helpers
- `scripts/optimize_poster_layout.py`
- CLI entrypoint to assign posters to sessions and output a proposed numbering/layout
Expand Down Expand Up @@ -199,7 +199,7 @@ PYTHONPATH=src .venv/bin/python scripts/analyze_poster_layout.py \
--output data/poster_layout/analysis.json
```

If this matures well, we can later fold the same functionality into `ohbmcli`.
If this matures well, we can later fold the same functionality into `aacli`.

## Test Plan

Expand Down Expand Up @@ -227,7 +227,7 @@ If this matures well, we can later fold the same functionality into `ohbmcli`.
## Implementation Phases

- [x] Confirm session-assignment model and physical numbering assumptions
- [x] Add a poster-layout design module in `src/ohbm2026/poster_layout.py`
- [x] Add a poster-layout design module in `src/abstractatlas/poster_layout.py`
- [x] Add synthetic tests for constraints and scoring before implementing search
- [x] Implement poster feature loading and validation
- [x] Implement conflict and locality scoring
Expand Down
2 changes: 1 addition & 1 deletion docs/poster-sequencing-benchmark-plan.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ We can think of this explicitly as a weighted graph reordering problem:

## Why This Phase Now

The current semantic-path ordering logic is still based on greedy nearest-neighbor walks in [src/ohbm2026/poster_layout.py](/Users/satra/software/temp/ohbm2026/src/ohbm2026/poster_layout.py).
The current semantic-path ordering logic is still based on greedy nearest-neighbor walks in [src/abstractatlas/poster_layout.py](/Users/satra/software/temp/ohbm2026/src/abstractatlas/poster_layout.py).

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The link contains an absolute path /Users/satra/software/temp/ohbm2026/src/abstractatlas/poster_layout.py which points to a local directory of a specific user. It should be updated to a relative path within the repository so that the link works for all users and environments.

Suggested change
The current semantic-path ordering logic is still based on greedy nearest-neighbor walks in [src/abstractatlas/poster_layout.py](/Users/satra/software/temp/ohbm2026/src/abstractatlas/poster_layout.py).
The current semantic-path ordering logic is still based on greedy nearest-neighbor walks in [src/abstractatlas/poster_layout.py](../src/abstractatlas/poster_layout.py).


Relevant functions:

Expand Down
14 changes: 7 additions & 7 deletions docs/repo-review-2026-03-28.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ to navigate confidently.

## Strengths To Preserve

- the canonical pipeline is resumable and centered on `ohbmcli`
- the canonical pipeline is resumable and centered on `aacli`
- experiment immutability is explicit and documented
- test coverage is broad across pipeline, UI export, poster layout, and
experiment tooling
Expand All @@ -37,15 +37,15 @@ to navigate confidently.

Current hotspots:

- `src/ohbm2026/neuroscape.py`
- `src/abstractatlas/neuroscape.py`
- about `2951` lines, `106` functions
- `src/ohbm2026/poster_sequencing.py`
- `src/abstractatlas/poster_sequencing.py`
- about `2371` lines, `59` functions
- `src/ohbm2026/openalex.py`
- `src/abstractatlas/openalex.py`
- about `2279` lines, `76` functions
- `src/ohbm2026/poster_layout.py`
- `src/abstractatlas/poster_layout.py`
- about `2191` lines, `60` functions
- `src/ohbm2026/enrichment.py`
- `src/abstractatlas/enrichment.py`
- about `1437` lines, `59` functions

Why this matters:
Expand Down Expand Up @@ -95,7 +95,7 @@ Mitigation landed in this pass:
Recommended next step:

- introduce a `scripts/README.md` or eventually promote the most stable
organizer flows into explicit `ohbmcli` or package-level namespaces
organizer flows into explicit `aacli` or package-level namespaces

### P2: Canonical versus exploratory outputs still need stronger machine-readable boundaries

Expand Down
Loading
Loading