Skip to content
Open
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/scripts/resolve-data-channel.sh
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@
# Dropbox shared links OR Cloudflare R2 (e.g. https://aadata.cirrusscience.org/
# <sha256>/<file>.parquet). This script and the site loader are unchanged — the
# URL is opaque to both; loader.ts only rewrites Dropbox hosts, so R2 URLs pass
# through verbatim. `ohbmcli upload-atlas-package` emits a ready-to-paste
# through verbatim. `aacli upload-atlas-package` emits a ready-to-paste
# channel entry in exactly the shape above.

set -euo pipefail
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/deploy-ui.yml
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ on:
- 'site/**'
- '.github/workflows/deploy-ui.yml'
- 'specs/008-ui-rewrite/contracts/references.yaml'
- 'src/ohbm2026/ui_data/link_check.py'
- 'src/abstractatlas/ui_data/link_check.py'
workflow_dispatch:
inputs:
target:
Expand Down Expand Up @@ -191,7 +191,7 @@ jobs:
# the deploy when a citation 4xx's or times out (T086 / T088).
# Runs on both targets — a broken reference shouldn't reach
# sandbox either.
run: PYTHONPATH=src python -m ohbm2026.ui_data.link_check specs/008-ui-rewrite/contracts/references.yaml
run: PYTHONPATH=src python -m abstractatlas.ui_data.link_check specs/008-ui-rewrite/contracts/references.yaml

# Resolve the parquet URLs from the keyed registry variable, using the
# channel this ref declares in site/data-channel.json. The same
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/pr-preview.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ on:
- 'site/**'
- '.github/workflows/pr-preview.yml'
- 'specs/008-ui-rewrite/contracts/references.yaml'
- 'src/ohbm2026/ui_data/link_check.py'
- 'src/abstractatlas/ui_data/link_check.py'
workflow_dispatch:

permissions:
Expand Down Expand Up @@ -71,7 +71,7 @@ jobs:
# Same check that gates the production deploy in deploy-ui.yml —
# mirrored here so PR previews surface a broken reference BEFORE
# the merge instead of failing at deploy time.
run: PYTHONPATH=src python -m ohbm2026.ui_data.link_check specs/008-ui-rewrite/contracts/references.yaml
run: PYTHONPATH=src python -m abstractatlas.ui_data.link_check specs/008-ui-rewrite/contracts/references.yaml

# Resolve the parquet URLs from the keyed registry variable, using the
# channel this PR branch declares in site/data-channel.json. A PR can
Expand Down
2 changes: 1 addition & 1 deletion .specify/feature.json
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
{
"feature_directory": "specs/026-neuroscape-year-density"
"feature_directory": "specs/027-abstractatlas-rename-ingestors"
}
57 changes: 45 additions & 12 deletions CLAUDE.md

Large diffs are not rendered by default.

146 changes: 73 additions & 73 deletions README.md

Large diffs are not rendered by default.

12 changes: 6 additions & 6 deletions docs/abstracts-book-plan.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ the per-abstract pipeline introduced in Stage 11.1 — each abstract
pandoc-renders in parallel with content+toolchain caching, then a
two-pass assembly attaches a page-numbered author index.

DOCX export was **retired in Stage 11.1 US3** (`ohbmcli book --format docx`
DOCX export was **retired in Stage 11.1 US3** (`aacli book --format docx`
exits 2 with a pointer at `--format md` and `--format pdf`). The
real-corpus DOCX hit 2.8 GB even after figure resizing — too large
for Word to open — and the markdown bundle + PDF cover every use
Expand All @@ -39,7 +39,7 @@ live under `specs/011-abstracts-book/`:
optional Tufte styling).
- `data-model.md` — three layers (Stage-1 inputs → markdown-bearing
in-memory model → outputs with figure-filename contract).
- `contracts/cli.md` — `ohbmcli book` flags, error path table,
- `contracts/cli.md` — `aacli book` flags, error path table,
known limitations.
- `quickstart.md` — operator runbook for install + first run.

Expand All @@ -53,13 +53,13 @@ uv pip install --python .venv/bin/python ".[abstracts_book]"
brew install pandoc tectonic

# Markdown-only first run (no system deps needed):
PYTHONPATH=src .venv/bin/python -m ohbm2026.cli book --format md
PYTHONPATH=src .venv/bin/python -m abstractatlas.cli book --format md

# Full PDF run:
PYTHONPATH=src .venv/bin/python -m ohbm2026.cli book --format pdf --sort poster_id
PYTHONPATH=src .venv/bin/python -m abstractatlas.cli book --format pdf --sort poster_id

# All three formats + Tufte typography for the PDF:
PYTHONPATH=src .venv/bin/python -m ohbm2026.cli book --format all --sort poster_id --style tufte
PYTHONPATH=src .venv/bin/python -m abstractatlas.cli book --format all --sort poster_id --style tufte
```

Outputs land at `data/outputs/book/book__<state-key>/`. Figure
Expand Down Expand Up @@ -108,7 +108,7 @@ Operator flags:
Single-abstract debug (re-render one chunk in isolation, populating
the cache):
```bash
PYTHONPATH=src .venv/bin/python -m ohbm2026.book.render_per_abstract \
PYTHONPATH=src .venv/bin/python -m abstractatlas.book.render_per_abstract \
--poster-id 0042 --style plain
```

Expand Down
8 changes: 4 additions & 4 deletions docs/cllm-claims-plan.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,8 @@ experience overwhelming.

## Execution

1. Run `ohbmcli analyze-figures` first if you want cached figure notes included in
1. Run `aacli analyze-figures` first if you want cached figure notes included in
the claim manuscript.
2. Run `ohbmcli extract-claims` to populate or resume the claim cache.
3. Run `ohbmcli enrich` to merge claim extraction into the enriched abstract data.
4. Run `scripts/build_ui_data.py` (Stage 6) so the SvelteKit site's data package picks up the new fields. (The legacy `ohbmcli export-ui` / `build-ui` commands have been removed; this plan predates Stage 2.1 + Stage 6 and is kept for historical reference only.)
2. Run `aacli extract-claims` to populate or resume the claim cache.
3. Run `aacli enrich` to merge claim extraction into the enriched abstract data.
4. Run `scripts/build_ui_data.py` (Stage 6) so the SvelteKit site's data package picks up the new fields. (The legacy `aacli export-ui` / `build-ui` commands have been removed; this plan predates Stage 2.1 + Stage 6 and is kept for historical reference only.)
2 changes: 1 addition & 1 deletion docs/clustering-analysis-plan.md
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,7 @@ Runtime checks on real data:

## Status

- [x] Added `ohbmcli cluster-benchmark`
- [x] Added `aacli cluster-benchmark`
- [x] Implemented method sweep across `kmeans`, `agglomerative-ward`, `agglomerative-average`, `gaussian-mixture`, and `birch`
- [x] Added label-independent ranking using silhouette, Davies-Bouldin, Calinski-Harabasz, inter/intracluster distance ratio, and cluster-balance terms
- [x] Added test coverage for method execution, metric computation, ranking, CLI wiring, and output writing
Expand Down
32 changes: 16 additions & 16 deletions docs/ohbm26-plan.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,36 +37,36 @@ This is the operational planning document for the repository.

## Architecture

- `src/ohbm2026/graphql_api.py`
- `src/abstractatlas/graphql_api.py`
- GraphQL requests, batching, retries, env loading, author/submission fetches
- `src/ohbm2026/assets.py`
- `src/abstractatlas/assets.py`
- local asset indexing, figure URL extraction, reuse-aware downloads, JSON-only asset refresh, and ingest CLI
- `src/ohbm2026/enrichment.py`
- `src/abstractatlas/enrichment.py`
- author export, HTML-to-markdown conversion, section mapping, local Qwen figure analysis, enriched abstract assembly, and phase-2 CLI
- `src/ohbm2026/neuroscape.py`
- `src/abstractatlas/neuroscape.py`
- stage-one embeddings, semantic graph generation, clustering, and NeuroScape handoff manifest
- `src/ohbm2026/cli.py`
- unified `ohbmcli` command
- `src/abstractatlas/cli.py`
- unified `aacli` command

## Unified CLI

- `ohbmcli ingest`
- `aacli ingest`
- fetch abstracts and figure assets from the API
- `ohbmcli refresh-assets`
- `aacli refresh-assets`
- rebuild `local_assets` from the local JSON database without rerunning abstract extraction
- `ohbmcli authors`
- `aacli authors`
- export author metadata
- `ohbmcli enrich`
- `aacli enrich`
- build enriched abstracts from local databases
- `ohbmcli analyze-figures`
- `aacli analyze-figures`
- analyze local figures with Ollama
- `ohbmcli embed-minilm`
- `aacli embed-minilm`
- generate local MiniLM embeddings
- `ohbmcli embed-voyage`
- `aacli embed-voyage`
- generate Voyage embeddings
- `ohbmcli semantic-analysis`
- `aacli semantic-analysis`
- build the semantic graph and semantic cluster summaries from any local embedding bundle
- `ohbmcli write-manifest`
- `aacli write-manifest`
- write the NeuroScape handoff manifest

## Execution Order
Expand Down Expand Up @@ -226,7 +226,7 @@ Checks:
- [x] Create a local MiniLM embedding pipeline and neighbor database.
- [x] Create a purely semantic graph-and-cluster pipeline for the single-year corpus.
- [x] Split the codebase into API, asset, enrichment, and embedding modules.
- [x] Add a unified `ohbmcli` entrypoint.
- [x] Add a unified `aacli` entrypoint.
- [ ] Complete a full cached figure-analysis sweep with local `qwen3.5:35b`.
- [ ] Validate the NeuroScape retraining path against the PR-based workflow and artifact requirements.

Expand Down
28 changes: 14 additions & 14 deletions docs/per-stage-pattern.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@
Every pipeline stage in this repo (Stage 1 fetch, Stage 2 enrich,
Stage 3+ analyses, …) satisfies the same six contracts. This doc
defines them and points at two canonical reference implementations
— **Stage 1 (`src/ohbm2026/fetch_stage.py`)** for a single-fetch
stage, and **Stage 2 (`src/ohbm2026/enrich_stage.py`)** for a
— **Stage 1 (`src/abstractatlas/fetch_stage.py`)** for a single-fetch
stage, and **Stage 2 (`src/abstractatlas/enrich_stage.py`)** for a
multi-component stage — for each.

A new stage author should be able to read this page plus
Expand Down Expand Up @@ -111,7 +111,7 @@ Stage 2 reference:
What failures the stage surfaces loudly, with what typed cause,
and at what exit code.

- Typed exception hierarchy in `ohbm2026.exceptions` rooted at a
- Typed exception hierarchy in `abstractatlas.exceptions` rooted at a
cross-stage `OhbmStageError(RuntimeError)`. Stage 1 subclass tree:
`Stage1Error` → `SchemaContractError`, `CheckpointError`,
`FigureFailureError`. Stage 2 subclass tree: `Stage2Error` →
Expand Down Expand Up @@ -219,36 +219,36 @@ Stage 2 shipped the per-stage pattern scaffolding with
(`specs/004-enrich-production-wiring/`) wires production runners
through those seams and keeps the orchestrator surface unchanged:

- **Figures**: `src/ohbm2026/stage2_figures.py` —
- **Figures**: `src/abstractatlas/stage2_figures.py` —
`run_figure_component(abstract, model_id, flex_enabled, client, cwd)`
groups all figures of one abstract into a single OpenAI Responses
API call with manuscript text as context. In-memory JPEG-q85
compression at 1024 px (`compress_image`) + a four-field local
quality probe (`src/ohbm2026/image_quality.py`:
quality probe (`src/abstractatlas/image_quality.py`:
`laplacian_variance`, `mean_brightness`, `native_max_dim`,
`compression_ratio`) populate `local_quality_estimate` on every
record. Pydantic `FigureInterpretationResponse` + a fixed
`model_quality_estimate` enum validate the response server-side.
- **Claims**: `src/ohbm2026/stage2_claims.py` —
- **Claims**: `src/abstractatlas/stage2_claims.py` —
`run_claims_component(abstract, ..., figure_interpretations)`
issues a single agentic Responses API call with three function
tools registered (`verify_source_quote`, `lookup_eco_code`,
`dedupe_check`). The model orchestrates extract → verify →
annotate → dedupe internally. Post-response validation drops
claims whose source quote isn't a substring of the manuscript
OR whose `evidence_eco_codes` include off-vocabulary ECO IDs.
- **References**: `src/ohbm2026/stage2_references.py` is a thin
- **References**: `src/abstractatlas/stage2_references.py` is a thin
adapter to the existing `openalex.collect_reference_metadata`
async pipeline.
- **Flex tier**: `src/ohbm2026/flex_tier.py` —
- **Flex tier**: `src/abstractatlas/flex_tier.py` —
`call_with_flex_fallback(...)` wraps each LLM call with a 1-flex +
1-standard retry budget; defaults to flex ON; per-component
disable flags `--no-flex-figures` / `--no-flex-claims`.
- **Concurrency**: `enrich_stage._run_component_concurrent`
fans out per-abstract work across a `ThreadPoolExecutor`
(default 30 in flight). Per-component caps via
`--concurrency-figures` / `--concurrency-claims`.
- **ECO v1 vocabulary** lives at `src/ohbm2026/data/eco_top_codes.json`
- **ECO v1 vocabulary** lives at `src/abstractatlas/data/eco_top_codes.json`
(committed source). The claims component's `lookup_eco_code`
tool draws from it without network access.
- **Provenance extensions**: per-component `flex_tier_enabled`,
Expand All @@ -269,7 +269,7 @@ When you write Stage N:
2. **Author the test file first** (Principle IV). One test per
contract element, organized into a class per contract.
3. **Implement the orchestrator** in
`src/ohbm2026/<stage_name>.py`. Use Stage 1's `fetch_stage.py`
`src/abstractatlas/<stage_name>.py`. Use Stage 1's `fetch_stage.py`
as the layout reference. Share helpers via `artifacts.py`,
`exceptions.py`, and `schema_diff.py` where possible.
4. **Add a CLI subcommand** in `cli.py` that delegates to your
Expand All @@ -280,12 +280,12 @@ When you write Stage N:

## Common Helpers

- `ohbm2026.artifacts.build_*_path(state_key)` — path derivation
- `abstractatlas.artifacts.build_*_path(state_key)` — path derivation
under gitignored roots.
- `ohbm2026.artifacts.build_dependency_basis` + `build_state_key` —
- `abstractatlas.artifacts.build_dependency_basis` + `build_state_key` —
deterministic state key from input fingerprint.
- `ohbm2026.exceptions.*` — typed exception hierarchy.
- `ohbm2026.schema_diff.*` — schema-diff / discovery primitives
- `abstractatlas.exceptions.*` — typed exception hierarchy.
- `abstractatlas.schema_diff.*` — schema-diff / discovery primitives
(reusable across stages that talk to GraphQL).
- The constitution lint at
`.specify/scripts/bash/constitution-check.sh` catches the
Expand Down
6 changes: 3 additions & 3 deletions docs/poster-layout-optimizer-plan.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ Confirmed decisions for the first pass:

## Proposed Deliverables

- `src/ohbm2026/poster_layout.py`
- `src/abstractatlas/poster_layout.py`
- reusable loading, feature extraction, optimization, numbering, and scoring helpers
- `scripts/optimize_poster_layout.py`
- CLI entrypoint to assign posters to sessions and output a proposed numbering/layout
Expand Down Expand Up @@ -199,7 +199,7 @@ PYTHONPATH=src .venv/bin/python scripts/analyze_poster_layout.py \
--output data/poster_layout/analysis.json
```

If this matures well, we can later fold the same functionality into `ohbmcli`.
If this matures well, we can later fold the same functionality into `aacli`.

## Test Plan

Expand Down Expand Up @@ -227,7 +227,7 @@ If this matures well, we can later fold the same functionality into `ohbmcli`.
## Implementation Phases

- [x] Confirm session-assignment model and physical numbering assumptions
- [x] Add a poster-layout design module in `src/ohbm2026/poster_layout.py`
- [x] Add a poster-layout design module in `src/abstractatlas/poster_layout.py`
- [x] Add synthetic tests for constraints and scoring before implementing search
- [x] Implement poster feature loading and validation
- [x] Implement conflict and locality scoring
Expand Down
2 changes: 1 addition & 1 deletion docs/poster-sequencing-benchmark-plan.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ We can think of this explicitly as a weighted graph reordering problem:

## Why This Phase Now

The current semantic-path ordering logic is still based on greedy nearest-neighbor walks in [src/ohbm2026/poster_layout.py](/Users/satra/software/temp/ohbm2026/src/ohbm2026/poster_layout.py).
The current semantic-path ordering logic is still based on greedy nearest-neighbor walks in [src/abstractatlas/poster_layout.py](/Users/satra/software/temp/ohbm2026/src/abstractatlas/poster_layout.py).

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The link contains an absolute path /Users/satra/software/temp/ohbm2026/src/abstractatlas/poster_layout.py which points to a local directory of a specific user. It should be updated to a relative path within the repository so that the link works for all users and environments.

Suggested change
The current semantic-path ordering logic is still based on greedy nearest-neighbor walks in [src/abstractatlas/poster_layout.py](/Users/satra/software/temp/ohbm2026/src/abstractatlas/poster_layout.py).
The current semantic-path ordering logic is still based on greedy nearest-neighbor walks in [src/abstractatlas/poster_layout.py](../src/abstractatlas/poster_layout.py).


Relevant functions:

Expand Down
14 changes: 7 additions & 7 deletions docs/repo-review-2026-03-28.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ to navigate confidently.

## Strengths To Preserve

- the canonical pipeline is resumable and centered on `ohbmcli`
- the canonical pipeline is resumable and centered on `aacli`
- experiment immutability is explicit and documented
- test coverage is broad across pipeline, UI export, poster layout, and
experiment tooling
Expand All @@ -37,15 +37,15 @@ to navigate confidently.

Current hotspots:

- `src/ohbm2026/neuroscape.py`
- `src/abstractatlas/neuroscape.py`
- about `2951` lines, `106` functions
- `src/ohbm2026/poster_sequencing.py`
- `src/abstractatlas/poster_sequencing.py`
- about `2371` lines, `59` functions
- `src/ohbm2026/openalex.py`
- `src/abstractatlas/openalex.py`
- about `2279` lines, `76` functions
- `src/ohbm2026/poster_layout.py`
- `src/abstractatlas/poster_layout.py`
- about `2191` lines, `60` functions
- `src/ohbm2026/enrichment.py`
- `src/abstractatlas/enrichment.py`
- about `1437` lines, `59` functions

Why this matters:
Expand Down Expand Up @@ -95,7 +95,7 @@ Mitigation landed in this pass:
Recommended next step:

- introduce a `scripts/README.md` or eventually promote the most stable
organizer flows into explicit `ohbmcli` or package-level namespaces
organizer flows into explicit `aacli` or package-level namespaces

### P2: Canonical versus exploratory outputs still need stronger machine-readable boundaries

Expand Down
Loading
Loading