Commit be7e6e2
feat(039): Robust Ingestion Pipeline — Parquet, batch CLI, aliases, ranges
* specify(039): Robust Ingestion Pipeline v2 — binary storage, pipeline routing, ranges
6 user stories:
- US1: Binary container storage for 2.7M+ entities (Parquet/SQLite)
- US2: All adapters through standard pipeline (no direct writes)
- US3: NDA cross-structure alias preservation for alignment
- US4: Element range display in UI (valueset, min/max, pattern, type_ref)
- US5: Batch CLI (--batch N, --all) for OpenNeuro and NDA
- US6: Enrichment scaling to 220K+ elements with expanded NCBITaxon
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* plan(039): Robust Ingestion Pipeline v2 — Parquet storage, batch CLI, NDA aliases
Research: Parquet chosen over SQLite/JSONL for binary entity storage.
One file per entity type per source. Threshold: >1K → Parquet, ≤1K → YAML.
Design: 5 phases — Parquet backend, batch CLI routing, NDA aliases,
element range UI, enrichment scaling. StorageBackend protocol extended.
Artifacts: research.md, data-model.md, plan.md, quickstart.md
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* tasks(039): Robust Ingestion Pipeline — 41 tasks across 9 phases, 6 user stories
Phase 1-2: Parquet infrastructure (ParquetStore, FileBackend hybrid, staging/commit)
US1: Scalable storage — CLI inspect, DB import, round-trip tests
US2: Pipeline routing — --batch N, --all flags, OpenNeuro/NDA wiring
US3: NDA alias preservation — dedup by name+type, alias_hints, alignment boost
US4: Element range display — adapter audit (8 adapters), frontend range section
US5: Batch CLI polish — error handling, consolidated summaries
US6: Enrichment scaling — chunk processing, species precision
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* analyze(039): address top 3 findings — FR-004 enforcement, storage size, alias baseline
E1: T019 now includes test asserting no direct registry writes outside pipeline
E4: T011 now asserts Parquet file size < 10x YAML equivalent (SC-003)
E5: T024 now compares alias-hint confidence against embedding-only baseline (SC-006)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* clarify(039): element ranges are part of identity hash, display in browse + detail
Different ranges → different sha256 → distinct elements. Each element
contains range from its source. UI must show range in both browse
tables and detail pages.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(039): ParquetStore — binary entity storage with 17 passing tests
T001-T003: ParquetStore class with write_batch/read/list/count/build_index.
One Parquet file per entity type per source. Dedup by sha256 with
provenance merge. Prefix-match reads. Cross-source index generation.
Tests: write/read, prefix match, dedup, list/count, scale (10K roundtrip),
size (Parquet >2x smaller than YAML), all 4 entity types.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(039): StorageBackend write_batch/read_batch + FileBackend Parquet hybrid
T004: Extended StorageBackend protocol with write_batch() and read_batch()
T005: FileBackend uses ParquetStore for batches > 1000 entities,
falls back to YAML for small batches. find_by_hash also checks
Parquet files. read_batch reads from both YAML + Parquet.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(039): Parquet-first storage — staging, commit, FileBackend all use ParquetStore
T006: staging.write_staged_batch() always uses Parquet. iter_staged()
reads from both YAML + Parquet files.
T007: commit reads from iter_staged (Parquet + YAML), writes committed
entities to Parquet in addition to YAML.
T008: ParquetStore.build_index() generates _index.parquet.
FileBackend: write_batch() always uses Parquet (no threshold).
list() yields from Parquet first, then YAML (dedup by sha256).
All 22 tests pass (17 parquet + 5 commit).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(039): batch pipeline CLI + inspect command + batch_ingest for OpenNeuro/NDA
T009: CLI inspect command — query Parquet/YAML by sha256, list, count
T013-T014: --batch N and --all flags on pipeline CLI
T015: batch_ingest() orchestrates extract → stage → enrich → commit
T016: OpenNeuro batch — git clone + datalad get + extract + Parquet staging
T017: NDA batch — API fetch all structures + extract + Parquet staging
T018: save_batch_summary() for batch run summaries
T020: Progress logging [N/total] dataset → entity_count
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(039): remove ad-hoc batch scripts, replaced by pipeline --batch/--all CLI
T019: Deleted ingest_openneuro_fast.py and ingest_openneuro_batch.py.
Batch ingestion now goes through: undata-library pipeline --source nda --all
or --source openneuro --batch 100
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(039): NDA alias dedup, range display UI, species precision filter
T021-T022: NDA adapter deduplicates elements by (name, type) across
structures, adds alias_hints list with all NDA structure refs.
T029-T031: Element detail page shows Range & Constraints section:
min/max range, pattern regex, type_ref link, response_options table.
T036: Enrichment post-filter removes genus-level NCBITaxon matches
when species-level match exists for the same organism.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(039): alias_hints boost in alignment + NDA alias tests + chunk enrichment
T023: Alignment boosts score to 0.95 when elements share alias_hints
T024: 3 tests — shared elements get alias_hints, unique preserved, boost works
T032-T033: Already implemented in batch_ingest (error handling + summary)
T035: Sequential file processing already manages memory; 220K at 384-dim = 320MB OK
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(039): import_service Parquet support, adapter audits, NDA alias names
T010: import_service reads Parquet files alongside YAML during registry import
T025: BIDS adapter already extracts response_options, min/max, pattern, type_ref
T028: OpenNeuro adapter extracts MinValue/MaxValue from JSON sidecars
NDA: preserves API-native alias names (e.g., sex→gender) in semantic.alias_hints
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(039): export Parquet, NDA native aliases, adapter range audit complete
T012: export_service writes Parquet alongside YAML for bulk consumption
T026-T027: NWB + openMINDS handle ranges via LinkML extraction path
(response_options from enums, type_ref from class refs)
NDA: extracts API-native alias names (sex→gender) into alias_hints
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(039): species precision tests — genus removed when species exists
T037: 3 tests for _prefer_species_over_genus(). 36/41 tasks done.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(039): type_ref resolved to schema sha256, array item types extracted
- DANDI adapter: List[X] → data_type=array + range=X (not "string")
- LinkML extractor already handles multivalued=True + range=ClassName → type_ref
- Commit: _resolve_cross_references now resolves element type_ref from
class names to schema sha256 hashes (alongside property + member resolution)
- NDA: native alias names (sex→gender) added to semantic.alias_hints
- Fix: staged_file reference in source validation
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(039): all 41/41 tasks complete — robust ingestion pipeline v2
463 library tests pass. Key deliverables:
- ParquetStore: binary entity storage (17 tests, 10K roundtrip, 2x+ compression)
- Batch CLI: --batch N / --all for OpenNeuro and NDA through full pipeline
- NDA aliases: dedup by name+type, API-native alias names, alias_hints boost
- Element ranges: all adapters audited, type_ref resolved to sha256, array item types
- Species precision: genus removed when species-level match exists
- Range display: consolidated UI section with min/max, pattern, type_ref links
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(039): ruff — remove unused imports in test files
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(039): remove v1 legacy components, separate ontology index from pipeline
Removed:
- migration-api/ (v1 standalone service)
- MigrationJobStatus, Pathway*, Comparison*, SchemaDiff* components
- lib/api/migration.ts
Changed: enrichment no longer auto-builds ontology index from scratch.
Run 'undata-library embed --include-ontology' before first enrichment.
Stale indexes still auto-rebuild on ontology store checksum change.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(039): remove presentation/ from git, add to .gitignore
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(039): build ontology embeddings from pyoxigraph store, not YAML cache
build_ontology_embeddings() now reads directly from the ontology store
(pyoxigraph) instead of YAML cache files. Eliminates the cache_dir
YAML bottleneck entirely. Falls back to YAML only if store is unavailable.
11 ontologies in store: NCIT, PATO, HP, NCBITaxon, RadLex, NIDM,
DICOM, EDAM, TMN, OBI (new), HoMBA (new).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(039): entity embeddings stored with entities in Parquet + compute_entity_embeddings()
- ParquetStore schema: added 'embedding' column (JSON-serialized float list)
- compute_entity_embeddings(): batch compute for any entity type
- _build_element_text() already handles elements, values, schemas
- Embeddings travel with entities through staging → commit → import
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(039): remove tests for deleted v1 components (ComparisonView, PathwayCard, SchemaDiff)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(039): Parquet-native batch enrichment — no YAML iteration
enrich_all() rewritten to use iter_staged() for all entity types.
Embeddings computed in one batch per entity type via compute_entity_embeddings().
No more per-file YAML iteration or per-entity-type embedding rebuilds.
Flow: read from Parquet → batch embed → match ontology → write back Parquet
Ontology index must be pre-built (not auto-rebuilt during enrichment).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(039): use pre-computed embeddings during DB import, skip recomputation
database_backend: checks entity.embedding first before calling
compute_embedding(). If entity already has a pre-computed embedding
(from pipeline enrichment), uses it directly.
import_service: proper ParquetStore import (no lazy import).
NCBITaxon filter description corrected (~90 taxa not 20).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(039): alignment reads from Parquet + uses pre-computed embeddings
align.py: _load_embedding_store builds EmbeddingStore from entity
Parquet files that have pre-computed embeddings (no separate index needed).
alias_detection.py: reads entities from ParquetStore first (with
embeddings), then YAML fallback. Deduplicates by sha256.
This should fix the 0-group alignment issue — previously the embedding
store wasn't loaded because embeddings.parquet didn't exist at the
expected path.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>1 parent 0826a82 commit be7e6e2
87 files changed
Lines changed: 2577 additions & 6064 deletions
File tree
- backend
- src
- services
- storage
- frontend
- app/elements/[sha256]
- components
- lib/api
- tests/unit
- library
- scripts
- src/undata_library
- adapters
- storage
- tests
- migration-api
- src
- api
- v1
- services
- tasks
- tests
- contract
- fixtures
- unit
- specs
- 038-system-hardening
- 039-robust-ingestion-pipeline
- checklists
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
55 | 55 | | |
56 | 56 | | |
57 | 57 | | |
| 58 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
62 | 62 | | |
63 | 63 | | |
64 | 64 | | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
65 | 79 | | |
66 | 80 | | |
67 | 81 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | | - | |
| 1 | + | |
2 | 2 | | |
3 | 3 | | |
4 | 4 | | |
| 5 | + | |
5 | 6 | | |
6 | 7 | | |
7 | 8 | | |
| |||
13 | 14 | | |
14 | 15 | | |
15 | 16 | | |
| 17 | + | |
16 | 18 | | |
17 | 19 | | |
18 | 20 | | |
| |||
58 | 60 | | |
59 | 61 | | |
60 | 62 | | |
| 63 | + | |
| 64 | + | |
61 | 65 | | |
62 | 66 | | |
63 | 67 | | |
64 | 68 | | |
65 | 69 | | |
66 | 70 | | |
67 | 71 | | |
| 72 | + | |
68 | 73 | | |
69 | 74 | | |
70 | 75 | | |
| |||
75 | 80 | | |
76 | 81 | | |
77 | 82 | | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
78 | 96 | | |
79 | 97 | | |
80 | 98 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
119 | 119 | | |
120 | 120 | | |
121 | 121 | | |
122 | | - | |
| 122 | + | |
123 | 123 | | |
124 | 124 | | |
125 | 125 | | |
126 | | - | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
127 | 143 | | |
128 | 144 | | |
129 | 145 | | |
130 | | - | |
131 | | - | |
132 | | - | |
133 | 146 | | |
134 | 147 | | |
135 | 148 | | |
| |||
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
5 | 5 | | |
6 | 6 | | |
7 | 7 | | |
| 8 | + | |
8 | 9 | | |
9 | 10 | | |
10 | 11 | | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
11 | 88 | | |
12 | 89 | | |
13 | 90 | | |
| |||
95 | 172 | | |
96 | 173 | | |
97 | 174 | | |
98 | | - | |
99 | | - | |
100 | | - | |
101 | | - | |
102 | | - | |
103 | | - | |
104 | 175 | | |
105 | 176 | | |
106 | 177 | | |
107 | 178 | | |
108 | 179 | | |
109 | 180 | | |
110 | | - | |
111 | | - | |
112 | | - | |
113 | | - | |
114 | | - | |
115 | | - | |
116 | | - | |
| 181 | + | |
117 | 182 | | |
118 | | - | |
119 | | - | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
120 | 192 | | |
121 | 193 | | |
122 | 194 | | |
123 | 195 | | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
124 | 199 | | |
125 | 200 | | |
126 | 201 | | |
| |||
This file was deleted.
0 commit comments