Skip to content

Commit be7e6e2

Browse files
satraclaude
andauthored
feat(039): Robust Ingestion Pipeline — Parquet, batch CLI, aliases, ranges
* specify(039): Robust Ingestion Pipeline v2 — binary storage, pipeline routing, ranges 6 user stories: - US1: Binary container storage for 2.7M+ entities (Parquet/SQLite) - US2: All adapters through standard pipeline (no direct writes) - US3: NDA cross-structure alias preservation for alignment - US4: Element range display in UI (valueset, min/max, pattern, type_ref) - US5: Batch CLI (--batch N, --all) for OpenNeuro and NDA - US6: Enrichment scaling to 220K+ elements with expanded NCBITaxon Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * plan(039): Robust Ingestion Pipeline v2 — Parquet storage, batch CLI, NDA aliases Research: Parquet chosen over SQLite/JSONL for binary entity storage. One file per entity type per source. Threshold: >1K → Parquet, ≤1K → YAML. Design: 5 phases — Parquet backend, batch CLI routing, NDA aliases, element range UI, enrichment scaling. StorageBackend protocol extended. Artifacts: research.md, data-model.md, plan.md, quickstart.md Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * tasks(039): Robust Ingestion Pipeline — 41 tasks across 9 phases, 6 user stories Phase 1-2: Parquet infrastructure (ParquetStore, FileBackend hybrid, staging/commit) US1: Scalable storage — CLI inspect, DB import, round-trip tests US2: Pipeline routing — --batch N, --all flags, OpenNeuro/NDA wiring US3: NDA alias preservation — dedup by name+type, alias_hints, alignment boost US4: Element range display — adapter audit (8 adapters), frontend range section US5: Batch CLI polish — error handling, consolidated summaries US6: Enrichment scaling — chunk processing, species precision Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * analyze(039): address top 3 findings — FR-004 enforcement, storage size, alias baseline E1: T019 now includes test asserting no direct registry writes outside pipeline E4: T011 now asserts Parquet file size < 10x YAML equivalent (SC-003) E5: T024 now compares alias-hint confidence against embedding-only baseline (SC-006) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * clarify(039): element ranges are part of identity hash, display in browse + detail Different ranges → different sha256 → distinct elements. Each element contains range from its source. UI must show range in both browse tables and detail pages. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(039): ParquetStore — binary entity storage with 17 passing tests T001-T003: ParquetStore class with write_batch/read/list/count/build_index. One Parquet file per entity type per source. Dedup by sha256 with provenance merge. Prefix-match reads. Cross-source index generation. Tests: write/read, prefix match, dedup, list/count, scale (10K roundtrip), size (Parquet >2x smaller than YAML), all 4 entity types. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(039): StorageBackend write_batch/read_batch + FileBackend Parquet hybrid T004: Extended StorageBackend protocol with write_batch() and read_batch() T005: FileBackend uses ParquetStore for batches > 1000 entities, falls back to YAML for small batches. find_by_hash also checks Parquet files. read_batch reads from both YAML + Parquet. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(039): Parquet-first storage — staging, commit, FileBackend all use ParquetStore T006: staging.write_staged_batch() always uses Parquet. iter_staged() reads from both YAML + Parquet files. T007: commit reads from iter_staged (Parquet + YAML), writes committed entities to Parquet in addition to YAML. T008: ParquetStore.build_index() generates _index.parquet. FileBackend: write_batch() always uses Parquet (no threshold). list() yields from Parquet first, then YAML (dedup by sha256). All 22 tests pass (17 parquet + 5 commit). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(039): batch pipeline CLI + inspect command + batch_ingest for OpenNeuro/NDA T009: CLI inspect command — query Parquet/YAML by sha256, list, count T013-T014: --batch N and --all flags on pipeline CLI T015: batch_ingest() orchestrates extract → stage → enrich → commit T016: OpenNeuro batch — git clone + datalad get + extract + Parquet staging T017: NDA batch — API fetch all structures + extract + Parquet staging T018: save_batch_summary() for batch run summaries T020: Progress logging [N/total] dataset → entity_count Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(039): remove ad-hoc batch scripts, replaced by pipeline --batch/--all CLI T019: Deleted ingest_openneuro_fast.py and ingest_openneuro_batch.py. Batch ingestion now goes through: undata-library pipeline --source nda --all or --source openneuro --batch 100 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(039): NDA alias dedup, range display UI, species precision filter T021-T022: NDA adapter deduplicates elements by (name, type) across structures, adds alias_hints list with all NDA structure refs. T029-T031: Element detail page shows Range & Constraints section: min/max range, pattern regex, type_ref link, response_options table. T036: Enrichment post-filter removes genus-level NCBITaxon matches when species-level match exists for the same organism. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(039): alias_hints boost in alignment + NDA alias tests + chunk enrichment T023: Alignment boosts score to 0.95 when elements share alias_hints T024: 3 tests — shared elements get alias_hints, unique preserved, boost works T032-T033: Already implemented in batch_ingest (error handling + summary) T035: Sequential file processing already manages memory; 220K at 384-dim = 320MB OK Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(039): import_service Parquet support, adapter audits, NDA alias names T010: import_service reads Parquet files alongside YAML during registry import T025: BIDS adapter already extracts response_options, min/max, pattern, type_ref T028: OpenNeuro adapter extracts MinValue/MaxValue from JSON sidecars NDA: preserves API-native alias names (e.g., sex→gender) in semantic.alias_hints Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(039): export Parquet, NDA native aliases, adapter range audit complete T012: export_service writes Parquet alongside YAML for bulk consumption T026-T027: NWB + openMINDS handle ranges via LinkML extraction path (response_options from enums, type_ref from class refs) NDA: extracts API-native alias names (sex→gender) into alias_hints Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(039): species precision tests — genus removed when species exists T037: 3 tests for _prefer_species_over_genus(). 36/41 tasks done. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(039): type_ref resolved to schema sha256, array item types extracted - DANDI adapter: List[X] → data_type=array + range=X (not "string") - LinkML extractor already handles multivalued=True + range=ClassName → type_ref - Commit: _resolve_cross_references now resolves element type_ref from class names to schema sha256 hashes (alongside property + member resolution) - NDA: native alias names (sex→gender) added to semantic.alias_hints - Fix: staged_file reference in source validation Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(039): all 41/41 tasks complete — robust ingestion pipeline v2 463 library tests pass. Key deliverables: - ParquetStore: binary entity storage (17 tests, 10K roundtrip, 2x+ compression) - Batch CLI: --batch N / --all for OpenNeuro and NDA through full pipeline - NDA aliases: dedup by name+type, API-native alias names, alias_hints boost - Element ranges: all adapters audited, type_ref resolved to sha256, array item types - Species precision: genus removed when species-level match exists - Range display: consolidated UI section with min/max, pattern, type_ref links Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(039): ruff — remove unused imports in test files Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(039): remove v1 legacy components, separate ontology index from pipeline Removed: - migration-api/ (v1 standalone service) - MigrationJobStatus, Pathway*, Comparison*, SchemaDiff* components - lib/api/migration.ts Changed: enrichment no longer auto-builds ontology index from scratch. Run 'undata-library embed --include-ontology' before first enrichment. Stale indexes still auto-rebuild on ontology store checksum change. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(039): remove presentation/ from git, add to .gitignore Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(039): build ontology embeddings from pyoxigraph store, not YAML cache build_ontology_embeddings() now reads directly from the ontology store (pyoxigraph) instead of YAML cache files. Eliminates the cache_dir YAML bottleneck entirely. Falls back to YAML only if store is unavailable. 11 ontologies in store: NCIT, PATO, HP, NCBITaxon, RadLex, NIDM, DICOM, EDAM, TMN, OBI (new), HoMBA (new). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(039): entity embeddings stored with entities in Parquet + compute_entity_embeddings() - ParquetStore schema: added 'embedding' column (JSON-serialized float list) - compute_entity_embeddings(): batch compute for any entity type - _build_element_text() already handles elements, values, schemas - Embeddings travel with entities through staging → commit → import Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(039): remove tests for deleted v1 components (ComparisonView, PathwayCard, SchemaDiff) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(039): Parquet-native batch enrichment — no YAML iteration enrich_all() rewritten to use iter_staged() for all entity types. Embeddings computed in one batch per entity type via compute_entity_embeddings(). No more per-file YAML iteration or per-entity-type embedding rebuilds. Flow: read from Parquet → batch embed → match ontology → write back Parquet Ontology index must be pre-built (not auto-rebuilt during enrichment). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(039): use pre-computed embeddings during DB import, skip recomputation database_backend: checks entity.embedding first before calling compute_embedding(). If entity already has a pre-computed embedding (from pipeline enrichment), uses it directly. import_service: proper ParquetStore import (no lazy import). NCBITaxon filter description corrected (~90 taxa not 20). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(039): alignment reads from Parquet + uses pre-computed embeddings align.py: _load_embedding_store builds EmbeddingStore from entity Parquet files that have pre-computed embeddings (no separate index needed). alias_detection.py: reads entities from ParquetStore first (with embeddings), then YAML fallback. Deduplicates by sha256. This should fix the 0-group alignment issue — previously the embedding store wasn't loaded because embeddings.parquet didn't exist at the expected path. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent 0826a82 commit be7e6e2

87 files changed

Lines changed: 2577 additions & 6064 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -55,3 +55,4 @@ docs/site/
5555
/pnpm-lock.yaml
5656
*.pkg
5757
docker-compose.override.yml
58+
presentation/

backend/src/services/export_service.py

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -62,6 +62,20 @@ async def export_full_registry(
6262
entity_counts[entity_type] = count
6363
logger.info("Exported %d %s", count, entity_type)
6464

65+
# Also export as Parquet for bulk consumption
66+
if count > 0:
67+
try:
68+
from undata_library.storage.parquet_store import ParquetStore
69+
70+
pq_store = ParquetStore(export_dir)
71+
pq_entities = []
72+
for row in rows:
73+
pq_entities.append(_row_to_dict(row, entity_type))
74+
pq_store.write_batch(entity_type, pq_entities, source="export")
75+
logger.info("Exported %d %s as Parquet", count, entity_type)
76+
except ImportError:
77+
pass # ParquetStore not available in this environment
78+
6579
# Export curation flags
6680
flags_dir = export_dir / "curation-flags"
6781
flags_dir.mkdir(exist_ok=True)

backend/src/services/import_service.py

Lines changed: 19 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,8 @@
1-
"""Import flat-file YAML registry into PostgreSQL via DatabaseBackend.
1+
"""Import flat-file YAML (and Parquet) registry into PostgreSQL via DatabaseBackend.
22
33
Reads YAML files from a registry directory and writes them through the
44
DatabaseBackend, which handles upsert (idempotent re-import).
5+
Also reads Parquet files using ParquetStore for large-scale registries.
56
"""
67

78
from __future__ import annotations
@@ -13,6 +14,7 @@
1314
import yaml
1415
from sqlalchemy.ext.asyncio import AsyncSession
1516
from undata_library.models import CurationFlag, FlagStatus, FlagType, RunSummary
17+
from undata_library.storage.parquet_store import ParquetStore
1618

1719
from src.storage.database_backend import DatabaseBackend
1820

@@ -58,13 +60,16 @@ async def import_registry(
5860
await session.execute(model.__table__.delete())
5961

6062
# Import core entity types via DatabaseBackend
63+
parquet_store = ParquetStore(registry_dir)
64+
6165
for entity_type in ("elements", "schemas", "values", "valuesets"):
6266
entity_dir = registry_dir / entity_type
6367
if not entity_dir.exists():
6468
stats[entity_type] = 0
6569
continue
6670

6771
count = 0
72+
# 1. Import from YAML files
6873
for f in sorted(entity_dir.glob("*.yaml")):
6974
try:
7075
data = yaml.safe_load(f.read_text(encoding="utf-8"))
@@ -75,6 +80,19 @@ async def import_registry(
7580
except Exception as exc:
7681
logger.warning("Failed to import %s: %s", f, exc)
7782

83+
# 2. Import from Parquet files
84+
parquet_files = sorted(entity_dir.glob("*.parquet"))
85+
if parquet_files:
86+
try:
87+
for entity in parquet_store.list(entity_type):
88+
sha = entity.get("sha256", "")
89+
if not sha or "semantic" not in entity:
90+
continue
91+
await backend.entities.write(entity_type, entity, identifier=sha)
92+
count += 1
93+
except Exception as exc:
94+
logger.warning("Failed to import Parquet for %s: %s", entity_type, exc)
95+
7896
stats[entity_type] = count
7997

8098
# Import transforms

backend/src/storage/database_backend.py

Lines changed: 18 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -119,17 +119,30 @@ async def write(self, entity_type: str, data: dict, identifier: str | None = Non
119119
}
120120
)
121121

122-
# Compute embedding and search tsvector if model supports it
122+
# Use pre-computed embedding if available, otherwise compute
123123
if hasattr(model, "embedding") and model.embedding is not None:
124124
from sqlalchemy import func
125125

126-
from src.services.embedding_service import build_search_text, compute_embedding
126+
pre_computed = data.get("embedding")
127+
if pre_computed and isinstance(pre_computed, list):
128+
kwargs["embedding"] = pre_computed
129+
else:
130+
from src.services.embedding_service import (
131+
build_search_text,
132+
compute_embedding,
133+
)
134+
135+
search_text = build_search_text(entity_type, data)
136+
if search_text:
137+
embedding = compute_embedding(search_text)
138+
if embedding:
139+
kwargs["embedding"] = embedding
140+
141+
# Always compute search tsvector (fast, no model needed)
142+
from src.services.embedding_service import build_search_text
127143

128144
search_text = build_search_text(entity_type, data)
129145
if search_text:
130-
embedding = compute_embedding(search_text)
131-
if embedding:
132-
kwargs["embedding"] = embedding
133146
kwargs["search_tsv"] = func.to_tsvector("english", search_text)
134147

135148
# Upsert by sha256

backend/uv.lock

Lines changed: 3 additions & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

frontend/app/elements/[sha256]/page.tsx

Lines changed: 90 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -5,9 +5,86 @@ import { useParams } from "next/navigation";
55
import { GET_ELEMENT, SCHEMAS_USING_ELEMENT, TRANSFORMS_FOR_ELEMENT, FLAGS_FOR_ENTITY } from "@/graphql/queries";
66
import { EntityDetailLayout } from "@/components/EntityDetailLayout";
77
import { EntityTag } from "@/components/EntityTag";
8+
import Link from "next/link";
89
import type { ElementNode, SchemaConnection, TransformConnection, CurationFlagConnection, Edge, SchemaNode, TransformNode, CurationFlagNode } from "@/graphql/types";
910
import { getStatusColor } from "@/lib/source-colors";
1011

12+
interface ResponseOption {
13+
value: string | number;
14+
label?: string;
15+
}
16+
17+
function RangeConstraintsSection({ element }: { element: ElementNode }) {
18+
const hasRange = element.minValue != null || element.maxValue != null;
19+
const hasPattern = !!element.pattern;
20+
const hasTypeRef = !!element.typeRef;
21+
const responseOptions = (element.semantic?.response_options ?? element.semantic?.responseOptions) as ResponseOption[] | undefined;
22+
const hasResponseOptions = Array.isArray(responseOptions) && responseOptions.length > 0;
23+
24+
if (!hasRange && !hasPattern && !hasTypeRef && !hasResponseOptions) return null;
25+
26+
return (
27+
<div className="mt-3">
28+
<div className="text-xs text-gray-500 mb-1">Range &amp; Constraints</div>
29+
<div className="border rounded p-3 space-y-2 bg-gray-50">
30+
{hasRange && (
31+
<div className="flex items-center gap-2 text-sm">
32+
<span className="text-gray-500 text-xs">Range:</span>
33+
<span className="font-mono">
34+
{element.minValue != null ? element.minValue : "−∞"}
35+
{" – "}
36+
{element.maxValue != null ? element.maxValue : "∞"}
37+
</span>
38+
</div>
39+
)}
40+
41+
{hasPattern && (
42+
<div className="flex items-center gap-2 text-sm">
43+
<span className="text-gray-500 text-xs">Pattern:</span>
44+
<code className="font-mono text-xs bg-white px-1.5 py-0.5 rounded border">{element.pattern}</code>
45+
</div>
46+
)}
47+
48+
{hasTypeRef && (
49+
<div className="flex items-center gap-2 text-sm">
50+
<span className="text-gray-500 text-xs">Type Ref:</span>
51+
<Link
52+
href={`/schemas/${element.typeRef}`}
53+
className="text-blue-600 hover:underline font-mono text-xs"
54+
>
55+
{element.typeRef}
56+
</Link>
57+
</div>
58+
)}
59+
60+
{hasResponseOptions && (
61+
<div>
62+
<div className="text-gray-500 text-xs mb-1">Response Options ({responseOptions!.length})</div>
63+
<div className="max-h-48 overflow-y-auto">
64+
<table className="w-full text-xs">
65+
<thead>
66+
<tr className="text-left text-gray-400 border-b">
67+
<th className="py-0.5 pr-3 font-medium">Value</th>
68+
<th className="py-0.5 font-medium">Label</th>
69+
</tr>
70+
</thead>
71+
<tbody>
72+
{responseOptions!.map((opt, i) => (
73+
<tr key={i} className="border-b border-gray-100 last:border-0">
74+
<td className="py-0.5 pr-3 font-mono">{String(opt.value)}</td>
75+
<td className="py-0.5 text-gray-600">{opt.label ?? "—"}</td>
76+
</tr>
77+
))}
78+
</tbody>
79+
</table>
80+
</div>
81+
</div>
82+
)}
83+
</div>
84+
</div>
85+
);
86+
}
87+
1188
export default function ElementDetailPage() {
1289
const params = useParams();
1390
const sha256 = params.sha256 as string;
@@ -95,32 +172,30 @@ export default function ElementDetailPage() {
95172
</div>
96173
</div>
97174
)}
98-
{element.pattern && (
99-
<div className="border rounded p-2">
100-
<div className="text-[10px] text-gray-500 uppercase">Pattern</div>
101-
<div className="font-mono text-xs truncate" title={element.pattern}>{element.pattern}</div>
102-
</div>
103-
)}
104175
{element.valueDomain && (
105176
<div className="border rounded p-2">
106177
<div className="text-[10px] text-gray-500 uppercase">Value Domain</div>
107178
<div className="text-sm">{element.valueDomain}</div>
108179
</div>
109180
)}
110-
{element.minValue != null && (
111-
<div className="border rounded p-2">
112-
<div className="text-[10px] text-gray-500 uppercase">Min Value</div>
113-
<div className="text-sm">{element.minValue}</div>
114-
</div>
115-
)}
116-
{element.maxValue != null && (
181+
{element.typeRef && (
117182
<div className="border rounded p-2">
118-
<div className="text-[10px] text-gray-500 uppercase">Max Value</div>
119-
<div className="text-sm">{element.maxValue}</div>
183+
<div className="text-[10px] text-gray-500 uppercase">Type Ref</div>
184+
<div className="text-sm">
185+
<Link
186+
href={`/schemas/${element.typeRef}`}
187+
className="text-blue-600 hover:underline font-mono text-xs"
188+
>
189+
{element.typeRef}
190+
</Link>
191+
</div>
120192
</div>
121193
)}
122194
</div>
123195

196+
{/* Range & Constraints — consolidated section */}
197+
<RangeConstraintsSection element={element} />
198+
124199
{/* Cross-references */}
125200
{schemas.length > 0 && (
126201
<div className="mt-3">

frontend/components/ComparisonPage.tsx

Lines changed: 0 additions & 57 deletions
This file was deleted.

0 commit comments

Comments
 (0)