Skip to content

Commit c5b2a27

Browse files
satraclaude
andcommitted
feat(040): all 33/33 tasks complete — unified embedding & storage
Parquet-only pipeline: zero YAML entity files at any stage Embeddings computed at commit for all entity types Comprehensive embedding text (name + desc + type + unit + annotations + provenance) Backend uses pre-computed embeddings (no model loading during import) All mutations recompute embeddings Missing embedding detection with on-demand computation Unified store interface via ParquetStore Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent 737387f commit c5b2a27

1 file changed

Lines changed: 8 additions & 8 deletions

File tree

  • specs/040-unified-embedding-storage

specs/040-unified-embedding-storage/tasks.md

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -45,7 +45,7 @@
4545
- [X] T012 [US1] Rewrite cross-reference resolution (_resolve_cross_references) to operate on DataFrames from ParquetStore in library/src/undata_library/commit.py
4646
- [X] T013 [US1] Remove yaml.dump calls from commit path and yaml_to_parquet conversion step in library/src/undata_library/commit.py and library/src/undata_library/staging.py
4747
- [X] T014 [US1] Update batch_ingest to use ParquetStore throughout (no YAML intermediaries) in library/src/undata_library/ingest.py
48-
- [ ] T015 [US1] Add test: pipeline run produces zero YAML entity files in library/tests/test_parquet_pipeline.py
48+
- [X] T015 [US1] Add test: pipeline run produces zero YAML entity files in library/tests/test_parquet_pipeline.py
4949

5050
**Checkpoint**: `find output -name "*.yaml" -not -path "*/runs/*"` returns 0
5151

@@ -61,9 +61,9 @@
6161
- [X] T017 [US2] Add embedding computation step to commit_staged — after sha256 computation, compute embeddings for all entities in batch in library/src/undata_library/commit.py
6262
- [X] T018 [US2] Ensure all entity types (elements, schemas, values, valuesets) get embeddings at commit in library/src/undata_library/commit.py
6363
- [X] T019 [US2] Update backend import to skip embedding model loading when all entities have pre-computed embeddings in backend/src/storage/database_backend.py
64-
- [ ] T020 [US2] Update backend import_service to read Parquet-only (remove YAML import path) in backend/src/services/import_service.py
65-
- [ ] T021 [US2] Add test: committed entity has embedding field with 384 floats in library/tests/test_parquet_pipeline.py
66-
- [ ] T022 [US2] Add test: backend import of 100 entities with embeddings completes without loading sentence-transformers model in backend/tests/test_import.py
64+
- [X] T020 [US2] Update backend import_service to read Parquet-only (remove YAML import path) in backend/src/services/import_service.py
65+
- [X] T021 [US2] Add test: committed entity has embedding field with 384 floats in library/tests/test_parquet_pipeline.py
66+
- [X] T022 [US2] Add test: backend import of 100 entities with embeddings completes without loading sentence-transformers model in backend/tests/test_import.py
6767

6868
**Checkpoint**: 100% of committed entities have embeddings; backend import <30s for 7K entities
6969

@@ -102,10 +102,10 @@
102102

103103
**Purpose**: Final verification, cleanup, test suite green
104104

105-
- [ ] T030 Run full library test suite — all tests pass with Parquet-only pipeline in library/tests/
106-
- [ ] T031 Run full pipeline for BIDS source end-to-end — verify zero YAML, embeddings present, cross-references resolved (schema properties → sha256), storage <2GB, import <30s
107-
- [ ] T032 [P] Verify CLI inspect command works with Parquet-only registry
108-
- [ ] T033 [P] Verify frontend displays entities imported from Parquet registry with pre-computed embeddings
105+
- [X] T030 Run full library test suite — all tests pass with Parquet-only pipeline in library/tests/
106+
- [X] T031 Run full pipeline for BIDS source end-to-end — verify zero YAML, embeddings present, cross-references resolved (schema properties → sha256), storage <2GB, import <30s
107+
- [X] T032 [P] Verify CLI inspect command works with Parquet-only registry
108+
- [X] T033 [P] Verify frontend displays entities imported from Parquet registry with pre-computed embeddings
109109

110110
---
111111

0 commit comments

Comments
 (0)