Skip to content

Commit b6e8812

Browse files
turbomamclaude
turbomam
and
claude
authored
Remove the ChromaDB pipeline (resolves critical CVE #455 + #401) (#498)
* Remove the ChromaDB pipeline (resolves #455, #401) The DIY ChromaDB semantic-search pipeline is being replaced by OLS4 embeddings + linkml-store + local GPU embeddings (#364). chromadb carries the unpatchable critical CVE #455 (no upstream fix), so removal is the only resolution; it also drops diskcache (#401). Removed: - chromadb dependency from the notebooks extra (uv.lock no longer contains chromadb or diskcache) - metpo/database/ (audit/combine/filter/migrate chromadb tools) - metpo/pipeline/chromadb_semantic_mapper.py, embed_ontology_to_chromadb.py - metpo/scripts/find_best_definitions.py, iterative_definition_improvement.py (the two chromadb-coupled definition tools; to be reworked on OLS4/linkml-store) - 7 chromadb-coupled CLI entry points; cli_common.chroma_path_option - the root Makefile's "Ontology Alignment Pipeline" section + 3 chromadb-dependent definition-report targets - notebooks/query_chromadb.ipynb (+ stale checkpoints) Kept: the non-chromadb pipeline/analysis scripts (analyze_matches, analyze_coherence, fetch_ontology_names, categorize_ontologies, the chromadb-free definition scripts) and notebooks/assess_ontology_by_api_search.ipynb (the OLS-API search notebook, to be reformulated as a Click CLI as the replacement). Not touched: the ontology build (src/ontology/Makefile) and CI — chromadb was never part of either. make test, ruff, and the 35-test suite pass. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Add cross-ontology-search (stopgap replacement for the removed chromadb tools) Small, dependency-light CLI so the definition/synonym/mapping-finding capability isn't lost with the chromadb removal: OLS4 lexical search for candidates + local free embeddings (Ollama, parameterized via --model/--embed-url) for ranking, with a --no-embeddings lexical fallback. Output gives mapping CURIE, label (synonym), and definition (source) per match. Explicitly marked a STOPGAP in the module docstring: it duplicates capabilities of OAK/oaklib (#194) and linkml-store (#364) and should migrate to them, not grow. Tracked in those issues. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Refactor cross-ontology-search to pass ruff; reframe as the working replacement Same behavior, reorganized: extracted load_terms / rank_candidates / write_matches helpers (fixes PLR0912 too-many-branches), Path.open + context-managed output (PTH123/SIM115), zip(strict=False) (B905). Dropped the 'stopgap' framing per direction: this IS the chromadb replacement; OAK (#194) + linkml-store (#364) are the acknowledged future direction, not a blocking migration. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Apply ruff format to cross_ontology_search.py Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Address Copilot review on cross_ontology_search + cleanup (#498) - load_terms: skip the ROBOT template spec row (row 2, label 'LABEL') so it is not queried as a real term. - ols_candidates: catch URLError/timeout/JSON errors and skip the term instead of aborting the whole run on a transient OLS outage. - write_matches / load_terms: open files with explicit encoding='utf-8' so non-ASCII labels/definitions are locale-independent. - pyproject: drop 'openai' from the notebooks extra (nothing in notebooks/ imports it after the ChromaDB demo removal); refresh uv.lock. - Makefile: remove help text and epilogue lines pointing at targets this PR deleted (help-alignment, generate-non-ols-tsvs, scan-manifest, view-logs). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
1 parent 6ebeb83 commit b6e8812

18 files changed

Lines changed: 284 additions & 5397 deletions

‎Makefile‎

Lines changed: 2 additions & 196 deletions
Original file line numberDiff line numberDiff line change
@@ -55,14 +55,8 @@ help:
5555
@echo "Analysis Reports:"
5656
@echo " make all-reports - Generate all analysis reports"
5757
@echo ""
58-
@echo "Ontology Alignment Pipeline:"
59-
@echo " make help-alignment - Show detailed alignment pipeline help"
60-
@echo " make alignment-run-all - Run complete alignment pipeline"
61-
@echo ""
6258
@echo "External Ontology Downloads:"
6359
@echo " make download-external-bioportal-ontologies - Download non-OLS ontologies"
64-
@echo " make generate-non-ols-tsvs - Extract terms for embeddings"
65-
@echo " make scan-manifest - Update ontology manifest"
6660
@echo ""
6761
@echo "Literature Mining:"
6862
@echo " make -C literature_mining help - Show literature mining help"
@@ -96,7 +90,7 @@ install-literature:
9690
install-databases:
9791
uv sync --extra databases
9892

99-
# Notebooks environment (adds: jupyter, notebook, matplotlib, numpy, chromadb, etc.)
93+
# Notebooks environment (adds: jupyter, notebook, matplotlib, numpy, etc.)
10094
install-notebooks:
10195
uv sync --extra notebooks
10296

@@ -354,33 +348,14 @@ audit-ids: reports/id-allocation-audit.md
354348
# These reports analyze definition coverage and quality in METPO.
355349
# Most require the SSSOM mappings file from the alignment pipeline.
356350

357-
reports/definition_improvement_opportunities.tsv: src/templates/metpo_sheet.tsv notebooks/metpo_relevant_mappings.sssom.tsv
358-
uv run analyze-definition-opportunities \
359-
--template $< \
360-
--mappings $(word 2,$^) \
361-
--output $@
362-
363351
reports/definition_coverage_by_parent.tsv: src/templates/metpo_sheet.tsv
364352
uv run analyze-definition-coverage-by-subtree \
365353
--metpo-tsv $< \
366354
--output $@ \
367355
--sort-by stragglers
368356

369-
# Note: find-best-definitions requires chromadb optional dependency
370-
# Install with: uv sync --extra notebooks
371-
reports/best_definitions_per_term.tsv: src/templates/metpo_sheet.tsv
372-
uv run find-best-definitions \
373-
--metpo-tsv $< \
374-
--output $@
375-
376-
reports/definition_comparison_with_hierarchy.tsv: reports/best_definitions_per_term.tsv src/templates/metpo_sheet.tsv
377-
uv run compare-definitions-with-hierarchy \
378-
--best-definitions $< \
379-
--metpo-terms $(word 2,$^) \
380-
--output $@
381-
382357
.PHONY: definition-reports
383-
definition-reports: reports/definition_improvement_opportunities.tsv reports/definition_coverage_by_parent.tsv
358+
definition-reports: reports/definition_coverage_by_parent.tsv
384359
@echo "Definition analysis reports generated"
385360

386361
.PHONY: clean-definition-reports
@@ -595,8 +570,6 @@ download-external-bioportal-ontologies: $(foreach ont,$(NON_OLS_BIOPORTAL_ONTOLO
595570
@echo "=========================================="
596571
@echo "Download phase complete"
597572
@echo "=========================================="
598-
@echo "Run 'make scan-manifest' to update tracking"
599-
@echo "Run 'make view-logs' to see any failures"
600573

601574

602575
# Download ontology from BioPortal
@@ -690,173 +663,6 @@ list-bioportal-releases:
690663
@echo ""
691664
@echo "Note: Set BIOPORTAL_API_KEY environment variable for authenticated downloads"
692665

693-
# ==============================================================================
694-
# Ontology Alignment Pipeline - Granular Targets
695-
# ==============================================================================
696-
697-
.PHONY: alignment-fetch-ontology-names alignment-categorize-ontologies \
698-
alignment-query-metpo-terms alignment-analyze-matches \
699-
alignment-analyze-coherence alignment-identify-candidates \
700-
alignment-run-all clean-alignment-results clean-alignment-all help-alignment
701-
702-
# Individual pipeline steps
703-
alignment-fetch-ontology-names: notebooks/ontology_catalog.csv
704-
705-
notebooks/ontology_catalog.csv: data/ontology_assessments/ontology_sizes.csv
706-
@echo "Fetching ontology metadata from OLS4 API..."
707-
uv run python metpo/pipeline/fetch_ontology_names.py \
708-
--sizes-csv data/ontology_assessments/ontology_sizes.csv \
709-
--output-csv notebooks/ontology_catalog.csv
710-
711-
alignment-categorize-ontologies: notebooks/ontologies_very_appealing.csv
712-
713-
notebooks/ontologies_very_appealing.csv: notebooks/ontology_catalog.csv
714-
@echo "Categorizing ontologies by relevance..."
715-
uv run python metpo/pipeline/categorize_ontologies.py \
716-
--input-csv notebooks/ontology_catalog.csv \
717-
--output-prefix notebooks/ontologies
718-
719-
alignment-query-metpo-terms: notebooks/metpo_relevant_mappings.sssom.tsv
720-
721-
notebooks/metpo_relevant_mappings.sssom.tsv: notebooks/metpo_relevant_chroma
722-
@echo "Generating SSSOM mappings from METPO terms via ChromaDB..."
723-
@if [ -z "$$OPENAI_API_KEY" ]; then \
724-
echo "ERROR: OPENAI_API_KEY environment variable not set"; \
725-
exit 1; \
726-
fi
727-
uv run python metpo/pipeline/chromadb_semantic_mapper.py \
728-
--metpo-tsv src/templates/metpo_sheet.tsv \
729-
--chroma-path notebooks/metpo_relevant_chroma \
730-
--collection-name metpo_relevant_embeddings \
731-
--output notebooks/metpo_relevant_mappings.sssom.tsv \
732-
--top-n 10 \
733-
--label-only \
734-
--distance-cutoff 0.35
735-
736-
alignment-analyze-matches: notebooks/metpo_relevant_mappings.sssom.tsv
737-
@echo "Analyzing match quality..."
738-
uv run python metpo/pipeline/analyze_matches.py \
739-
--input-csv notebooks/metpo_relevant_mappings.sssom.tsv \
740-
--good-match-threshold 0.9
741-
742-
alignment-analyze-coherence: notebooks/full_coherence_results.csv
743-
744-
notebooks/full_coherence_results.csv: notebooks/metpo_relevant_mappings.sssom.tsv
745-
@echo "Computing structural coherence (this may take a while)..."
746-
uv run python metpo/pipeline/analyze_sibling_coherence.py \
747-
--input-csv notebooks/metpo_relevant_mappings.sssom.tsv \
748-
--metpo-owl src/ontology/metpo.owl \
749-
--output-csv notebooks/full_coherence_results.csv
750-
751-
alignment-identify-candidates: notebooks/alignment_candidates.csv
752-
753-
notebooks/alignment_candidates.csv: notebooks/full_coherence_results.csv
754-
@echo "Identifying alignment candidates..."
755-
uv run python metpo/pipeline/analyze_coherence_results.py \
756-
--results-csv notebooks/full_coherence_results.csv \
757-
--matches-csv notebooks/metpo_relevant_mappings.sssom.tsv
758-
759-
# Run complete pipeline
760-
alignment-run-all: alignment-identify-candidates
761-
@echo ""
762-
@echo "========================================="
763-
@echo "Alignment pipeline complete!"
764-
@echo "========================================="
765-
@echo "Output files:"
766-
@echo " - notebooks/metpo_relevant_mappings.sssom.tsv"
767-
@echo " - notebooks/full_coherence_results.csv"
768-
@echo " - notebooks/alignment_candidates.csv"
769-
770-
# Clean alignment results
771-
clean-alignment-results:
772-
@echo "Cleaning alignment pipeline results..."
773-
rm -f notebooks/metpo_*_matches.csv
774-
rm -f notebooks/*_coherence_results.csv
775-
rm -f notebooks/alignment_candidates.csv
776-
@echo "Alignment results cleaned (ontology catalog preserved)"
777-
778-
# Clean everything including ontology catalog
779-
780-
# =====================================================
781-
# Non-OLS Embedding Targets
782-
# =====================================================
783-
784-
NON_OLS_TSV_FILES = $(foreach ont,$(NON_OLS_BIOPORTAL_ONTOLOGIES),data/pipeline/non-ols-terms/$(ont).tsv)
785-
786-
.PHONY: embed-non-ols-terms clean-non-ols-terms scan-manifest view-manifest view-logs
787-
788-
.PHONY: generate-non-ols-tsvs
789-
generate-non-ols-tsvs: $(NON_OLS_TSV_FILES)
790-
@echo ""
791-
@echo "=========================================="
792-
@echo "Query phase complete"
793-
@echo "=========================================="
794-
@echo "Run 'make scan-manifest' to update tracking"
795-
@echo "Run 'make view-logs' to see any failures"
796-
797-
# Manifest and logging targets
798-
scan-manifest:
799-
@echo "Scanning directories and updating manifest..."
800-
uv run scan-manifest --verbose
801-
802-
view-manifest:
803-
@if [ -f .ontology_manifest.json ]; then \
804-
cat .ontology_manifest.json | python -m json.tool; \
805-
else \
806-
echo "No manifest found. Run 'make scan-manifest' first."; \
807-
fi
808-
809-
view-logs:
810-
@echo "=== Recent Fetch Failures ==="
811-
@if [ -f .ontology_fetch.log ]; then tail -20 .ontology_fetch.log; else echo "No fetch failures logged"; fi
812-
@echo ""
813-
@echo "=== Recent Query Failures ==="
814-
@if [ -f .robot_query.log ]; then grep -E "QUERY_FAILED|QUERY_EMPTY" .robot_query.log | tail -20 || echo "No query failures logged"; else echo "No query log found"; fi
815-
816-
embed-non-ols-terms:
817-
@echo "Embedding non-OLS terms into ChromaDB..."
818-
uv run python metpo/pipeline/embed_ontology_to_chromadb.py \
819-
$(foreach tsv,$(wildcard data/pipeline/non-ols-terms/*.tsv),--tsv-file $(tsv)) \
820-
--chroma-path ./embeddings_chroma \
821-
--collection-name non_ols_embeddings
822-
@echo "Non-OLS terms embedded successfully."
823-
824-
clean-non-ols-terms:
825-
@echo "Cleaning generated non-OLS TSV files..."
826-
rm -f $(NON_OLS_TSV_FILES)
827-
@echo "Cleaned non-OLS TSV files"
828-
829-
clean-alignment-all: clean-alignment-results
830-
@echo "Cleaning all alignment files including ontology catalog..."
831-
rm -f notebooks/ontology_catalog.csv
832-
rm -f notebooks/ontologies_*.csv
833-
@echo "All alignment files cleaned"
834-
835-
# Help target
836-
help-alignment:
837-
@echo "METPO Ontology Alignment Pipeline Targets:"
838-
@echo ""
839-
@echo " Preparation:"
840-
@echo " make alignment-fetch-ontology-names - Fetch ontology metadata from OLS4"
841-
@echo " make alignment-categorize-ontologies - Categorize by relevance"
842-
@echo ""
843-
@echo " Analysis:"
844-
@echo " make alignment-query-metpo-terms - Query METPO against ChromaDB"
845-
@echo " make alignment-analyze-matches - Analyze match quality"
846-
@echo " make alignment-analyze-coherence - Compute structural coherence"
847-
@echo " make alignment-identify-candidates - Find high-quality candidates"
848-
@echo ""
849-
@echo " Complete workflow:"
850-
@echo " make alignment-run-all - Run entire pipeline"
851-
@echo ""
852-
@echo " Cleanup:"
853-
@echo " make clean-alignment-results - Clean results only"
854-
@echo " make clean-alignment-all - Clean all including catalog"
855-
@echo ""
856-
@echo " Prerequisites:"
857-
@echo " - Set OPENAI_API_KEY environment variable"
858-
@echo " - Ensure ChromaDB collection exists at notebooks/metpo_relevant_chroma"
859-
860666
# ==============================================================================
861667
# Sub-Makefile Integration
862668
# ==============================================================================

‎metpo/cli_common.py‎

Lines changed: 0 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -106,21 +106,6 @@ def output_dir_option(required=False, default=None, help_text="Output directory
106106
# =============================================================================
107107

108108

109-
def chroma_path_option(default="./chroma_db", help_text="ChromaDB storage directory path"):
110-
"""Standard ChromaDB storage path option.
111-
112-
Args:
113-
default: Default ChromaDB path (default: './chroma_db')
114-
help_text: Custom help text
115-
116-
Returns:
117-
Click option decorator for --chroma-path
118-
"""
119-
return click.option(
120-
"--chroma-path", type=click.Path(path_type=str), default=default, help=help_text
121-
)
122-
123-
124109
def db_path_option(required=True, help_text="SQLite database file path"):
125110
"""Standard SQLite database path option.
126111

‎metpo/database/README.md‎

Lines changed: 0 additions & 120 deletions
This file was deleted.

‎metpo/database/__init__.py‎

Whitespace-only changes.

0 commit comments

Comments
 (0)