feat(018): rich data element model — reproschema + PROV-O + ontology + similarity#15
Conversation
Models: - ResponseOption (value, label, ontology_term) for structured choices - SemanticIdentity: +response_options, +question_text, +value_domain, +min_value, +max_value (reproschema-aligned) - ProvenanceEntry: +generated_at, +attributed_to, +activity, +derived_from (W3C PROV-O enriched flat) - ActivityType enum (ingestion, curation, enrichment, migration) Hashing: - min_value/max_value IN hash (different ranges = different element) - response_options IN hash (sorted by value for determinism) - question_text, value_domain EXCLUDED from hash - constraints.minimum/maximum EXCLUDED (deprecated by min_value/max_value) Dependencies: +requests (base), +sentence-transformers (optional [similarity]) ontology-cache/ directory created 45/45 tests, lint clean. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…late - ingest.py: auto-populates generated_at, attributed_to, activity on all provenance entries; resolves response_option values to ValueConcept URIs - BIDS extractor: response_options from enum, min/max from numeric constraints, value_domain (categorical/numeric/text/boolean) - AIND extractor: response_options from enum, min/max from JSON Schema minimum/maximum/exclusiveMinimum, question_text from title - DANDI extractor: response_options from Enum annotations, min/max from Pydantic ge/le/gt/lt metadata - Re-ingested: 3,395 elements (210 with response_options, 40 with ranges, 250 with PROV-O). 0 violations, 45/45 tests, lint clean. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- AIND extractor: $defs starting with _ (e.g., _Abcam, _Nikon) filtered
from element extraction, reclassified as ValueConcepts via extract_aind_values()
- Source-qualified tags: aind.{schema}.{parent_class}.{value}
- ingest.py: _ingest_extracted_values() writes AIND vendor values to values/
- Element count: 3,395 → 2,697 (698 underscore entries removed)
- Value count: 596 → 1,207 (611 new ValueConcepts from underscore entries)
- 0 violations, 45/45 tests, lint clean
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ilarity + alias detection Phase 5 (Ontology): - ontology_cache.py: OntologyCache class (load/lookup/save YAML term files) - ontology_fetch.py: fetch terms from OLS API (NCIT, PATO, HP, OBI, NCBITaxon) - verify.py: check term existence + deprecated flag + label similarity - CLI: `verify` (check elements), `ontology refresh` (update cache from OLS) Phase 6 (Similarity): - similarity.py: name_similarity (sentence-transformers or difflib fallback), range_overlap_score (intersection/union), valueset_jaccard (Jaccard index), compute_similarity (weighted composite), map_to_skos (exactMatch/closeMatch/ broadMatch/narrowMatch/relatedMatch) - alias_detection.py: detect_aliases (pairwise scan, skip different data_types) - CLI: `similarity` (compare two files), `detect-aliases` (scan all, threshold) 15 new tests (range overlap, valueset Jaccard, similarity scoring, ontology verification: valid/missing/deprecated terms) 60/60 total tests, lint clean. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request introduces a comprehensive upgrade to the library's data modeling capabilities, focusing on integrating advanced semantic features and robust provenance tracking. By adopting reproschema alignment and W3C PROV-O, the changes aim to create a richer, more interoperable, and semantically aware data infrastructure. The update also streamlines data element definitions by replacing rigid constraints with more flexible value domains and adds new tools for ontology management and data similarity analysis. Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
… polish - Full re-ingest: 2,697 elements + 482 schemas + 1,207 values (0 violations) - Ontology verify: 27 warnings (terms not in cache — cache empty, expected) - Alias detection: 10 candidates at threshold 0.7 (name, sex, status matches) - Backend: SemanticInput updated with response_options, min/max, question_text - Frontend: ElementDetail shows ranges, question_text, response_options badges, PROV-O provenance (activity, generated_at, attributed_to) - TypeScript types: SemanticIdentity + ProvenanceEntry enriched with new fields - README updated with current counts and new features - 53/53 frontend tests, 60/60 library tests - All 49 tasks marked complete Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 participant
Summary
Enriches the library data model with reproschema-aligned fields, W3C PROV-O provenance, ontology verification, and semantic similarity with SKOS alias detection.
Model changes:
Extractors enriched: BIDS (response_options, min/max, value_domain), AIND (response_options, min/max, question_text from title), DANDI (response_options from enums, min/max from Pydantic validators)
Underscore reclassification: AIND
_Abcametc. → ValueConcepts (elements 3395→2697, values 596→1207)New CLI commands:
verify,ontology refresh,similarity,detect-aliasesNew modules: ontology_cache.py, ontology_fetch.py, verify.py, similarity.py, alias_detection.py
Test plan
🤖 Generated with Claude Code