Cumulative JGI isolate modeling (supersedes #423/#432/#436/#440/#441/#446) by turbomam · Pull Request #448 · microbiomedata/submission-schema

turbomam · 2026-06-09T23:05:38Z

This is the submission-schema side of JGI isolate modeling. It is one cumulative PR (replacing six smaller ones) so the work can be reviewed as a single diff.

What it adds

Three DataHarmonizer interfaces for isolate submissions:

IsolateInterface: the organism biology (genus, species, strain, NCBI taxonomy, genome size, GC content, ploidy, single-colony provenance). These are stored in nmdc-schema.
JgiIsolateGenomeInterface: JGI submission logistics for isolate genome sequencing, plus the ribosomal strain-verification sequences and fungal screening.
JgiIsolateTranscriptomeInterface: JGI submission logistics for isolate transcriptome sequencing, plus the RNA experiment collection date.

The two JGI interfaces share their common logistics fields through a mixin, so a field that applies to only one project type lives only on that interface. src/docs/jgi-isolate-routing.md lists, per interface, which slot lands where and whether it is defined here or imported from nmdc-schema.

What reviewers should know

Example data in src/data/valid and src/data/invalid exercises every slot. The JGI logistics values are synthetic but shaped to match real GOLD formats (proposal, sequencing-project, and sample IDs, contact-name format); they do not identify any real proposal or person. Each invalid example fails for a single reason.
make test passes (pytest plus the valid and invalid example checks).
CONTRIBUTING.md now documents how the schema is built, which generated files are committed and why, and the example-data conventions.

nmdc-schema pin: one change needed before merge

This builds on nmdc-schema main after the two isolate PRs there merged (#3155 and #3157), which is not yet released. So pyproject.toml temporarily pins nmdc-schema to that git commit instead of a released version. When the nmdc-schema release is tagged, repin to the released version and remove the temporary git override. CI builds against main's committed artifacts, which lag slightly, but this PR does not use the removed slots, so that does not affect the result.

Sources this aligns to

This work is checked against these inputs. Use this list to audit it.

Montana-SlotEval tab of the Pupo sheet (Google Drive 1IgZyBU8wjBRFKo5Kw2LyRtdsmCL3ma0E): which interface each slot belongs to. This is the authority for slot placement.
nmdc-schema src/docs/jgi-isolate-field-routing.md: which schema slot each JGI form field maps to.
esplims (JGI's submission templates and validators): what JGI actually requires and validates.
GOLD and the JGI lakehouse (via Dremio): real value shapes for the example data.
The #squad-isolates Slack channel and the GitHub issue and PR threads: squad decisions.
The squad rolling agenda and notes doc (Google Doc 12EiYlypNm4n4AD31AUejocxA9zRr5SBdPM_NhDwYb4E): meeting decisions and action items.
nmdc-server validation: the JGI-legal submission gate. No isolate validation exists there yet, so that gate is future work.
The NMDC Schema and Metadata and Sync meeting notes.

When two sources conflict, defer to the latest input where that can be determined.

Alignment items found while checking against Montana-SlotEval, all resolved:

host_taxid: moved onto the JGI interfaces to match Montana-SlotEval.
NCBI Tax ID: kept classified_as (the merged, linked-data form); the MIxS samp_taxon_id alternative is tracked in Reconsider MIxS samp_taxon_id vs classified_as for the isolate NCBI Tax ID field #451.
Sample Isolation Method: replaced the DNA-specific dna_isolate_meth with a native isolate_meth covering DNA and RNA; the dna/rna alternatives are tracked in Revisit Sample Isolation Method slot: native isolate_meth vs MIxS dna_isolate_meth / rna_isolate_meth #452.

Closes

Closes #444: split into genome and transcriptome interfaces.
Closes #422: wire nmdc-schema slots into IsolateInterface.
Closes #433: add sample_isolated_from and collection_site_or_growth_conditions local slots.
Closes #442: collect the RNA experiment date as a single combined date.
Closes #449: add host_genus, host_species, host_strain to the JGI isolate interfaces.
Closes #428: add the submission-schema JGI isolate routing doc.

Adds two new DataHarmonizer interface classes for isolate genome sequencing: - IsolateInterface: thin shell; biological identity slots (genus, species, strain, isolate_name, classified_as, host_taxid, collection_date) will be wired in via config/nmdc_schema_import.yaml once nmdc-schema PRs #2977 and #2975 merge - JgiIsolateInterface: full JGI submission logistics + 10 new isolate-specific slots for strain verification sequences (16S/ITS/28S), single-colony status, known contaminants, and fungal QC fields New enums: RibosomalSequenceTypeEnum, JgiIsolateBiosafetyMaterialCategoryEnum analysis_type is overridden to required: false in both interfaces because AnalysisTypeEnum has no isolate WGS value; a TODO is noted in both class definitions. Adds exhaustive example data file using real GOLD data for Dictyoglomus turgidum DSM 6724 (Ga0028539, ITS project 1377532). Adds TEMPLATE_INSTANCES entries for both new classes so test_examples covers them. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

… doesn't honor slot_usage required:false

…ements When isolate_ribosomal_seq_type or isolate_second_ribosomal_seq_type is 'Other', comments are required. When a second sequence type is provided, the sequence itself must also be present. Follows same rule pattern as cont_type/cont_well. Addresses Copilot review on PR #423.

Updated example value for ribosomal sequence and modified pattern for validation.

Stray double quote at tests/test_schema.py:262 was preventing pytest collection; CI failed at AST parse time on Python 3.10. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

YAML folded block scalar (>-) joins continuation lines with spaces; the resulting example values failed ^[ACGTN]{200,5000}$ at validation time. Replace both wrapped examples with single-line sequences matching the proven fixture in tests/test_schema.py. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Adds 5 new import entries (organism_genus, organism_species, strain_name, isolate_name, classified_as) all routed to IsolateInterface only, in a new topical slot_group `organism_section`. Adds IsolateInterface as an additional destination on three existing entries (collection_date, host_taxid, source_mat_id). JgiIsolateInterface is intentionally NOT a destination; the multi-tab DH model has the JGI submitter open IsolateInterface alongside JgiIsolateInterface, mirroring how JgiMgInterface sits next to SoilInterface today. Ranks use a 100-block keyed to JGI Isolate (NA) v19 form field numbers (rank = 100 + form-field-#) for visual coherence with the source form. classified_as is flattened from `range: NcbiTaxon` to `range: string + structured_pattern: NCBITaxon:\d+` so the value fits a DataHarmonizer cell; this follows the established global-flatten pattern in this file. NOTE: blocked on bumping nmdc-schema dependency from 11.17.1 to a release that includes #2977 (currently only v11.19.2-rc.1). Build will fail until the pin is updated; that policy call is pending. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

The IsolateInterface wiring imports Organism/OrganismSample slots (organism_genus, organism_species, strain_name, classified_as) added in nmdc-schema PRs #2977/#2975, which ship only in 11.19.2rc1. The 11.17.1 pin made the build fail with "Slot 'organism_genus' not found". Bumping exposed two latent issues the build never reached: - imported slots carry in_subset: jgi_isolate, but the subset definition was not declared locally; mirror nmdc-schema's definition in the base. - the wiring makes collection_date + the four organism slots required on IsolateInterface; populate the test template instance and the exhaustive valid example (real Dictyoglomus turgidum DSM 6724 identity, taxids verified against NCBI). collection_date requiredness for isolates is under audit in #429. Regenerated artifacts. TODO to move to stable 11.19.2 when released. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

JGI's actual validation uses `^[ATCGN]+$` with no length bounds. The previous `^[ACGTN]{200,5000}$` imposed a 200-5000 nt window that the JGI form does not enforce. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…JgiIsolateInterface JGI v19 form fields 44 and 45 were deferred from nmdc-schema (see microbiomedata/nmdc-schema#3054) because they are JGI logistics fields, not core biology. They describe what a microbe was isolated from and where/how it was grown, which is JGI-specific submission metadata and belongs on the JGI Isolate tab, not the biology-facing IsolateInterface. Both are free-text string slots with rank 40 and 41 respectively, added to JgiIsolateInterface slots and slot_usage (jgi_isolate_section). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

JGI v19 requires a Reference Genome field for resequencing projects. The slot already exists in nmdc-schema (Organism class, MIXS:0000025) with a structured_alias binding it to "Reference Genome" at the JGI v19 URL. This commit routes it into JgiIsolateInterface via the import config (source: Organism) and adds it to the slots and slot_usage lists in the base schema. Once #438 lands, this slot should move to JgiIsolateGenomeInterface and JgiIsolateTranscriptomeInterface instead. A comment in both edited files marks that intent. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

The RNA experiment date refers to when the culture was grown for transcriptomics, which is distinct from collection_date (the DNA collection date, ISO-8601, already imported from nmdc-schema on IsolateInterface). Montana requested separate Y/M/D slots rather than a combined date string, as local submission-schema slots rather than nmdc-schema imports. Slots defined: rna_collection_year (integer, rank 43), rna_collection_month (string, rank 44), rna_collection_day (integer 1-31, rank 45). All three carry a structured alias mapping "Collection Year" / "Collection Month" to the JGI isolate submission form v19. Slots are added to JgiIsolateInterface for now; they will move to JgiIsolateTranscriptomeInterface once #438 is resolved. Addresses Alicia's action item on the Montana-SlotEval sheet (collection year row: "we dont have a slot for rna experiment date yet"). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Per Montana's slot routing (biology tab). Accepts Mb from submitters; nmdc-schema stores bp. Ingest conversion tracked in microbiomedata/nmdc-server#2171. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…gi-isolate-interfaces

JGI submission requires all three date components. Override the schema-wide pattern (which allows year-only or year-month) with a stricter YYYY-MM-DD pattern in IsolateInterface slot_usage. Update exhaustive example and add two invalid examples covering year-only and year-month truncation. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

… slot Use one ISO 8601 YYYY-MM-DD string rather than three separate fields. The export layer splits into JGI's Collection Year, Collection Month (full name), and Collection Day columns. Structured aliases bind all three JGI column names. Add valid and invalid example data files. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…quencing AnalysisTypeEnum now has isolate genome sequencing and isolate transcriptome sequencing values (added in nmdc-schema v11.19.2-rc.1). Drop the required:false overrides on IsolateInterface and JgiIsolateInterface and update the exhaustive example to use the correct value. Closes microbiomedata/nmdc-schema#3017 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

JGI labels this field "Biosafety Material Category" but the values are organism or genetic-element categories (Bacteria, Fungi, etc.), not biosafety levels. Rename the enum to JgiIsolateMaterialKindEnum to reflect actual semantics. Add NCBITaxon meanings for unambiguous values; descriptions for polyphyletic/ non-taxonomic entries (Alga, Protist, Plasmid). Add structured_alias binding the slot to JGI's column name. Keep slot name biosafety_mat_cat to preserve fidelity to the JGI form. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…CONTRIBUTING Fills the empty TODO sections: how the schema is assembled from the base plus the nmdc-schema import config, which generated files are committed and why, and the valid/invalid example-data rules (including why a counter-example is deleted when its only failure is removed). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Montana-SlotEval places host_taxid on the genome and transcriptome interfaces with the other host fields, not on IsolateInterface. Route it to JgiIsolateCommonMixin and move it in the exhaustive example accordingly. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…late interfaces Montana-SlotEval maps the JGI Sample Isolation Method field to isolate_meth and marks it JGI only. Its definition is the DNA or RNA extraction method, so the DNA-specific MIxS dna_isolate_meth was wrong on the transcriptome interface. Add a native isolate_meth string (required, with a structured_alias for the JGI form label) and use it on both JGI isolate interfaces. dna_isolate_meth stays on the metagenome interfaces. The NCBI Tax ID field keeps classified_as. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 32 out of 34 changed files in this pull request and generated 6 comments.

…ata sourcing Regenerate the routing-doc tables from the current schema so host_taxid shows on the JGI interfaces (not IsolateInterface) and the isolate interfaces show isolate_meth instead of dna_isolate_meth. Add a CONTRIBUTING note that example data is synthetic and not copied from the JGI spreadsheet or lakehouse, which is why the earlier lakehouse-derived valid example was replaced. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Lakehouse and GOLD values are fine to use unless a record is embargoed, private, or very recent. Only the JGI example submission spreadsheet is off-limits. Drop the PR-specific deletion history from CONTRIBUTING (it lives in the commit messages). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Replace the realistic-looking invented names with Jane Doe and John Doe so no one mistakes them for real people. The organism, taxon, genome size, GC, and isolation source remain real GOLD values; the JGI submission IDs remain GOLD-shaped placeholders. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Per Patrick's review on #432: modifications of imported nmdc-schema slots belong in config/nmdc_schema_import.yaml, not in base-yaml slot_usage. Move estimated_size (title, unit) and the IsolateInterface-specific collection_date override (strict full-date pattern and the Collection Year/Month/Day aliases, scoped with in_classes so the other interfaces keep their looser pattern). The two raw # comments become notes. No induced-slot behavior changes; 25 tests pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 32 out of 34 changed files in this pull request and generated 5 comments.

…uting doc, single-failure examples - Fix the import-config comment that still said host_taxid stays on IsolateInterface - Make the base biosafety_mat_cat description accurate for its metagenome-category range (the isolate interfaces narrow it to organism kinds); drop the raw # comments on the slot and the JgiIsolateMaterialKindEnum - Qualify the routing-doc line so readers do not assume all IsolateInterface fields are stored in nmdc-schema - Add the required isolate_single_colony to the two collection_date counterexamples so each fails for a single reason; use the isolate analysis type Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

aclum · 2026-06-10T15:55:34Z

+    strain_name: DSM 6724
+    classified_as: NCBITaxon:515635
+    isolate_single_colony: "yes"
+jgi_isolate_genome_data:


estimated genome size is missing

are you saying that you want estimated genome size to always be required?

I want estimated genome size to be required for JGIIsolateGenomeInterface. The spreadsheet says
(required for draft genomes, resequencing, and methylation detection projects) which is all the DNA isolate products.

pkalita-lbl · 2026-06-10T16:19:10Z

 license = "MIT"
 dependencies = [
    "linkml-runtime>=1.6.2,<2",
+    "nmdc-schema",


I understand the reason for using a git source for the nmdc-schema dependency temporarily. In addition to removing the tool.uv.sources stanza, please also be sure to remove this before merging. nmdc-schema is a dev dependency of this project, not a main dependency.

…only Per Alicia's review on #448: estimated genome size is required for all DNA isolate products (draft genomes, resequencing, methylation), not transcriptome. Move estimated_size from the shared IsolateInterface to JgiIsolateGenomeInterface with required: true, so it is required for genome submissions and never asked of transcriptome submissions. Move it out of the isolate_data example blocks into the jgi_isolate_genome_data blocks, add a missing-estimated_size counterexample, and retarget the high-value counterexample to the genome interface. Per Patrick's review: nmdc-schema is only needed to build the schema, so it is a dev dependency, not a main one. Consumers use the committed artifacts. Regenerate the routing doc and the committed artifacts to match. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…rate artifacts nmdc-schema v11.20.0 is published, so pin the dev dependency to ==11.20.0 and remove the temporary [tool.uv.sources] git override (closing Patrick's review point). Regenerating against the real release instead of the stale git-main install updates the committed artifacts: the JGI alias source is now the real public URL rather than the fabricated form URL, plus upstream description fixes. make all and make test pass against the released schema, the true post-merge integration test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

turbomam and others added 30 commits April 27, 2026 10:57

Revert generated files to main; source-only changes in this PR

b65827b

Add analysis_type placeholder to isolate example; linkml-run-examples…

42ce31e

… doesn't honor slot_usage required:false

Revise ribosomal sequence example and pattern

9961fd5

Updated example value for ribosomal sequence and modified pattern for validation.

Update nmdc_submission_schema_base.yaml

666bf20

Update ribosomal sequences in test schema

f85fc34

Add pattern validation for ribosomal sequence fields

cfd3842

Fix formatting of ribosomal sequences in test schema

f227cd8

Refactor ribosomal sequence fields in test schema

07ee627

Refactor isolate_ribosomal_seq for better readability

537230b

Refactor isolate_ribosomal_seq assignment

9e94185

Fix unterminated string literal in isolate_ribosomal_seq fixture

2b46ca0

Stray double quote at tests/test_schema.py:262 was preventing pytest collection; CI failed at AST parse time on Python 3.10. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Align ribosomal seq pattern with JGI spec

c08bf36

JGI's actual validation uses `^[ATCGN]+$` with no length bounds. The previous `^[ACGTN]{200,5000}$` imposed a 200-5000 nt window that the JGI form does not enforce. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Move form field references from aliases to comments

d91223f

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Trim verbose comments to one line each

ebacfab

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Trim verbose comments

e4ee1d0

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Wire estimated_size into IsolateInterface

cea406e

Per Montana's slot routing (biology tab). Accepts Mb from submitters; nmdc-schema stores bp. Ingest conversion tracked in microbiomedata/nmdc-server#2171. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Merge remote-tracking branch 'origin/main' into 418-add-isolate-and-j…

0bd580f

…gi-isolate-interfaces

Add JGI column aliases to collection_date slot_usage on IsolateInterface

cffdeaa

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

turbomam mentioned this pull request Jun 10, 2026

Move biosafety_mat_cat to IsolateInterface; add conditional max rules for estimated_size #435

Closed

turbomam and others added 4 commits June 9, 2026 20:48

Plainer wording in CONTRIBUTING build/test docs

d1d4104

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

turbomam mentioned this pull request Jun 10, 2026

Revisit Sample Isolation Method slot: native isolate_meth vs MIxS dna_isolate_meth / rna_isolate_meth #452

Open

turbomam requested a review from Copilot June 10, 2026 01:38

Copilot started reviewing on behalf of turbomam June 10, 2026 01:38 View session

Copilot AI reviewed Jun 10, 2026

View reviewed changes

Comment thread src/docs/jgi-isolate-routing.md Outdated

Comment thread src/docs/jgi-isolate-routing.md

Comment thread src/docs/jgi-isolate-routing.md

Comment thread src/docs/jgi-isolate-routing.md

Comment thread src/docs/jgi-isolate-routing.md

Comment thread pyproject.toml

turbomam and others added 4 commits June 9, 2026 21:46

Document the obvious-placeholder convention for invented example values

e5d1dbb

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

turbomam requested review from aclum, mslarae13 and pkalita-lbl June 10, 2026 02:04

turbomam requested a review from Copilot June 10, 2026 02:12

Copilot started reviewing on behalf of turbomam June 10, 2026 02:13 View session

Copilot AI reviewed Jun 10, 2026

View reviewed changes

aclum reviewed Jun 10, 2026

View reviewed changes

Comment thread src/nmdc_submission_schema/schema/nmdc_submission_schema_base.yaml

aclum approved these changes Jun 10, 2026

View reviewed changes

pkalita-lbl reviewed Jun 10, 2026

View reviewed changes

turbomam and others added 2 commits June 10, 2026 16:48

pkalita-lbl merged commit 40e6d60 into main Jun 10, 2026
2 checks passed

pkalita-lbl deleted the jgi-isolate-cumulative branch June 10, 2026 22:04

aclum linked an issue Jun 26, 2026 that may be closed by this pull request

Add Classes for a isolate modeling #418

Closed

4 tasks

Uh oh!

Conversation

turbomam commented Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What it adds

What reviewers should know

nmdc-schema pin: one change needed before merge

Sources this aligns to

Closes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aclum Jun 10, 2026

Choose a reason for hiding this comment

Uh oh!

turbomam Jun 10, 2026

Choose a reason for hiding this comment

Uh oh!

aclum Jun 10, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pkalita-lbl Jun 10, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

turbomam commented Jun 9, 2026 •

edited

Loading