nmdc-runtime should use materalized_pattern version of json schema

[aclum]

I believe the runtime code is currently loading nmdc.schema.json, this does not have the regular expression patterns we need, those are in nmdc_materialized_patterns.schema.json which is also part of the pypi distribution.

We discovered this when debugging migrators for Berkeley schema and didn't get errors I was expecting.

Eric said that from nmdc_schema.nmdc_data import get_nmdc_jsonschema_dict under the hood fetches nmdc.schema.json so we need to figure out how instead to load nmdc_materialized_patterns.schema.json

cc @eecavanna @turbomam @shreddd

It would be great if this could be updated in the next few days so it can be tested next week for the june 2024 release on the 24th. cc @pkalita-lbl

[eecavanna]

nmdc-schema v11.* PyPI package versions newer than v11.0.0rc11 will allow consumers to specify which variant of the JSON Schema they want to load—either the default variant or the "materialized patterns" variant. This new capability was implemented via microbiomedata/berkeley-schema-fy24#204.

[eecavanna]

Looks to me like the Runtime is already — at least to some extent — accessing the "materialized patterns" variant of the schema (pulling it directly from the nmdc-schema package's file tree instead of accessing it via the nmdc_schema.nmdc_data module).

nmdc-runtime/nmdc_runtime/util.py

Lines 103 to 113 in 8de5ef1

	@lru_cache
	def get_nmdc_jsonschema_dict(enforce_id_patterns=True):
	"""Get NMDC JSON Schema with materialized patterns (for identifier regexes)."""
	d = json.loads(
	BytesIO(
	pkgutil.get_data("nmdc_schema", "nmdc_materialized_patterns.schema.json")
	)
	.getvalue()
	.decode("utf-8")
	)
	return d if enforce_id_patterns else without_id_patterns(d)

[aclum]

@eecavanna Is there anything else do to here? Did you confirm that the materialized pattern is always used in runtime.

[shreddd]

Reassigning to @eecavanna - please comment or close as needed.

[eecavanna]

Sorry, @aclum. I missed (or saw back then and then forgot about) your question addressed to me on August, 2024, about whether this was done or not.

Here are all the places in the nmdc-runtime codebase where I see a variant of NMDC Schema other than nmdc_materialized_patterns.schema.json (or nmdc_materialized_patterns.yaml) being referenced:

1. In nmdc_runtime/api/models/run.py, there is a reference (from back in 2022) to a file named schema.json residing in the nmdc-runtime file tree (maybe the nmdc-schema Python package wasn't a thing yet 🤷 ).

nmdc-runtime/nmdc_runtime/api/models/run.py

Lines 18 to 23 in 89c7d03

	SCHEMA_URL_BASE_DEFAULT = (
	"https://github.com/microbiomedata/nmdc-runtime/tree/main/nmdc_runtime/"
	)
	PRODUCER_URL = PRODUCER_URL_BASE_DEFAULT.replace("/main/", "/v0-0-1/") + "producer"
	SCHEMA_URL = SCHEMA_URL_BASE_DEFAULT.replace("/main/", "/v0-0-1/") + "schema.json"

• Next step: Hi @dwinston, I am not familiar with this code. Assuming it is still in use (i.e. it isn't obsolete code), is this reference one that could be changed to refer to the nmdc-schema package?

2. In metadata-translation/src/bin/validate_json.py, there is another reference (from back in 2021) to a file named schema.json residing in the nmdc-runtime file tree (see bullet point 1 for my speculation about why this might have been done).

nmdc-runtime/metadata-translation/src/bin/validate_json.py

Lines 33 to 41 in 89c7d03

	def test_gold_study_json(
	data_path="output/nmdc_etl/gold_study.json",
	schema_path="../../../schema/nmdc.schema.json",
	log_file="study_error.log",
	):
	valid = validate_json(data_path, schema_path, log_file)
	assert valid
	return valid

• Next step A: The reference is the default value of a function parameter. I am comfortable editing this to be /path/to/schema.json. I will make a sub-issue for that and assign it to myself.
• Next step B: Hi @dwinston, do you happen to know whether anyone uses this module? It was last updated in February, 2022. Maybe a more modern alternative has been introduced into the repo since then.

3. In metadata-translation/notebooks/ghissue_252_253_linked_samples.ipynb, there is a Markdown sentence that refers to a file named nmdc.schema.json.

nmdc-runtime/metadata-translation/notebooks/ghissue_252_253_linked_samples.ipynb

Line 305 in 89c7d03

"Need to update mongo collection schema to account for updated fields in nmdc.schema.json:"

• Next step: I am comfortable editing this to say the NMDC Schema. I will make a sub-issue for that and assign it to myself.

4. In metadata-translation/notebooks/202106_workflow_execution_demo.ipynb, there is an environment variable whose value is a file path ending in nmdc.schema.gsp.json.

nmdc-runtime/metadata-translation/notebooks/202106_workflow_execution_demo.ipynb

Lines 1451 to 1460 in 89c7d03

	{
	"cell_type": "code",
	"execution_count": null,
	"id": "4c11001d",
	"metadata": {},
	"outputs": [],
	"source": [
	"os.environ[\"NMDC_JSON_SCHEMA_FILE\"] = \"/Users/dwinston/Desktop/nmdc.schema.gsp.json\""
	]
	},

• I don't recommend any changes be made here, since (a) I expect readers to recognize that the file path refers to someone's local machine and (b) the file name nmdc.schema.gsp.json doesn't exist in the nmdc-schema repository.

5. In docs/tutorials/json.md, there is a Markdown sentence that refers to (and links to) the file named nmdc.schema.json in the nmdc-schema GitHub repo.

nmdc-runtime/docs/tutorials/json.md

Lines 137 to 141 in 89c7d03

	The `collection_name` must be one defined for a
	[nmdc:Database](https://microbiomedata.github.io/nmdc-schema/Database/), in the form expected by the
	JSON Schema,
	[nmdc.schema.json](https://github.com/microbiomedata/nmdc-schema/blob/69fd1ee91afac1a943b2cc9bfbfdecd0e2cdd089/jsonschema/nmdc.schema.json#L987).
	This typically means that any spaces in the name should be entered as underscores (`_`) instead.

• Next step: I am comfortable replacing this to say nmdc_materialized_patterns.schema.json and updating the link URL to point to the nmdc_materialized_patterns.schema.json file in the nmdc-schema GitHub repo. I will make a sub-issue for that and assign it to myself.

Summary

• I'll update occurrences 2, 3, and 5.
• I have questions for @dwinston about occurrences 1 and 2.
• I plan to leave occurrence 4 as is.

[eecavanna]

I have updated occurrences 2, 3, and 5, in this PR: #887

[eecavanna]

Once that PR has been merged in and occurrence 1 has been updated (or "waived" as a non-issue), we can close this issue.