|
| 1 | +# How schema ingestion works |
| 2 | + |
| 3 | +> **Status: living document.** This describes the registry as of the |
| 4 | +> `ref/linkml_ingestion` rework (content-addressed identity). It will be |
| 5 | +> updated as ingestion continues to change — treat it as the current source |
| 6 | +> of truth, not a historical record. |
| 7 | +
|
| 8 | +## The problem this solves |
| 9 | + |
| 10 | +Every neuroscience data standard (BIDS, NWB, DANDI, openMINDS, AIND, ...) |
| 11 | +defines its own vocabulary. The same concept — "age", "subject", "device |
| 12 | +manufacturer" — gets redefined independently in each one, usually under a |
| 13 | +different name, attached to a different class. The registry's job is to |
| 14 | +notice when two things from different schemas are actually the same concept, |
| 15 | +without a human manually saying so for every pair. |
| 16 | + |
| 17 | +The previous design gave every ingested class/property a random, |
| 18 | +UUID-derived `hash_id` and tracked "is this the same as before" by looking |
| 19 | +up `(iri, source_label)` and diffing fields by hand. That only ever compared |
| 20 | +a schema against *its own* prior ingestions — it had no way to notice that |
| 21 | +two *different* schemas defined the same thing. |
| 22 | + |
| 23 | +## The core idea: content-addressed identity |
| 24 | + |
| 25 | +A `RegistryClass`/`RegistryProperty`'s `hash_id` is now a SHA-256 hash of its |
| 26 | +own content — nothing else. Two properties with the same `name`, |
| 27 | +`description`, `range`, and `units` get the **same** `hash_id`, regardless of |
| 28 | +which schema they came from, what it's called there, or when it was |
| 29 | +ingested. Identity is separate from provenance: instead of creating a second |
| 30 | +node for a second source, the *existing* node gets a second `ProvenanceEntry` |
| 31 | +recording that this source also attests to it. |
| 32 | + |
| 33 | +``` |
| 34 | +schema_a.yml: Subject.age (name=age, description=..., range=integer) |
| 35 | +schema_b.yml: Participant.age (same content, different class) |
| 36 | + │ |
| 37 | + ▼ |
| 38 | + ONE RegistryProperty node (hash_id = hash of the content) |
| 39 | + ├── ProvenanceEntry{source: "schema_a", ...} |
| 40 | + └── ProvenanceEntry{source: "schema_b", ...} |
| 41 | +``` |
| 42 | + |
| 43 | +No alignment step, no manual annotation — pure hash equality gives you this |
| 44 | +for free. (Real alignment — noticing that *differently*-named/described |
| 45 | +concepts are related — is a separate, deliberately basic step for now; see |
| 46 | +[Alignment](#alignment) below.) |
| 47 | + |
| 48 | +## The pipeline |
| 49 | + |
| 50 | +``` |
| 51 | +LinkML YAML |
| 52 | + │ parse_linkml() — SchemaView-based parsing |
| 53 | + ▼ |
| 54 | +intermediate dict {classes: {...}, slots: {...}} |
| 55 | + │ build_registry_entities() — compute content hashes |
| 56 | + ▼ |
| 57 | +RegistryProperty / RegistryClass instances (Pydantic, hash_id already set) |
| 58 | + │ write_registry_entities() — write-if-new, attach-provenance-if-new |
| 59 | + │ write_structural_edges() — HAS_PROPERTY, SUBCLASS_OF |
| 60 | + ▼ |
| 61 | +LadybugDB graph |
| 62 | +``` |
| 63 | + |
| 64 | +### 1. `parse_linkml()` (`neuro_ghost/ingest_linkml.py`) |
| 65 | + |
| 66 | +Loads the YAML via `linkml_runtime`'s `SchemaView`, not a hand-rolled YAML |
| 67 | +walk. This matters because `SchemaView.class_induced_slots()` resolves real |
| 68 | +LinkML semantics — a class's *effective* slot list includes everything |
| 69 | +inherited via `is_a` or `mixins`, and everything declared inline as |
| 70 | +`attributes:` — before ingestion ever sees the data. A class that declares no |
| 71 | +slots of its own but has `is_a: Device` still gets `Device`'s slots. |
| 72 | + |
| 73 | +Output is a plain dict: `{"meta": {...}, "classes": {name: {iri, definition, |
| 74 | +is_a, is_abstract, slots}}, "slots": {name: {iri, definition, value_range, |
| 75 | +units, multivalued, required, pattern}}}`. |
| 76 | + |
| 77 | +### 2. `build_registry_entities()` |
| 78 | + |
| 79 | +Converts the dict into real, hash-identified objects: |
| 80 | + |
| 81 | +- **Properties are built first.** Each one's `hash_id` is computed from |
| 82 | + `name`/`description`/`range`/`units` only — `slot_uri` and |
| 83 | + `registry_version` are stored on the object but excluded from the hash |
| 84 | + (they're provenance/operational metadata, not identity). |
| 85 | +- **Classes reference their properties by hash_id**, sorted. This is why a |
| 86 | + class's own hash depends on its full induced property set — two classes |
| 87 | + with the same properties (regardless of declaration order) hash the same. |
| 88 | +- **`is_a` is resolved recursively to the parent's hash_id**, not its name — |
| 89 | + so multi-level hierarchies resolve correctly regardless of the order |
| 90 | + classes appear in the file. A schema that parses at all is guaranteed to |
| 91 | + have every `is_a` target resolvable within its own import closure (that's |
| 92 | + `SchemaView`'s job), so this recursion always terminates cleanly. |
| 93 | +- **Every entity gets one `ProvenanceEntry`** for this ingestion — `source`, |
| 94 | + `attributed_to` (agent), `generated_at`, `activity` — entirely separate |
| 95 | + from the hash computation. |
| 96 | + |
| 97 | +### 3. `write_registry_entities()` + `write_structural_edges()` (`neuro_ghost/db.py`) |
| 98 | + |
| 99 | +For each entity: does a node with this `hash_id` already exist? |
| 100 | +- **No** → create it. |
| 101 | +- **Either way** → attach this ingestion's `ProvenanceEntry`, *unless this |
| 102 | + exact source already has one on that node* (idempotent re-ingestion: running |
| 103 | + the same file from the same source twice adds nothing the second time). |
| 104 | + |
| 105 | +Then `HAS_PROPERTY` and `SUBCLASS_OF` edges get created from the resolved |
| 106 | +hash references. These two functions are shared between `ingest_linkml.py` |
| 107 | +and `seed.py` — schema.org is ingested through the exact same path, just with |
| 108 | +`source="schema.org"`. |
| 109 | + |
| 110 | +## The data model |
| 111 | + |
| 112 | +| Field | On | Notes | |
| 113 | +|---|---|---| |
| 114 | +| `hash_id` | every entity | Content-derived. `RegistryClass`/`RegistryProperty` only — `ProvenanceEntry` uses a random `uid` instead, since it's a per-attestation record, not a deduplicated concept. | |
| 115 | +| `name`, `description` | `RegistryClass`, `RegistryProperty` | Identity-defining (part of the hash). | |
| 116 | +| `range`, `units` | `RegistryProperty` | Identity-defining. | |
| 117 | +| `properties`, `is_a`, `mixins` | `RegistryClass` | Identity-defining — all stored as hash_id references. | |
| 118 | +| `class_uri` / `slot_uri` | `RegistryClass` / `RegistryProperty` | Ontology IRI preserved from the source. **Not** part of the hash — two schemas using different IRIs (or none) for the same content still collapse to one node. | |
| 119 | +| `registry_version` | both | Operational stamp. Not part of the hash. | |
| 120 | +| `provenance` | both | List of `ProvenanceEntry`. Accumulates, never affects `hash_id`. | |
| 121 | +| `source`, `attributed_to`, `generated_at`, `activity`, `derived_from` | `ProvenanceEntry` | PROV-O–grounded (`slot_uri: prov:wasAttributedTo` etc.) | |
| 122 | + |
| 123 | +Field names were deliberately aligned with LinkML's own metamodel |
| 124 | +(`description`, `range`, `class_uri`/`slot_uri`, `is_a`, `abstract`) rather |
| 125 | +than inventing parallel terminology — e.g. `parse_linkml()` already produces |
| 126 | +`is_a` straight from `SchemaView`, so there's no translation step. |
| 127 | + |
| 128 | +**Deliberately not modeled yet:** `required`/`multivalued` used to live on |
| 129 | +`RegistryProperty` directly, which meant a property required in one schema's |
| 130 | +usage and optional in another's could never share a hash. They've been |
| 131 | +removed entirely — that's a **`Rule`** concern (still a stub), not identity. |
| 132 | + |
| 133 | +## Alignment |
| 134 | + |
| 135 | +`neuro_ghost/align.py` runs *after* ingestion, computing a similarity score |
| 136 | +(IRI exact-match + embedding-based name/description similarity) between |
| 137 | +already-distinct `hash_id`s and writing `ALIGNED_TO` edges. It never merges |
| 138 | +identities — that's deliberate for now. Content-hashing already handles |
| 139 | +"these are byte-for-byte the same"; alignment's job is "these are *related* |
| 140 | +but not identical" (`age_years` vs `age_at_scan`), which needs real |
| 141 | +similarity judgment. Until that's built out, ordering it after commit (not |
| 142 | +before, the way some richer designs do) is the correct choice — there's |
| 143 | +nothing yet that could inform the hash before commit anyway. |
| 144 | + |
| 145 | +## Testing: two layers, don't conflate them |
| 146 | + |
| 147 | +`tests/test_ingest_linkml.py` tests two distinct steps against the same |
| 148 | +fixture (`tests/fixtures/comprehensive.yml`, which packs a mixin, an |
| 149 | +abstract base, `is_a` inheritance, both `slots:` and inline `attributes:`, |
| 150 | +prefix resolution from both the schema's own `prefixes:` and the built-in |
| 151 | +fallback map, `multivalued`/`required`/`pattern`, and a units-in-description |
| 152 | +extraction into one schema) — and it matters which one a given assertion is |
| 153 | +about: |
| 154 | + |
| 155 | +- **`parse_linkml()`'s intermediate dict** legitimately has `multivalued`/ |
| 156 | + `required`/`pattern` — that's a raw, general-purpose extraction of what the |
| 157 | + LinkML slot declares, independent of what the registry keeps. |
| 158 | +- **`build_registry_entities()`'s output** (`RegistryProperty`/`RegistryClass`) |
| 159 | + does **not** have those fields at all — `RegistryProperty.model_fields` |
| 160 | + doesn't even define them. Since `hash_id` is a pure content hash (no |
| 161 | + randomness), this layer's test hardcodes the exact expected hash strings — |
| 162 | + reproducible on any machine, and any change to the hash computation, the |
| 163 | + fields carried into the model, or the `is_a`/`properties` resolution shows |
| 164 | + up as a failure here. |
| 165 | + |
| 166 | +Saying "parse_linkml extracts X" and "the registry stores X" are different |
| 167 | +claims — test each layer for what it actually is, not for what you assume |
| 168 | +the other layer does with it. |
| 169 | + |
| 170 | +## Known gaps (as of this writing) |
| 171 | + |
| 172 | +- **`index.html`** (frontend) still expects the pre-rework JSON shape |
| 173 | + (`source` singular; `multivalued`/`required` on properties). A minimal |
| 174 | + compat patch was written and then deliberately reverted — not worth fixing |
| 175 | + until a proper UI pass. |
| 176 | +- **`derived_from`** on `ProvenanceEntry` is never populated — nothing yet |
| 177 | + detects "this hash supersedes that one" (would need an anchor like |
| 178 | + `(name, source)` to correlate an edit against prior content). |
| 179 | +- **`Rule`/`Transform`/`ValueSet`** are stubs (`hash_id`/`name`/`description` |
| 180 | + only). |
| 181 | +- **`SemanticIdentity`/`PRIOR_VERSION*`** tables in `db.py`'s DDL are dead |
| 182 | + (superseded by content-hash identity) but not yet removed. |
| 183 | +- **`pandas`** isn't in `requirements.txt`, so `align.py`'s embedding cache |
| 184 | + silently no-ops (pre-existing gap). |
0 commit comments