Schema enrichment pipeline: from raw data to validated, semantically-rich schemas

[amc-corey-cox]

Vision

Schema Automator currently infers structural schemas from data (ranges, enums, optionality). The next step is enrichment — producing schemas that encode real-world semantics: units, coded value meanings, constraints, and ontology mappings.

Every constraint embedded in the schema is work you don't have to encode in downstream transformation logic. If we detect and embed units in the schema, we don't need to transform metadata into data by specifying it in transformation specs. Richer schemas → better validation → simpler transforms.

Note: Validation itself is handled by the linkml toolchain, not schema-automator. Our job is to produce the richest, most accurate schema possible so that downstream validation and transformation tools have good material to work with.

The pipeline (LLM-independent)

All core functionality works without LLMs:

1. Infer structure from data (current: ranges, enums, optionality, mixed types; planned: units, patterns, better booleans/identifiers)
2. Ingest a structured data dictionary alongside data files (Define structured data dictionary input format #191, Ingest structured data dictionary to enrich inferred schemas #192) — handling for non-canonical / partial / legacy DDs is tracked in Handle non-canonical and legacy data dictionaries #200
3. Enrich the inferred schema with the dictionary's declared metadata
4. Reconcile — surface discrepancies between what the data shows and what the dictionary declares (Reconciliation report: inferred schema vs. declared data dictionary #193)

Two-layer data dictionary strategy

The data dictionary path has two layers:

• Canonical target format (Define structured data dictionary input format #191) — the prescriptive, opinionated format we ask new studies to produce. Ingested directly by Ingest structured data dictionary to enrich inferred schemas #192.
• Non-canonical input handling (Handle non-canonical and legacy data dictionaries #200) — vocabulary normalization, partial-declaration handling, and conflict policy for legacy / messy / partial DDs that don't (or can't) conform to the canonical format. Outputs into the canonical format from Define structured data dictionary input format #191.

LLM-assisted transformation of messy DDs into the canonical format is one option that lives in the #200 layer, alongside hand-written per-source normalizers. Schema-automator's core ingestion path remains LLM-independent.

Sub-issues

Data dictionary path

• Define structured data dictionary input format #191 — Define canonical data dictionary input format (target spec for new studies)
• Handle non-canonical and legacy data dictionaries #200 — Handle non-canonical and legacy data dictionaries (normalization + partial/conflict handling)
• Ingest structured data dictionary to enrich inferred schemas #192 — Ingest structured data dictionary to enrich inferred schemas
• Reconciliation report: inferred schema vs. declared data dictionary #193 — Reconciliation report: inferred schema vs. declared data dictionary
• Evaluate incorporating Schema Sheets #176 — Evaluate incorporating SchemaSheets

Improved inference

• can any of the importers or generalizers infer that 0/1/True/False etc. are of type boolean? #93 — Richer boolean detection (0/1, Yes/No, True/False)
• Schema automator for OMOP: concept_class_id falsely inferred as idenitifier #109 — Fix false-positive identifier heuristic
• Infer units from data values using quantulum3 #194 — Infer units from data values using quantulum3
• Infer pattern constraints from consistent value formats #195 — Infer pattern constraints from consistent value formats

Related work

• Existing `--annotator` flag and `SchemaAnnotator` (OAK-based ontology annotation)
• Existing `llm_annotator.py` (basic LLM description generation)
• Existing `--data-dictionary-row-count` feature (minimal data dictionary support)
• Recent inference flags: `--infer-optional`, `--infer-mixed-types`, `--infer-enum-from-integers`
• Cross-reference: make a data dictionary template dm-bip#103, Trans-spec authoring pipeline overview dm-bip#306, Enrich generated schemas with data dictionary metadata dm-bip#307

[amc-corey-cox]

Cross-reference: linkml/dm-bip#103

Clarification on data dictionary ingestion: We shouldn't expect to parse arbitrary real-world data dictionaries directly — they're too varied and unstructured. A more realistic approach is to use an LLM to transform messy data dictionaries into a conforming structured format that schema-automator can then ingest. The LLM acts as the adapter between the human-readable mess and the computable input.

This also connects to the SchemaSheets question (#176): rather than asking data owners to fill out LinkML-specific spreadsheets, we could accept their existing data dictionaries and use LLM transformation to bridge the gap.