Enum derivations pipeline with auto-generated specs

[Sigfried]

Summary

Proves enum_derivations work end-to-end through the dm-bip pipeline (issue #211), with the proof automated in tests/.

The from_raw toy pipeline gains a single twin column. SEX_CODE (string M/F) is added to pht000001 as a semantic duplicate of the existing integer SEX column. Demography.sex is still populated via value_mappings (unchanged); the new Demography.sex_derived slot is populated via enum_derivations from SEX_CODE. Both resolve to the same target_sex_enum permissible values, and an integration test asserts they match row-for-row. The existing value_mappings pathway stays the comparison baseline; enum_derivations is exercised as an additive capability against the same data — no parallel fixtures.

Changes

• toy_data/ fixture — SEX_CODE column on pht000001; Demography.sex_derived slot (range target_sex_enum) in the target schema; sex_derived slot derivation + top-level enum_derivations block in from_raw/specs/demography.yaml.
• toy_data/from_raw/config.mk — DM_MAX_ENUM_SIZE := 3, the narrowest bound that lets SEX_CODE (2 distinct values) be inferred as a source enum while keeping SMOKING (5 distinct values) a non-enum string so its value_mappings keep working.
• src/dm_bip/map_data/compose_specs.py — preserve top-level enum_derivations blocks during spec composition; they were previously dropped silently, making enum derivations invisible to linkml-map.
• tests/integration/test_from_raw_pipeline.py — test_enum_derivation_twin_matches_value_mapping: asserts sex == sex_derived on every row.
• Dependencies — linkml>=1.11.0, schema-automator>=0.5.5, linkml-map git-pinned (see Staging).

Staging

linkml-map is pinned to a git ref pending an upstream release that carries the CLI --target-schema support and the delimited-loader forwarding this pipeline needs. The PR stays draft + staged until that release; the pin then bumps to a version constraint and the label drops. Upstream context: linkml/linkml-map#235; this mirrors the same maneuver in linkml/schema-automator#211.

Test plan

• make test — 146 tests pass (incl. the new twin-column assertion)
• make lint — clean
• make pipeline CONFIG=toy_data/from_raw/config.mk — sex and sex_derived match across all 110 Demography records
• Re-review / dismiss the stale CHANGES_REQUESTED before un-drafting
• Bump linkml-map pin to a released version once available

Origin & direction

This started as the enum-derivations work for #211 and changed shape along the way; the original intent is recorded here so the history stays legible.

Originally: the branch auto-generated enum_derivations specs from existing value_mappings specs via a new generate_enum_specs.py, wired through a generate-enum-specs Make target gated on DM_ENUM_DERIVATIONS=true, and proved it with a self-contained toy_data_w_enums/ directory that duplicated the toy data to run two parallel pipelines (config-orig-valmaps.mk vs config-enums.mk), plus a docs/pipeline-steps.md comparison reference. Dependencies were carried via [tool.uv.sources] pointing at local forks set up by a scripts/setup-enum-forks.sh script.

Why it changed: #211's goal is narrowly to prove enum_derivations work end-to-end with the proof in tests/. The spec auto-generator was a separable concern; the parallel duplicated fixture, planning docs, fork-cloning script, and embedded reproduction project buried the actual change and were flagged in review. The redirected approach folds the comparison into the existing from_raw fixture as one twin column — same data, both mechanisms, one assertion — and relies on released linkml/schema-automator with only linkml-map git-pinned. The salvageable substance (enum_derivations Make/compose plumbing, a working end-to-end enum test) is kept; the scaffolding is not.

[amc-corey-cox]

@Sigfried I hope this doesn't offend but it looks like this PR got pulled off-course by the LLM. The goal of #211 is proving enum_derivations work end-to-end in the pipeline with proof automated in tests/.

generate_enum_specs.py is a useful tool but it's a separate concern — split it into its own PR and we'll get this core part in separately.

There's a lot of material here that doesn't belong in the repo: planning docs, an embedded reproduction project, an AI-generated reference doc, a fork-cloning shell script. These bury the actual work and make the PR hard to review.

The dependency situation ([tool.uv.sources] pointing at local filesystem paths, unpinned deps with TODO comments) and the .gitignore regression are merge blockers — please see #290 for how I did it there.

Generally, you should also strip any descriptive comments the LLM is throwing in - that's just noise.

[Sigfried]

@ccox-work Cleaned up most of the review feedback in 2a6c980:

• Removed generate_enum_specs.py, issue-211-planning.md, linkml-int-enum-repro/, setup script, enum_test/
• Switched [tool.uv.sources] from local filesystem paths to git URL sources (pinned to commit revs)
• Restored output/ to .gitignore, removed local clone entries
• Removed DM_ENUM_DERIVATIONS flag and generate-enum-specs target from pipeline.Makefile; map-data now uses DM_MAP_TARGET_SCHEMA and DM_TRANS_SPEC_DIR directly
• Config files point at committed specs (with_enum_derivations/ and with_value_mappings/ subdirs)
• Stripped generated comments from enum spec YAMLs
• Rewrote test_from_enum_pipeline.py with enum-specific assertions
• Updated docs and README

Blocker: uv sync fails with git URL sources

The linkml fork's pyproject.toml uses uv-dynamic-versioning with fallback-version = "0.0.0". When uv builds it from a git URL, it can't resolve git tags for versioning and falls back to 0.0.0. This causes a transitive dependency resolution failure:

schema-automator depends on linkml>=1.9.1,<2.0.0
linkml (from git source) resolves to version 0.0.0
→ no solution

Even after that's resolved with override-dependencies = ["linkml>=0"], the same pattern cascades to linkml-runtime (schema-automator also requires linkml-runtime>=1.9.2,<2.0.0).

This didn't happen with the local editable installs because uv could read the git history directly and compute a proper version.

Options I see:

1. Tag the fork branches (e.g., v1.10.0-sa-loader) so uv-dynamic-versioning produces a valid version from the git URL
2. Change the fork's fallback-version from "0.0.0" to something like "1.10.0.dev0"
3. Use a different pinning approach you may know about from your experience with Replace map_data.py with linkml-map CLI (#275) #290

Happy to go whichever direction you prefer.

[amc-corey-cox]

Use PEP 440 direct references in [project.dependencies] instead of [tool.uv.sources]. This sidesteps version resolution entirely:

[project.dependencies]
linkml @ git+https://github.com/Sigfried/linkml.git@<commit-or-branch>
schema-automator @ git+https://github.com/Sigfried/schema-automator.git@<commit-or-branch>
linkml-map @ git+https://github.com/Sigfried/linkml-map.git@<commit-or-branch>

Remove the [tool.uv.sources] section and any override-dependencies. See #290's pyproject.toml for the pattern.

[amc-corey-cox]

This is definitely more complicated for your situation. You may have to make a test branch in schema-automator or linkml-map, or both, with the dependencies for linkml from your branch there in order to push through this. I'm not really sure... but that is what I would try.

[Sigfried]

@amc-corey-cox, is there anything you're waiting for me to do on this? I think I addressed your previous comments. At this point main may have changed in ways that require more conflicts to be resolved

[Sigfried]

Handoff status (2026-05-14)

Corey, per our chat — here's where this stands in case you decide to pick it up.

What's on this branch right now

The branch is bddf25b (just pushed): a fresh commit that drops the linkml and schema-automator git pins now that the upstream changes are released:

• linkml>=1.11.0 (PR Make delimited file loader schema-aware to preserve string/enum columns linkml#3289 merged, released in v1.11.0)
• schema-automator>=0.5.5 (PR Add --infer-enum-from-integers flag for integer enum detection schema-automator#188 merged, released in v0.5.5)
• linkml-map still git-pinned — PR Forward schema_path/target_class to linkml's delimited file loader linkml-map#235 is open, not yet released

What's salvageable vs obsolete after your f215a1d rewrite

Salvageable (the substantive enum-derivations work):

• pipeline.Makefile: DM_INFER_ENUM_FROM_INTEGERS Makefile variable + wiring to schemauto generalize-tsvs --infer-enum-from-integers
• toy_data_w_enums/: full test fixture directory with two parallel spec variants (with_value_mappings/ vs with_enum_derivations/) and matching target schemas
• tests/integration/test_from_enum_pipeline.py
• docs/pipeline-steps.md

Obsolete (replaced by your linkml-map CLI flow):

• All edits to src/dm_bip/map_data/map_data.py (the schema_path/target_class plumbing was needed because we were calling linkml-map's loaders directly; your compose_specs.py + CLI invocation in pipeline.Makefile makes those edits unnecessary — the schema is passed via -s at the CLI level, and once linkml-map#235 lands, the CLI will propagate it to the loaders)
• streams.py (already deleted on main)

Merge state

The branch is 14 commits behind main and has unresolved conflicts in pyproject.toml, README.md, uv.lock, map_data.py (modify/delete), and a binary .txt.gz. Given the architectural shift, a fresh branch off main with cherry-picked fixtures + the Makefile wiring is probably easier than a conflict-by-conflict merge. I'd estimate that's an afternoon's work.

I'll get back to it next week unless you take it over. Either way, this branch has everything you'd need to reconstruct it.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 80.00000% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 79.74%. Comparing base (a65ec29) to head (bbb4482).

Files with missing lines	Patch %	Lines
src/dm_bip/map_data/compose_specs.py	80.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #291      +/-   ##
==========================================
- Coverage   79.87%   79.74%   -0.13%     
==========================================
  Files           9        9              
  Lines         626      632       +6     
==========================================
+ Hits          500      504       +4     
- Misses        126      128       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.