Pipeline & Pilot Data Release

[amc-corey-cox]

Upstream Tool Releases

linkml-map → v0.4.0

• Land cross-table lookup
• Land coerce numeric strings in comparisons
• Cut RC, bake, then final release

schema-automator → new release

• Evaluate open PRs (#180, #175, #173) — land or close
• Evaluate whether any_of range support is needed for pilot schemas
• Cut RC, bake, then final release

dm-bip Pipeline Updates

• Land parallel validation
• StreamWriter migration (Migrate stream writers to linkml-map StreamWriter classes #253) and MultiStreamWriter (Use MultiStreamWriter for simultaneous multi-format output #256)
• Update toolchain dependencies (linkml-map, schema-automator) to new releases
• Tag Docker branches (docker-dev, docker-push-7bridges) with release version
• Add pipeline version logging via environment variable
• Cut dm-bip release

Trans-Specs & Configuration

• Finalize trans-specs in NHLBI-BDC-DMC-HV
• Release bdc-harmonized-variables v0.1.0
• Merge FHS config split in bdc-pilot-delivery
• Cut bdc-pilot-delivery release

BDC Cloud Integration Testing

• Test upstream tools, configuration, and deployment on BDC cloud (see comments for detailed test plan)

End-to-End Validation

• Run full pipeline on all pilot cohorts with finalized toolchain and trans-specs
• Validate all consent groups are ingested (Ensure all Cohort groups are ingested #221)
• Test longitudinal transformation specs (Test new longitudinal transformation specs from HV repo #252)

[amc-corey-cox]

Related issues in this repo:

Validation (schema-automator)

• Investigate: Schema-automator generated schemas fail validation on mixed string/integer columns #232 — Schema-automator fails validation on mixed string/integer columns (the any_of range need)
• Track: Schema-automator should optionally mark slots as optional for columns with null values #233 — Schema-automator should mark slots as optional for null-value columns

Upstream features (linkml-map)

• LinkML-Map needs to be able to make GUIDs #235 — LinkML-Map GUIDs (uuid5 — already in v0.4.0-rc1)
• Migrate stream writers to linkml-map StreamWriter classes #253 — Migrate stream writers to linkml-map StreamWriter classes
• Use MultiStreamWriter for simultaneous multi-format output #256 — Use MultiStreamWriter for multi-format output (depends on Migrate stream writers to linkml-map StreamWriter classes #253)

Docker / deployment

• Remove temp map_data.py code under docker-push-7bridges #269 — Remove temp map_data.py code under docker-push-7bridges
• Create GitHub workflow to container registry #201 — Create GitHub workflow to container registry

Release process

• Update Dependencies and Create Release Strategy #180 — Update Dependencies and Create Release Strategy

[amc-corey-cox]

Schema-automator priorities for this release: The existing PRs (#180, #175, #173) are OWL/delimiter fixes — not critical for the pilot. The real priorities are:

• Investigate: Schema-automator generated schemas fail validation on mixed string/integer columns #232 — Fix mixed string/integer validation (any_of range support)
• Track: Schema-automator should optionally mark slots as optional for columns with null values #233 — Optional slots for null-value columns (upstream: schema-automator#179)

[amc-corey-cox]

BDC Cloud Integration Test Plan

These can be run in parallel as pieces become ready — they don't need to be sequential:

1. Test docker BDC deployment branch — verify the current docker-push-7bridges branch works with its defaults on BDC
2. Test new trans-specs — update the BDC docker branch to pull trans-specs from bdc-harmonized-variables and verify they work
3. Test bdc-pilot-delivery configs — switch deployment to use the bdc-pilot-delivery configuration repo and verify
4. Test toolchain RCs — update dm-bip to linkml-map and schema-automator release candidates and verify on BDC
5. dm-bip release candidate — combine all tested changes into a dm-bip RC, test end-to-end on BDC

Each of these validates a different axis of the release in the actual deployment environment. Parallel validation (PR #264) should also be verified on BDC as part of steps 4/5.

Target: land everything by end of next week (2026-03-13).

[amc-corey-cox]

Remaining for final release (post v0.1.0-rc1)

• Bump upstream pins to final releases (linkml-map, schema-automator, bdc-harmonized-variables) — or update tag if versions don't change
• Add build provenance: write github.sha to BUILD_COMMIT in CI, add RUN date -u > /BUILD_TIMESTAMP to Dockerfile
• Restore docker-push-7bridges branch from main for Chris's prod deploys

[amc-corey-cox]

v1.1.0 released on 2026-03-16. All release items completed or deferred to future work.