scribe-iq-lakehouse

This project takes raw, messy, hospital-style patient data and turns it into one clean, reliable, well-documented dataset that AI systems can safely build on. It's built to run the same way on a laptop or in the cloud, on synthetic (non-real) patient data only.

Production-pattern healthcare data lakehouse on Synthea Coherent (1,278 patients → 1,280 FHIR R4 bundles): a Bronze → Silver → Gold medallion that turns raw multimodal clinical bundles into one governed, versioned, test-gated Gold data contract.

Built twice, on purpose — Polars + delta-rs + DuckDB on a laptop and Spark + Delta + OneLake on Microsoft Fabric — orchestrated as a Dagster asset graph with a streaming-ingest simulation of Fabric's Auto Loader. Two independent, engine-native implementations converge on the same contract by schema parity and a lockstep version, not shared code (ADR-022).

Built with: Polars · Apache Spark · Delta Lake (delta-rs / OneLake) · DuckDB · Dagster · Microsoft Fabric · FHIR R4 · DICOM · AWS Open Data S3 — Python 3.11 · synthetic data, no PHI.

Why it exists. scribe-iq proved the clinical-documentation product on a corpus assembled heuristically (Synthea CSV + public note sets — ACI-Bench, MTSamples, MedSynth). This repo industrializes that foundation the rigorous way; next, a roadmap Ollama loop will generate scribe-iq's next corpus from the Gold contract. Full story in the docs.

Status: Bronze → Silver → Gold fully built and run end-to-end on the complete 1,278-patient dataset on the LocalLite tier (143,946 encounter summaries). DICOM imaging headers ingested. Dagster local orchestration renders the medallion as a software-defined asset graph (a second local execution surface alongside the CLI). The Fabric tier ran green end-to-end on F4 capacity against a 100-patient sample (notebooks 00–10); the full 1,280-bundle re-run is pending. Synthetic data only — no PHI.

flowchart LR
    S3["AWS Open Data S3<br/>Synthea Coherent · FHIR R4<br/>1,278 patients · ~4.6 GiB"]
    subgraph LH["scribe-iq-lakehouse — Bronze → Silver → Gold (built twice)"]
        direction TB
        BR["Bronze<br/>raw, append-only"]
        SV["Silver — 10 typed Delta tables · CDC · validated<br/>LocalLite: Polars + delta-rs   ·   Fabric: Spark from_json"]
        GD["Gold<br/>gold.encounter_summary · 143,946 rows · 1 / encounter"]
        BR --> SV --> GD
    end
    S3 --> BR
    GD ==>|contract v1.1.0 · versioned · test-gated| C
    subgraph C["Downstream AI consumers"]
        direction TB
        BERT["clinical-bert-pipeline · NLP"]
        OLL["Ollama pipeline (roadmap)<br/>→ scribe-iq RAG corpus"]
    end
    classDef plat fill:#eef2ff,stroke:#6366f1;
    classDef cons fill:#f0fdf4,stroke:#22c55e;
    classDef road fill:#fff7ed,stroke:#f59e0b,stroke-dasharray:4 3;
    class LH plat
    class C cons
    class OLL road

Quick start

Requires Python 3.11+ and (for ingest only) the AWS CLI.

python -m venv .venv && source .venv/bin/activate
pip install -e ".[local,dev]"     # core + local-lite (polars/delta-rs/duckdb) + dev tooling
# Optional: add ",orchestration" for the Dagster asset-graph UI (dagster + dagster-webserver)
pytest                            # 129 tests (124 core + 5 fabric contract), fixture-only — no cloud / network

Parse a single FHIR bundle (pure, no I/O):

import json
from core.transforms.fhir_parser import FHIRBundleParser

bundle = json.load(open("tests/fixtures/sample_bundle.json"))
records = FHIRBundleParser().parse_bundle(bundle)
#  -> {"patient": [...], "encounter": [...], "soap_note": [...], "condition": [...], ...}

Run the whole lakehouse locally (downloads ~4.6 GiB FHIR, then builds Silver + Gold):

python -m core.ingest.download --bronze-root data/bronze   # FHIR → Bronze (~18 min, network-bound)
python -m core.surfaces.cli.pipeline --with-gold                        # Bronze → Silver → Gold (~2.5 min)

Full operational procedures — ingest, rebuilds, DICOM, verification, troubleshooting — are in the Runbook.

See one patient flow through the medallion

python -m core.scripts.demo_walkthrough           # auto-picks a good demo patient
python -m core.scripts.demo_walkthrough --patient-id <uuid>

Renders one synthetic patient's journey Bronze → Parse → Silver → Gold in the terminal: FHIR resource-type counts, the raw Patient JSON, parsed records, that patient's typed Silver rows, and finally one gold.encounter_summary row with the SOAP note rendered as readable clinical text. The same data shape is rendered inline in the Dagster asset graph — click any asset and the metadata panel shows schema + sample rows for that materialization.

Query the lakehouse interactively (DuckDB UI)

brew install duckdb                                     # needs ≥1.2 for the -ui flag
duckdb docs/demo/notebooks/demo.duckdb -ui              # opens browser at http://localhost:4213

20-cell notebook over the Delta tables — corpus headlines, top conditions (anemia, hypertension, diabetes), as-of-date condition evolution for one patient, full SOAP notes, keyword cohort search, coverage stats. Pure SQL, no Spark. See docs/demo/notebooks/README.md for setup and the per-cell guide; recording guide is in docs/demo/PLAYBOOK.md.

Operations (common tasks)

Task	Command
Install (local + dev)	`pip install -e ".[local,dev]"`
Install + Dagster orchestration tier	`pip install -e ".[local,dev,orchestration]"`
Download FHIR → Bronze	`python -m core.ingest.download --bronze-root data/bronze`
Download DICOM + CSV (optional, ~10 GB)	`python -m core.ingest.download --assets-only --with-dicom --with-csv`
Build Bronze → Silver → Gold	`python -m core.surfaces.cli.pipeline --with-gold`
Rebuild Gold only (Silver exists)	`python -m core.surfaces.cli.pipeline --gold-only`
Process a single cohort	`python -m core.surfaces.cli.pipeline --cohort A`
Full clean rebuild	`rm -rf data/silver data/gold && python -m core.surfaces.cli.pipeline --with-gold`
Launch Dagster UI (asset graph)	`DAGSTER_HOME="$PWD/dagster_home" dagster dev`
One-patient demo walkthrough	`python -m core.scripts.demo_walkthrough`
Interactive SQL notebook (DuckDB)	`duckdb docs/demo/notebooks/demo.duckdb -ui`
Run tests / lint / format	`pytest` · `ruff check core fabric` · `black core fabric`
Regenerate generated docs	`python core/scripts/gen_data_dictionary.py` · `python core/scripts/gen_corpus_schema.py`

Gotcha — full re-runs need a clean slate. delta-rs MERGE upsert is for incremental per-cohort landing, not whole-table re-update; a full re-run on top of existing tables errors. Remove data/silver + data/gold first (both rebuild from Bronze). See the Runbook → Troubleshooting.

The execution engine is chosen by the LAKEHOUSE_PLATFORM env var (default local_lite); the local storage root is data/ (override with LAKEHOUSE_LOCAL_ROOT). Nothing under data/ is committed.

Architecture at a glance

See ARCHITECTURE.md for the as-built diagram and module map, and the ADRs for why.

The headline decision — two independent, engine-native tiers converging on one governed contract (ADR-022):

flowchart TB
    subgraph CORE["core/ — LocalLite tier (laptop, $0)"]
        direction TB
        C1["Polars + delta-rs + DuckDB"]
        C2["own transforms<br/>core/transforms/silver_*.py → pa.Table"]
        C1 --> C2
    end
    subgraph FAB["fabric/ — Fabric tier (Spark / OneLake)"]
        direction TB
        F1["Spark + Delta + OneLake"]
        F2["own transforms<br/>fabric/transforms/silver_*.py → Spark DataFrame"]
        F1 --> F2
    end
    CONTRACT{{"Gold contract — gold.encounter_summary v1.1.0<br/>schema parity + lockstep CONTRACT_VERSION<br/>(compatibility, NOT shared code)"}}
    C2 --> CONTRACT
    F2 --> CONTRACT
    NOTE["Rejected: one shared transform layer<br/>(lowest-common-denominator + applyInPandas bridge tax) → ADR-022"]
    NOTE -.-> CONTRACT
    classDef contract fill:#eef2ff,stroke:#6366f1,font-weight:bold;
    class CONTRACT contract

Independent per-platform implementations (ADR-022) — each tier owns its complete Silver + Gold + validation stack written engine-native: core/ (LocalLite) transforms return pyarrow.Table (Polars + delta-rs); fabric/ transforms return Spark DataFrames (Spark + OneLake Delta). Cross-tier compatibility is by schema parity + lockstep CONTRACT_VERSION, not code sharing — core/ never imports a platform tier (ADR-017).
Pure transforms, no I/O — transforms under core/transforms/ and core/gold/ (and their fabric/ counterparts) take data and return data: no file paths, no spark.read, no platform imports. The platform layer owns all Delta I/O.
Execution surfaces — the LocalLite tier runs under the dependency-light core.surfaces.cli.pipeline CLI and a Dagster asset graph (ADR-015, ADR-016) with cohort-partitioned backfill and validate_table surfaced as asset checks; the Fabric tier runs under its own notebooks 00–10 (ADR-021).
CDC everywhere — Change Data Feed is enabled on every Silver/Gold Delta table on creation (ADR-009).
Honest data modeling — genomic data_limitation is a first-class column (ADR-007); DICOM headers extracted without pixel data (ADR-006, ADR-013); the Gold problem list is point-in-time, as of each encounter date (ADR-014).
PHI-safe by construction — logs never contain patient/encounter identifiers or bundle filenames; identifier-bearing values are redacted to a non-reversible ref:<hash> (ADR-010).

Data products

Product	What	Contract
`silver.*` (10 tables)	Typed, deduped, CDC-enabled FHIR entities	DATA_DICTIONARY.md (generated)
`gold.encounter_summary`	Denormalized corpus, one row per encounter	CORPUS_CONTRACT.md v1.1.0 + `schemas/gold_encounter_summary.json`
`gold/_metadata/corpus_manifest.json`	Lineage + coverage stats per build	—

The corpus contract is the versioned handoff to downstream consumers; a test fails if the code, the JSON Schema, and the contract ever drift apart (ADR-011). Real run metrics (timings, row counts, coverage) live in BENCHMARKS.md.

Repository layout

The repo is split into two top-level domains — core/ (platform-agnostic + local execution) and fabric/ (Fabric-specific impl + notebooks + deploy). Future Databricks and AWS reference implementations land as siblings of fabric/ (ADR-017, ADR-018, multi-platform-reorg.md).

core/                               ← platform-agnostic kernel; built as a wheel
  platform/      base + LocalLite   ← LakehousePlatform interface + Polars/delta-rs impl
  transforms/    pure FHIR → Silver ← engine-agnostic, return pa.Table
  gold/          encounter_summary  ← denormalized corpus + manifest
  validation/    schema + rules     ← ingest_log
  ingest/        S3 + DICOM         ← Bronze landing
  orchestration/dagster/            ← cohort-partitioned asset graph (local-only, ADR-015/16)
  surfaces/cli/pipeline.py          ← CLI orchestrator
  redaction.py                      ← PHI-safe log refs (ADR-010)
  tests/  scripts/  docs/

fabric/                             ← Fabric tier; consumes `core` wheel via Environment
  platform.py                       ← FabricPlatform — Spark-native, independent (ADR-022)
  notebooks/                        ← Git-Integration-synced to the workspace
  environments/                     ← Fabric Environment spec (wheel + Spark config)
  deploy/                           ← fabric-cicd config + REST upload helper
  data_factory/                     ← Fabric pipeline JSON (when added)
  tests/  scripts/  docs/

databricks/  aws/                   ← future siblings (same shape as fabric/)

.github/workflows/                  ← core-build · core-pr-tests · docs (fabric/databricks/aws-deploy disabled)
docs/adr/  docs/roadmap/            ← ADRs + planning docs
schemas/                            ← machine-readable corpus JSON Schema

One-way dependency rule: fabric/ (and future siblings) import from core/; core/ never imports from any platform tier. Enforced by a CI grep check.

Documentation map

RUNBOOK.md — operational procedures, verification, troubleshooting
ARCHITECTURE.md — as-built design, diagram, module map
DATA_DICTIONARY.md — every Silver column + validation rule (generated)
CORPUS_CONTRACT.md — the Gold handoff contract (v1.1.0)
BENCHMARKS.md — real run metrics + engine matrix
demo/PLAYBOOK.md — recording guide for the demo video
docs/adr/ — 22 Architecture Decision Records (19 active + 3 superseded)
docs/roadmap/ — full spec + cross-repo plan + multi-platform reorg

Testing & quality

pytest                                            # 129 tests (124 core + 5 fabric contract); fixture-only, no cloud/network
ruff check core fabric                            # lint
black --check core fabric                         # format check
python core/scripts/gen_data_dictionary.py --check   # docs-as-test (CI gate)
python core/scripts/gen_corpus_schema.py --check

Tests run against the synthetic core/tests/fixtures/sample_bundle.json — never real patient data. pre-commit adds secret scanning (detect-secrets, gitleaks), security linting (bandit, semgrep), and read-only doc-currency gates.

License

MIT. Built on Synthea Coherent synthetic data — contains no real patient information.

Name		Name	Last commit message	Last commit date
Latest commit History 68 Commits
.claude		.claude
.github/workflows		.github/workflows
.semgrep		.semgrep
core		core
docs		docs
fabric		fabric
schemas		schemas
.env.example		.env.example
.gitignore		.gitignore
.gitleaksignore		.gitleaksignore
.pre-commit-config.yaml		.pre-commit-config.yaml
.secrets.baseline		.secrets.baseline
CHANGELOG.md		CHANGELOG.md
CLAUDE.md		CLAUDE.md
HANDOFF.md		HANDOFF.md
README.md		README.md
mkdocs.yml		mkdocs.yml
pyproject.toml		pyproject.toml
requirements-docs.txt		requirements-docs.txt
requirements.txt		requirements.txt

Navigation Menu

Repository files navigation

scribe-iq-lakehouse

Quick start

See one patient flow through the medallion

Query the lakehouse interactively (DuckDB UI)

Operations (common tasks)

Architecture at a glance

Data products

Repository layout

Documentation map

See also

Testing & quality

License

About

Releases

Packages

Contributors

Languages

Folders and files

Latest commit

History

Repository files navigation

scribe-iq-lakehouse

Quick start

See one patient flow through the medallion

Query the lakehouse interactively (DuckDB UI)

Operations (common tasks)

Architecture at a glance

Data products

Repository layout

Documentation map

See also

Testing & quality

License

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages