docs: stack-forward homepage, architecture diagrams, provenance, and repo metadata (#5)

* docs: stack-forward homepage + README lead, provenance, reconciled numbers

Lead with the engineering (Polars/Spark/Dagster/Delta/Fabric/streaming) above the comparison table; add a 'Built with' band and a 'Why this exists' provenance section (scribe-iq's heuristic corpus from Synthea CSV + ACI-Bench/MTSamples/MedSynth → this principled rebuild → roadmap Ollama loop), led by ADR-022. Reorder 'What this shows' engineering-first + add the streaming-sim row. Reconcile 1,278 patients / 1,280 bundles and keep the 129-test badge (confirmed via pytest --collect-only). Mark the Ollama loop roadmap.

* docs: architecture/parity/portfolio diagrams; Ollama loop marked roadmap

ARCHITECTURE: system-context (D1) + medallion-with-engines (D2) + Dagster asset graph (D5). parity: dual-engine convergence with the rejected shared-abstraction / applyInPandas bridge-tax note (D3). portfolio: the Gold→Ollama→scribe-iq loop (D4), with prose corrected so it no longer implies scribe-iq's CURRENT corpus came from this lakehouse.

* docs: plain-language lead (layered disclosure) on homepage, README, reviewer guide

ADD-1: a jargon-free 2-sentence lead now precedes the technical hero on docs/index.md (hero moved into a 'For technical reviewers' abstract) and README.md; reviewer-guide gains a 'Not technical?' tier above '90 seconds'. Both audiences served by sequence — the technical depth is unchanged.

* docs(site): tighten ADR-022 framing, roadmap honesty, and number consistency

Review-driven fixes to the docs revamp (PR #5), from a 3-lens "how it lands"
read (recruiter / senior-DE / honesty audit):

- ADR-022 accuracy: drop "the same transforms run three ways." The two
  LocalLite surfaces (CLI + Dagster) share one transform set; the Fabric tier
  reimplements its own engine-native transforms. Fixed index.md and the README
  ASCII flow; amended ADR-015 (it cited the superseded ADR-002/004) and ADR-009.
- Roadmap honesty: CORPUS_CONTRACT intro no longer presents the Ollama ->
  scribe-iq loop as live (clinical-bert is the current consumer; the loop is
  roadmap and does not produce scribe-iq's current corpus). Added the Ollama
  row to the reviewer-guide "what's real vs in-progress" table; standardized
  roadmap wording to "not built" (dropped "in progress").
- Numbers: standardized Bronze->Silver on the measured 2m19s (removed the
  contradictory 2m30s); fixed "1,278-bundle" (a patient count) -> "1,280-bundle"
  across 5 files.
- Tone/scope: softened "proven on real enterprise infrastructure" -> "validated
  end-to-end on Fabric (F4) at sample scale"; dropped the dead local_spark
  column and the non-ADR-backed GCP column from the engine matrix; added a
  "New here?" cue to the homepage 90-second-tour card.

mkdocs build --strict green; gen_data_dictionary/gen_corpus_schema --check pass.

Author: sandeep-jay

README.md

@@ -1,28 +1,42 @@
 # scribe-iq-lakehouse
 
+This project takes raw, messy, hospital-style patient data and turns it into one clean, reliable,
+well-documented dataset that AI systems can safely build on. It's built to run the same way on a
+laptop or in the cloud, on synthetic (non-real) patient data only.
+
 Production-pattern healthcare data lakehouse on [Synthea Coherent](https://registry.opendata.aws/synthea-coherent-data/)
-(1,278 synthetic patients, FHIR R4). **Medallion** architecture — Bronze → Silver → Gold —
-implemented as **two independent, engine-native tiers** that emit the same Gold contract: a
-LocalLite tier (`core/` — Polars + delta-rs, runs on a laptop) and a Fabric tier (`fabric/` —
-Spark + OneLake). The Gold corpus (`gold.encounter_summary`) feeds
-[`scribe-iq`](https://sandeep-jay.github.io/scribe-iq/) (clinical RAG),
-`clinical-bert-pipeline` (NLP), and an Ollama dialogue-generation pipeline.
+(1,278 patients → 1,280 FHIR R4 bundles): a **Bronze → Silver → Gold medallion** that turns raw
+multimodal clinical bundles into one governed, **versioned, test-gated** Gold data contract.
+
+**Built twice, on purpose** — **Polars + delta-rs + DuckDB** on a laptop and **Spark + Delta +
+OneLake** on **Microsoft Fabric** — orchestrated as a **Dagster** asset graph with a
+**streaming-ingest simulation** of Fabric's Auto Loader. Two independent, engine-native
+implementations converge on the *same* contract by schema parity and a lockstep version, not
+shared code ([ADR-022](docs/adr/022-platform-independent-implementations.md)).
+
+**Built with:** Polars · Apache Spark · Delta Lake (delta-rs / OneLake) · DuckDB · Dagster ·
+Microsoft Fabric · FHIR R4 · DICOM · AWS Open Data S3 — Python 3.11 · synthetic data, no PHI.
+
+**Why it exists.** `scribe-iq` proved the clinical-documentation product on a corpus assembled
+heuristically (Synthea CSV + public note sets — ACI-Bench, MTSamples, MedSynth). This repo
+industrializes that foundation the rigorous way; next, a **roadmap** Ollama loop will generate
+`scribe-iq`'s next corpus from the Gold contract. Full story in the
+[docs](https://sandeep-jay.github.io/scribe-iq-lakehouse/portfolio/).
 
 **Status:** Bronze → Silver → **Gold** fully built and run end-to-end on the complete
 1,278-patient dataset on the LocalLite tier (143,946 encounter summaries). DICOM imaging
 headers ingested. **Dagster** local orchestration renders the medallion as a software-defined
 asset graph (a third local execution surface alongside the CLI). The **Fabric tier** ran green
-end-to-end on F4 capacity against a 100-patient sample (notebooks 00–10); the full 1,278-bundle
+end-to-end on F4 capacity against a 100-patient sample (notebooks 00–10); the full 1,280-bundle
 re-run is pending. Synthetic data only — **no PHI**.
 
 ```
- AWS Open Data S3            Bronze (raw)              Silver (10 Delta tables)        Gold
- coherent/unzipped/   ──►   fhir/ · dicom/ · csv/  ──► patient · encounter · …    ──►  encounter_summary
- (no credentials)           + manifests               condition · observation …       (1 row / encounter)
-                                                       CDC enabled, validated          + corpus_manifest.json
-                                                                                       └─► scribe-iq · BERT · Ollama
+ AWS Open Data S3      Bronze (raw, append-only)       Silver (10 typed Delta tables)        Gold
+ coherent/  ─►  streaming_sim ─►  fhir· dicom· csv  ─►  Polars + delta-rs (local)         ─►  gold.encounter_summary
+ (no creds)     (Auto Loader sim)  + manifests           / Spark from_json (Fabric)            1 row/encounter · contract v1.1.0
+                                                          CDC · validated (Dagster checks)     └─► clinical-bert · scribe-iq via Ollama (roadmap)
 
-  execution surfaces (same pure transforms): CLI · Dagster (core/orchestration/dagster/) · Fabric notebooks
+  local surfaces share one transform set: CLI · Dagster asset graph (core/orchestration/dagster/)  |  Fabric tier reimplements its own (ADR-022)
 ```
 
 ---

docs/ARCHITECTURE.md

@@ -18,29 +18,29 @@ CLI or a **Dagster** software-defined asset graph (ADR-015/016); on Fabric it ra
 green end-to-end on F4 capacity against a 100-patient demo sample via notebooks 00–10.
 
 ```mermaid
-flowchart TD
-    S3["AWS Open Data<br/>s3://synthea-open-data/coherent<br/>(1,280 FHIR bundles, 4.6 GB)"]
-
-    subgraph BRONZE["BRONZE — raw, append-only"]
-        B["data/bronze/fhir/cohort=A|B|C/*.json<br/>+ dicom/*.dcm (headers) + csv/ (reference)<br/>+ _metadata/manifest.json"]
-    end
-
-    subgraph SILVER["SILVER — Delta, CDC enabled ✅ built"]
-        ST["10 tables: patient · encounter · condition · observation<br/>medication_request · procedure · soap_note · ecg_metadata<br/>imaging_study · genomic_report  (+ ingest_log audit)"]
+flowchart LR
+    SRC["AWS Open Data S3<br/>Synthea Coherent · FHIR R4<br/>1,278 patients · ~4.6 GiB"]
+    subgraph LH["scribe-iq-lakehouse — the data platform"]
+        direction TB
+        BR["Bronze<br/>raw · append-only"]
+        SV["Silver<br/>10 typed Delta tables · CDC · validated"]
+        GD["Gold<br/>gold.encounter_summary<br/>143,946 rows · 1 / encounter"]
+        BR --> SV --> GD
     end
-
-    subgraph GOLD["GOLD — denormalized corpus ✅ built"]
-        G["gold.encounter_summary (143,946 rows, 1/encounter)<br/>+ _metadata/corpus_manifest.json"]
+    SRC --> BR
+    GD == "corpus contract v1.1.0<br/>versioned · test-gated" ==> CON
+    subgraph CON["Downstream AI consumers"]
+        direction TB
+        SIQ["scribe-iq<br/>clinical RAG / docs"]
+        BERT["clinical-bert-pipeline<br/>NLP"]
+        OLL["Ollama pipeline (roadmap)<br/>note + dialogue generation"]
     end
-
-    S3 -->|"download.py · aws s3 sync"| B
-    B -->|"pipeline.py · per-cohort micro-batch<br/>parse → build → MERGE"| ST
-    ST -->|"build_gold · Polars denormalize → overwrite"| G
-    G -.->|"corpus contract v1.1.0"| DS["scribe-iq (RAG)<br/>clinical-bert-pipeline (NLP)<br/>Ollama generation"]
-
-    classDef done fill:#d4edda,stroke:#28a745;
-    classDef planned fill:#fff3cd,stroke:#ffc107,stroke-dasharray:4 3;
-    class B,ST,G done;
+    classDef plat fill:#eef2ff,stroke:#6366f1;
+    classDef cons fill:#f0fdf4,stroke:#22c55e;
+    classDef road fill:#fff7ed,stroke:#f59e0b,stroke-dasharray:4 3;
+    class LH plat
+    class CON cons
+    class OLL road
 ```
 
 The LocalLite tier runs under **two local execution surfaces** (ADR-015/016): the
@@ -59,6 +59,67 @@ both render via [`core/preview.py`](https://github.com/sandeep-jay/scribe-iq-lak
 shape appears in the UI, the terminal, and the SQL notebook — one set of
 renderers, three audiences. Recording guide: [`docs/demo/PLAYBOOK.md`](demo/PLAYBOOK.md).
 
+## Medallion, with the engine at every hop
+
+Bronze ingest (S3 → append-only; `streaming_sim.py` replays cohorts as an Auto Loader analogue)
+→ Silver (one pure-Python FHIR parse fans out to both tiers — Polars + delta-rs locally, Spark
+`from_json(BUNDLE_SCHEMA)` on Fabric — into 10 typed, CDC-enabled Delta tables, validated by
+`validate_table`) → Gold (Polars join/agg → one `gold.encounter_summary` under the versioned
+contract).
+
+```mermaid
+flowchart TB
+    S3["AWS Open Data S3<br/>FHIR bundles"]
+    STREAM["streaming_sim.py<br/>cohort replay · Auto Loader pattern"]
+    S3 --> STREAM --> BRONZE
+
+    subgraph BRONZE["Bronze — raw, append-only"]
+        B1["fhir/ · dicom/ · csv/ + manifests"]
+    end
+
+    subgraph SILVER["Silver — 10 typed Delta tables · CDC · validated"]
+        direction LR
+        PARSE["pure-Python FHIRBundleParser<br/>(engine-agnostic dicts)"]
+        LOCALS["LocalLite: Polars + delta-rs"]
+        FABS["Fabric: Spark from_json(BUNDLE_SCHEMA)<br/>distributed"]
+        PARSE --> LOCALS
+        PARSE --> FABS
+    end
+
+    subgraph GOLD["Gold — one governed contract"]
+        G1["gold.encounter_summary<br/>Polars join/agg · 1 row / encounter<br/>contract v1.1.0"]
+    end
+
+    BRONZE --> SILVER --> GOLD
+    VAL["validate_table → Dagster asset checks"] -.-> SILVER
+
+    classDef gold fill:#fff7ed,stroke:#f59e0b;
+    class GOLD gold
+```
+
+## Dagster asset graph (ADR-015/016)
+
+The LocalLite tier also renders the medallion as a **software-defined asset graph** — cohorts
+become partitions, one parse-once `@multi_asset` fans out to 10 Silver nodes, and `validate_table`
+is surfaced as an `@asset_check` (rule-by-rule pass/fail in the UI).
+
+```mermaid
+flowchart LR
+    subgraph BRONZE_G["bronze (cohort-partitioned)"]
+        BF["bronze_fhir<br/>inventory per cohort"]
+    end
+    subgraph SILVER_G["silver (multi-asset: parse-once → 10 nodes)"]
+        S1["patient"]; S2["encounter"]; S3["condition"]; S4["observation"]; S5["… 6 more"]
+    end
+    subgraph GOLD_G["gold"]
+        GA["gold_encounter_summary<br/>+ corpus_manifest.json"]
+    end
+    BF --> S1 & S2 & S3 & S4 & S5
+    S1 & S2 & S3 & S4 & S5 --> GA
+    AC["@asset_check: validate_table"] -.-> S1
+    AC -.-> S2
+```
+
 ## Layers (as-built)
 
 | Layer | State | Storage | Notes |
@@ -67,7 +128,7 @@ renderers, three audiences. Recording guide: [`docs/demo/PLAYBOOK.md`](demo/PLAY
 | Silver | ✅ built (local) | 10 Delta tables + `ingest_log` | CDC enabled; validated; MERGE-upsert per cohort |
 | Gold | ✅ built (local) | `encounter_summary` Delta + manifest | 1 row/encounter; CDC; as-of-date problem list; corpus contract v1.1.0 (ADR-012/014) |
 | Dagster orchestration | ✅ built (local) | `core/orchestration/dagster/` package | medallion as asset graph; cohort partitions; `validate_table` as asset checks (ADR-015/016) |
-| Fabric execution | ✅ green end-to-end (F4, SAMPLE_SIZE=100) | OneLake | independent Spark-native impl (ADR-022); notebooks 00–10; anonymous S3 ingest in 01; full 1,278-bundle re-run pending |
+| Fabric execution | ✅ green end-to-end (F4, SAMPLE_SIZE=100) | OneLake | independent Spark-native impl (ADR-022); notebooks 00–10; anonymous S3 ingest in 01; full 1,280-bundle re-run pending |
 
 ## Module map
 
@@ -129,7 +190,7 @@ docs/demo/
 
 ## Current scale (full local run)
 
-1,280 bundles (1,278 patients) → 10 Silver Delta tables in **~2m30s**, then →
+1,280 bundles (1,278 patients) → 10 Silver Delta tables in **2m19s**, then →
 **143,946** `gold.encounter_summary` rows in **~6.5s** on a single laptop, all validations passing.
 Per-table counts, corpus coverage, and methodology: [BENCHMARKS.md](BENCHMARKS.md). Operational
 procedures: [RUNBOOK.md](RUNBOOK.md). The Gold corpus contract is documented in

docs/BENCHMARKS.md

@@ -90,13 +90,13 @@ Delta table (overwrite + CDC) plus the corpus manifest.
 Independent engine-native tiers emitting the same Gold contract (ADR-022). LocalLite is measured
 on the full dataset; the Fabric tier ran green on F4 capacity against a 100-patient sample.
 
-| Capability | local_lite | local_spark | Fabric | Databricks | AWS | GCP |
-|------------|-----------|-------------|--------|------------|-----|-----|
-| Bronze→Silver (full) | ✅ 2m30s | — | ✅ F4 (100-sample) | roadmap | roadmap | roadmap |
-| Silver→Gold (full) | ✅ ~6.5s | — | ✅ F4 (100-sample) | roadmap | roadmap | roadmap |
-| CDC | ✅ | — | ✅ | roadmap | roadmap | roadmap |
-| Streaming | sim only | — | 🔜 Auto Loader | roadmap | roadmap | roadmap |
-| Cost (1.3k pts) | $0 | $0 | trial | — | — | — |
+| Capability | local_lite | Fabric | Databricks | AWS |
+|------------|-----------|--------|------------|-----|
+| Bronze→Silver (full) | ✅ 2m19s | ✅ F4 (100-sample) | roadmap | roadmap |
+| Silver→Gold (full) | ✅ ~6.5s | ✅ F4 (100-sample) | roadmap | roadmap |
+| CDC | ✅ | ✅ | roadmap | roadmap |
+| Streaming | sim only | 🔜 Auto Loader | roadmap | roadmap |
+| Cost (1.3k pts) | $0 | trial | — | — |
 
 ## Execution surfaces
 
@@ -147,5 +147,5 @@ python -m core.surfaces.cli.pipeline --gold-only                        # rebuil
 - `local_lite` holds one cohort's records in memory at a time (~1/3 of the data); peak
   RSS stayed well under what a typical dev laptop offers. Full-dataset-in-memory was
   deliberately avoided.
-- Fabric/Spark full-run figures will be filled in after the full 1,278-bundle re-run; the
+- Fabric/Spark full-run figures will be filled in after the full 1,280-bundle re-run; the
   100-patient F4 run is green end-to-end (notebooks 00–10).

docs/CORPUS_CONTRACT.md

@@ -4,9 +4,10 @@
 
 This is the handoff interface between the lakehouse and its downstream AI consumers:
 
-- **Ollama generation pipeline** — grounds synthetic clinical dialogue on each summary.
-- **scribe-iq** (RAG) — replaces the 19-patient dev corpus with this 1,278-patient corpus.
-- **clinical-bert-pipeline** (NLP) — consumes `soap_note_text` + structured labels.
+- **clinical-bert-pipeline** (NLP) — consumes `soap_note_text` + structured labels from the contract.
+- **Ollama generation pipeline** *(roadmap)* — will derive synthetic notes/dialogue from each summary.
+- **scribe-iq** (RAG) *(roadmap loop)* — its next corpus will be that Ollama-generated text, superseding
+  its current heuristic 19-patient dev corpus (which this lakehouse does not yet produce).
 
 One row = one clinical encounter. The machine-readable schema is
 [`schemas/gold_encounter_summary.json`](https://github.com/sandeep-jay/scribe-iq-lakehouse/blob/main/schemas/gold_encounter_summary.json) (JSON

docs/RUNBOOK.md

@@ -62,7 +62,7 @@ python -m core.ingest.download --bronze-root data/bronze
 # 3. (Optional) Land DICOM + CSV assets (~10 GB; enables imaging header extraction)
 python -m core.ingest.download --assets-only --with-dicom --with-csv
 
-# 4. Build Bronze → Silver → Gold (~2.5 min Silver + ~6.5 s Gold)
+# 4. Build Bronze → Silver → Gold (~2m19s Silver + ~6.5 s Gold)
 python -m core.surfaces.cli.pipeline --with-gold
 
 # 5. Verify (see §5)

docs/adr/009-local-silver-materialization.md

@@ -49,7 +49,8 @@ real Coherent data forced:
 ## Consequences
 
 **Positive:**
-- Full local pipeline runs with `pip install` only; same transforms target Fabric later.
+- Full local pipeline runs with `pip install` only; the Fabric tier later targets the same
+  *contract* (its own engine-native transforms, per ADR-022).
 - CDC enabled from creation — downstream change-feed consumers work on day one.
 - Flat, portable Silver schemas; Gold can still recover BP via `components_json`.
 

docs/adr/015-dagster-local-orchestration.md

@@ -47,8 +47,11 @@ the persistence pattern are specified separately in [ADR-016](016-dagster-asset-
 - Per-cohort **partitioned materialization + backfill** replaces the `rm -rf` full rebuild:
   each cohort is the incremental MERGE path that actually works (ADR-016).
 - Validation surfaces as first-class **asset checks**; run history/metadata live in the UI.
-- The same pure transforms now demonstrably run under **three orchestrators** (CLI, Dagster,
-  Fabric Data Factory) — the concrete payoff of ADR-002 (platform isolation) and ADR-004 (Arrow).
+- The LocalLite pure transforms run under **two local orchestrators** (CLI, Dagster) with zero
+  duplicated logic — the concrete payoff of platform isolation and the Arrow interchange type.
+  *Amended per [ADR-022](022-platform-independent-implementations.md): the Fabric tier is a separate
+  engine-native implementation, not a third orchestrator over the same transforms; the originally
+  cited ADR-002/004 are superseded by ADR-022.*
 
 **Negative:**
 - New dev dependency (`dagster` + `dagster-webserver`) and a UI process to run.