Skip to content

Commit f3dca26

Browse files
authored
docs(readme): accuracy fixes + architecture diagrams (#6)
* docs(readme): correct as-built inaccuracies and a stale Fabric reference - Dagster is the *second* local execution surface, not the third — matches the "two LocalLite surfaces (CLI + Dagster)" framing used across the site docs. - Repo map: FabricPlatform is Spark-native and independent (ADR-022), not a LakehousePlatform subclass (fabric/platform.py is `class FabricPlatform:`); dropped the internal "Session 5" annotation that leaked into the public README. - Repo map: workflow list now reflects as-built CI (core-build / core-pr-tests / docs); the *-deploy workflows are disabled stubs (.disabled). - ~4.6 GB -> ~4.6 GiB to match the published docs. - RUNBOOK: replaced the stale "Fabric procedures land ... in Session 5" with a link to the Fabric Deployment guide (the tier is built and ran green). mkdocs build --strict green; gen-doc --check gates pass. * docs(readme): add mermaid architecture diagrams (system context + ADR-022 parity) GitHub renders mermaid natively, so the README front door now carries the two highest-signal diagrams from the docs site: - Hero (replaces the ASCII flow): system context — S3 -> Bronze/Silver/Gold with engines named -> the versioned contract -> downstream AI consumers (clinical-bert live; Ollama -> scribe-iq roadmap). - Architecture section: the ADR-022 dual-engine parity — two independent engine-native tiers converging on one contract, with the rejected shared-transform layer (applyInPandas bridge tax) called out. Diagrams mirror the published site; mermaid kept conservative for GitHub rendering.
1 parent c055849 commit f3dca26

2 files changed

Lines changed: 55 additions & 12 deletions

File tree

README.md

Lines changed: 53 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -26,17 +26,33 @@ industrializes that foundation the rigorous way; next, a **roadmap** Ollama loop
2626
**Status:** Bronze → Silver → **Gold** fully built and run end-to-end on the complete
2727
1,278-patient dataset on the LocalLite tier (143,946 encounter summaries). DICOM imaging
2828
headers ingested. **Dagster** local orchestration renders the medallion as a software-defined
29-
asset graph (a third local execution surface alongside the CLI). The **Fabric tier** ran green
29+
asset graph (a second local execution surface alongside the CLI). The **Fabric tier** ran green
3030
end-to-end on F4 capacity against a 100-patient sample (notebooks 00–10); the full 1,280-bundle
3131
re-run is pending. Synthetic data only — **no PHI**.
3232

33-
```
34-
AWS Open Data S3 Bronze (raw, append-only) Silver (10 typed Delta tables) Gold
35-
coherent/ ─► streaming_sim ─► fhir· dicom· csv ─► Polars + delta-rs (local) ─► gold.encounter_summary
36-
(no creds) (Auto Loader sim) + manifests / Spark from_json (Fabric) 1 row/encounter · contract v1.1.0
37-
CDC · validated (Dagster checks) └─► clinical-bert · scribe-iq via Ollama (roadmap)
38-
39-
local surfaces share one transform set: CLI · Dagster asset graph (core/orchestration/dagster/) | Fabric tier reimplements its own (ADR-022)
33+
```mermaid
34+
flowchart LR
35+
S3["AWS Open Data S3<br/>Synthea Coherent · FHIR R4<br/>1,278 patients · ~4.6 GiB"]
36+
subgraph LH["scribe-iq-lakehouse — Bronze → Silver → Gold (built twice)"]
37+
direction TB
38+
BR["Bronze<br/>raw, append-only"]
39+
SV["Silver — 10 typed Delta tables · CDC · validated<br/>LocalLite: Polars + delta-rs · Fabric: Spark from_json"]
40+
GD["Gold<br/>gold.encounter_summary · 143,946 rows · 1 / encounter"]
41+
BR --> SV --> GD
42+
end
43+
S3 --> BR
44+
GD ==>|contract v1.1.0 · versioned · test-gated| C
45+
subgraph C["Downstream AI consumers"]
46+
direction TB
47+
BERT["clinical-bert-pipeline · NLP"]
48+
OLL["Ollama pipeline (roadmap)<br/>→ scribe-iq RAG corpus"]
49+
end
50+
classDef plat fill:#eef2ff,stroke:#6366f1;
51+
classDef cons fill:#f0fdf4,stroke:#22c55e;
52+
classDef road fill:#fff7ed,stroke:#f59e0b,stroke-dasharray:4 3;
53+
class LH plat
54+
class C cons
55+
class OLL road
4056
```
4157

4258
---
@@ -63,7 +79,7 @@ records = FHIRBundleParser().parse_bundle(bundle)
6379
# -> {"patient": [...], "encounter": [...], "soap_note": [...], "condition": [...], ...}
6480
```
6581

66-
Run the whole lakehouse locally (downloads ~4.6 GB FHIR, then builds Silver + Gold):
82+
Run the whole lakehouse locally (downloads ~4.6 GiB FHIR, then builds Silver + Gold):
6783

6884
```bash
6985
python -m core.ingest.download --bronze-root data/bronze # FHIR → Bronze (~18 min, network-bound)
@@ -135,6 +151,32 @@ the local storage root is `data/` (override with `LAKEHOUSE_LOCAL_ROOT`). Nothin
135151
See **[ARCHITECTURE.md](docs/ARCHITECTURE.md)** for the as-built diagram and module map, and the
136152
[ADRs](docs/adr/README.md) for *why*.
137153

154+
The headline decision — two independent, engine-native tiers converging on one governed contract
155+
([ADR-022](docs/adr/022-platform-independent-implementations.md)):
156+
157+
```mermaid
158+
flowchart TB
159+
subgraph CORE["core/ — LocalLite tier (laptop, $0)"]
160+
direction TB
161+
C1["Polars + delta-rs + DuckDB"]
162+
C2["own transforms<br/>core/transforms/silver_*.py → pa.Table"]
163+
C1 --> C2
164+
end
165+
subgraph FAB["fabric/ — Fabric tier (Spark / OneLake)"]
166+
direction TB
167+
F1["Spark + Delta + OneLake"]
168+
F2["own transforms<br/>fabric/transforms/silver_*.py → Spark DataFrame"]
169+
F1 --> F2
170+
end
171+
CONTRACT{{"Gold contract — gold.encounter_summary v1.1.0<br/>schema parity + lockstep CONTRACT_VERSION<br/>(compatibility, NOT shared code)"}}
172+
C2 --> CONTRACT
173+
F2 --> CONTRACT
174+
NOTE["Rejected: one shared transform layer<br/>(lowest-common-denominator + applyInPandas bridge tax) → ADR-022"]
175+
NOTE -.-> CONTRACT
176+
classDef contract fill:#eef2ff,stroke:#6366f1,font-weight:bold;
177+
class CONTRACT contract
178+
```
179+
138180
- **Independent per-platform implementations** ([ADR-022](docs/adr/022-platform-independent-implementations.md)) —
139181
each tier owns its complete Silver + Gold + validation stack written engine-native: `core/`
140182
(LocalLite) transforms return `pyarrow.Table` (Polars + delta-rs); `fabric/` transforms return
@@ -198,7 +240,7 @@ core/ ← platform-agnostic kernel; built as a whe
198240
tests/ scripts/ docs/
199241
200242
fabric/ ← Fabric tier; consumes `core` wheel via Environment
201-
platform.py ← FabricPlatform(LakehousePlatform)Session 5
243+
platform.py ← FabricPlatform — Spark-native, independent (ADR-022)
202244
notebooks/ ← Git-Integration-synced to the workspace
203245
environments/ ← Fabric Environment spec (wheel + Spark config)
204246
deploy/ ← fabric-cicd config + REST upload helper
@@ -207,7 +249,7 @@ fabric/ ← Fabric tier; consumes `core` wheel via E
207249
208250
databricks/ aws/ ← future siblings (same shape as fabric/)
209251
210-
.github/workflows/ ← core-build · core-pr-tests · fabric-deploy
252+
.github/workflows/ ← core-build · core-pr-tests · docs (fabric/databricks/aws-deploy disabled)
211253
docs/adr/ docs/roadmap/ ← ADRs + planning docs
212254
schemas/ ← machine-readable corpus JSON Schema
213255
```

docs/RUNBOOK.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,8 @@ Operational procedures for ingesting, building, verifying, and troubleshooting t
44
on the `local_lite` platform (Polars + delta-rs, zero cloud). The local stack ships with two
55
execution surfaces — the `core.surfaces.cli.pipeline` CLI (default, dependency-light, the CI path) and
66
a **Dagster** asset graph (`core/orchestration/dagster/`, ADR-015/016, optional `[orchestration]` extra);
7-
both reuse the same pure transforms. Fabric procedures land with the notebooks in Session 5.
7+
both reuse the same pure transforms. Fabric deployment is covered separately in the
8+
[Fabric Deployment](platforms/fabric-deployment.md) guide.
89
For *why* the system is shaped this way, see [ARCHITECTURE.md](ARCHITECTURE.md) and the
910
[ADRs](adr/README.md); for reference numbers, see [BENCHMARKS.md](BENCHMARKS.md).
1011

0 commit comments

Comments
 (0)