Skip to content

Commit 2a2923f

Browse files
sandeep-jaysandeep-jay
authored
Merge pull request #1 from sandeep-jay/feat/fabric-spark-native
feat(fabric): Spark-native Fabric tier — independent medallion (ADR-022)
2 parents 0475911 + de4995f commit 2a2923f

60 files changed

Lines changed: 3575 additions & 1514 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.claude/rules/fabric-transforms.md‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
# Rules: fabric/transforms/ (Fabric tier — Spark-native, ADR-022)
2+
3+
These rules apply when editing any file in `fabric/transforms/`. For the
4+
LocalLite tier rules see [transforms.md](transforms.md).
5+
6+
## Platform isolation (ADR-022)
7+
- NO imports from `core.transforms.*`, `core.gold.*`, `core.validation.*` —
8+
Fabric is an independent end-to-end implementation, not a thin wrapper.
9+
Narrow utilities (`core.redaction`) are allowed if needed.
10+
- NO imports from `databricks.`, `aws.`, or any other platform tier.
11+
- NO file paths in transforms — paths come from `FabricPlatform.storage_path()`.
12+
- NO `spark.read.format("delta")` inside a transform — transforms receive a
13+
DataFrame and return one; the platform owns Delta I/O.
14+
15+
## Spark-native, no Python bridge
16+
- ALL parsing is `from_json(value, BUNDLE_SCHEMA)` against the shared union
17+
schema in `fabric/transforms/bundle_schema.py`. No `applyInPandas`, no
18+
`udf(...)`, no driver-side Python loops.
19+
- ALL transform functions take a Spark DataFrame (`bundles_df`) + `ingest_ts`
20+
and return a Spark DataFrame.
21+
- Imports use the `from pyspark.sql import functions as F` alias and the
22+
explicit `from pyspark.sql.types import StructType, StructField, ...`
23+
pattern — Spark schemas are declared, never inferred.
24+
25+
## Schema parity with core/ (ADR-022)
26+
- Silver column names + types must match `core/transforms/silver_<table>.py`.
27+
- Gold field names + struct shapes must match `core/gold/encounter_summary.py`.
28+
- Any change here requires the symmetric change in core/ in the same PR.
29+
30+
## FHIR safety (healthcare-data skill)
31+
- Reference fields (`subject.reference`, `encounter.reference`) are stripped
32+
to bare ids via `_common.strip_reference()` — never written raw with the
33+
`urn:uuid:` / `Patient/` prefix.
34+
- Clinical codes (SNOMED, LOINC, ICD) stay as `StringType` — never cast to
35+
numeric.
36+
- patient_id / encounter_id never appear in logging statements; use
37+
`core.redaction.redact()` if a reference is needed.
38+
39+
## Data limitation (ADR-007)
40+
- `fabric/transforms/silver_genomics.py` SCHEMA marks `data_limitation` as
41+
`nullable=False` and the builder writes the canonical string literal.
42+
43+
## Test requirement
44+
- Every builder gets at least one test in `fabric/tests/` (a small
45+
hand-rolled Spark fixture is fine; we don't need full Coherent for unit
46+
tests).
47+
- The contract test for `FabricPlatform.storage_path` lives in
48+
`fabric/tests/test_fabric_platform.py`.

‎.claude/rules/notebooks.md‎

Lines changed: 42 additions & 46 deletions
Original file line numberDiff line numberDiff line change
@@ -23,70 +23,66 @@ by `# CELL ********************` (code) or `# MARKDOWN ********************`
2323
declaring its language. For interactive editing, use the Fabric notebook UI
2424
(round-trips via Git Integration).
2525

26-
## Distributed Spark pattern (ADR-020)
26+
## Spark-native pattern (ADR-022)
2727

28-
Every Silver notebook (02–07) is **Spark-distributed**:
28+
Every Silver notebook (02–07) is pure-Spark — no `applyInPandas`, no
29+
`core.transforms.*` import, no `pa.Table` round-trip:
2930

3031
```python
31-
import os; os.environ["LAKEHOUSE_PLATFORM"] = "fabric"
32-
from core.platform.factory import get_platform
33-
from core.transforms.registry import SILVER_TABLES
34-
from fabric.spark_helpers import (
35-
make_partition_parser, pa_to_spark_schema, read_fhir_bundles_distributed,
36-
)
37-
from pyspark.sql import functions as F
38-
39-
platform = get_platform()
40-
spark = platform.get_spark_session()
41-
42-
# 1. Read FHIR bundles as a partitioned Spark DataFrame
43-
bundles_df = read_fhir_bundles_distributed(spark, platform.storage_path("bronze", "fhir"))
32+
from datetime import UTC, datetime
33+
from fabric.platform import FabricPlatform
34+
from fabric.transforms.registry import REGISTRY
4435

45-
# 2. Distributed parse + build via applyInPandas — each executor runs the
46-
# pure-Python FHIRBundleParser + build_silver_<table> on its partition
47-
spec = SILVER_TABLES[TABLE]
48-
parse_udf = make_partition_parser(TABLE, spec.build, ingest_ts)
49-
silver_df = bundles_df.groupBy(F.spark_partition_id()).applyInPandas(
50-
parse_udf, schema=pa_to_spark_schema(spec.schema)
51-
)
36+
TABLE = "patient"
37+
platform = FabricPlatform() # ADR-022: no factory, no env var
38+
spark = platform.get_spark_session()
39+
ingest_ts = datetime.now(UTC)
40+
spec = REGISTRY[TABLE]
5241

53-
# 3. Spark-native Delta MERGE (CDC + ADR-019 dedup guard)
54-
platform.write_silver_spark(TABLE, silver_df, mode="merge")
42+
bundles_df = platform.read_bronze_bundles_spark() # (path, value) text DataFrame
43+
silver_df = spec.build(bundles_df, ingest_ts) # Spark-native from_json + project
44+
platform.write_silver_spark(TABLE, silver_df, mode="merge") # Delta MERGE + CDC + ADR-019 guard
5545
```
5646

57-
Multi-table silver notebooks (04 clinical, 07 ecg+genomics) cache
58-
`bundles_df` and run one `applyInPandas` pipeline per output table.
47+
Multi-table notebooks (04 clinical, 07 ecg+genomics) cache `bundles_df`
48+
and loop over their table list — one `from_json` parse fans out to every
49+
builder.
5950

60-
Gold notebook (09) uses Spark for reads + writes but does the global
61-
denormalization on the driver via `build_encounter_summary` (Polars). See
62-
ADR-020 §"When applyInPandas vs driver-side compute" for the trade.
51+
Gold (09) reads each Silver as a Spark DataFrame and calls
52+
`fabric.gold.encounter_summary.build_encounter_summary(silver_dict, ...)`
53+
which returns a Spark DataFrame; no driver-side Polars step.
54+
55+
Validation (08) uses `fabric.validation.validate.validate_table(name, df)`
56+
— a single `.agg()` per table.
6357

6458
## Cell template
6559

66-
Each notebook uses cells in this order (count varies by notebook — clarity
67-
over rigid count):
60+
Each notebook uses cells in this rough order (count varies — clarity over
61+
rigid count). All Silver notebooks (02–07) follow the same shape:
6862

6963
1. **Markdown** — title, purpose, I/O, scale, screenshot filename
70-
2. **Markdown** — architecture context (ADR refs, distributed strategy)
71-
3. **Code** — imports + platform + spark + table constants
64+
2. **Markdown** — architecture (ADR refs, native engine strategy)
65+
3. **Code** — imports + `FabricPlatform()` + spark + table constants
7266
4. **Markdown** — step 1 description (read bundles distributed)
73-
5. **Code** — `read_fhir_bundles_distributed(...)` → `bundles_df`
74-
6. **Markdown** — step 2 description (parse + build via applyInPandas)
75-
7. **Code** — `make_partition_parser` + `applyInPandas` → `silver_df`
76-
8. **Markdown** — step 3 description (Spark-native MERGE)
77-
9. **Code** — `platform.write_silver_spark(...)`
78-
10. **Markdown** — validation
79-
11. **Code** — `platform.read_silver_spark(...)` + count + `display()`
80-
12. **Code** — `platform.log_metric(...)` + "next:" pointer
67+
5. **Code** — `platform.read_bronze_bundles_spark()` → `bundles_df`
68+
6. **Markdown** — step 2 description (build + MERGE)
69+
7. **Code** — `spec.build(...)` + `platform.write_silver_spark(...)`
70+
8. **Markdown** — validation
71+
9. **Code** — `platform.read_silver_spark(...)` + count + `display()` + `log_metric`
8172

8273
## Paths
83-
Never hardcode abfss:// paths — always `platform.storage_path()`.
74+
Never hardcode `abfss://` paths — always `platform.storage_path()`.
75+
76+
## Imports
77+
Never import from `core.transforms.*` / `core.gold.*` / `core.validation.*`
78+
in a Fabric notebook — Fabric is an independent end-to-end implementation
79+
(ADR-022). Use `fabric.transforms.*` / `fabric.gold.*` / `fabric.validation.*`.
8480

8581
## Notebook 05 (SOAP notes) — special rule
86-
**Demo centerpiece.** Cell 11 (or equivalent code cell after validation
87-
prints) MUST render a *decoded* SOAP note via `displayHTML(...)`. A reviewer
88-
must see readable clinical text — not a Base64 blob — in the notebook
89-
output. This is the highest-priority screenshot of the project.
82+
**Demo centerpiece.** The validation cell MUST render a *decoded* SOAP note
83+
in `display()` / `print()` output. A reviewer must see readable clinical
84+
text — not a Base64 blob — in the notebook output. This is the
85+
highest-priority screenshot of the project.
9086

9187
## Screenshot rule
9288
Capture screenshots as you run — don't batch at the end. Evidence of a

‎.claude/rules/transforms.md‎

Lines changed: 26 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -1,35 +1,43 @@
1-
# Rules: core/transforms/
1+
# Rules: core/transforms/ (LocalLite tier — Polars + delta-rs)
22

3-
These rules apply when editing any file in core/transforms/.
3+
These rules apply when editing any file in `core/transforms/`. For
4+
`fabric/transforms/` rules, see [fabric-transforms.md](fabric-transforms.md).
45

5-
## Platform isolation (ADR-002, ADR-015, ADR-017)
6+
## Platform isolation (ADR-015, ADR-017, ADR-022)
67
- NO imports from core.platform, pyspark, notebookutils, mssparkutils, or delta
7-
- NO imports from dagster or orchestration — the orchestration tier imports
8-
transforms, never the reverse (same rule as Spark/notebooks)
8+
- NO imports from dagster or orchestration — orchestration imports transforms,
9+
never the reverse
910
- NO imports from `fabric.`, `databricks.`, `aws.`, or any other platform-specific
10-
package — core never depends on platform tiers (ADR-017 one-way dependency rule)
11+
package — `core/` never depends on platform tiers
1112
- NO file paths — all paths come from the platform parameter
12-
- NO spark.read or spark.write — transforms receive data, they don't fetch it
13+
- NO `spark.read` or `spark.write` — transforms receive data, they don't fetch it
1314

14-
## Arrow return type (ADR-004)
15-
- ALL transform functions return pa.Table
16-
- Import: import pyarrow as pa
15+
## Arrow return type
16+
- ALL transform functions return `pa.Table`
17+
- Import: `import pyarrow as pa`
1718
- Schema must be explicitly defined, not inferred
19+
- (ADR-004 archived; pa.Table is the LocalLite tier's interchange type — not
20+
cross-platform; the Fabric tier returns Spark DataFrames per ADR-022)
1821

1922
## FHIR safety (healthcare-data skill)
20-
- ALL FHIR field access uses .get() with a default — never direct key access
21-
- Clinical codes (SNOMED, LOINC, ICD) always stored as str, never cast to int
23+
- ALL FHIR field access uses `.get()` with a default — never direct key access
24+
- Clinical codes (SNOMED, LOINC, ICD) always stored as `str`, never cast to int
2225
- patient_id and encounter_id never appear in logging statements
2326

2427
## Data limitation (ADR-007)
25-
- extract_genomic_report() must always set data_limitation field
26-
- data_limitation = "Synthea simulated inheritance — not clinical variants"
27-
- This field is non-nullable — raise ValueError if somehow None
28+
- `extract_genomic_report()` must always set `data_limitation`
29+
- `data_limitation = "Synthea simulated inheritance — not clinical variants"`
30+
- Non-nullable — raise `ValueError` if somehow None
2831

2932
## DICOM (ADR-006)
30-
- pydicom.dcmread() always called with stop_before_pixels=True
33+
- `pydicom.dcmread()` always called with `stop_before_pixels=True`
3134
- Never load pixel data in any transform in this directory
3235

36+
## Schema parity with fabric/ (ADR-022)
37+
- Silver column names + types must match `fabric/transforms/silver_<table>.py`
38+
- Gold field names + struct shapes must match `fabric/gold/encounter_summary.py`
39+
- Any change here requires the symmetric change in fabric/ in the same PR
40+
3341
## Test requirement
34-
- Every function in this directory has a corresponding test in core/tests/
35-
- Tests use core/tests/fixtures/sample_bundle.json — never real patient data
42+
- Every function in this directory has a corresponding test in `core/tests/`
43+
- Tests use `core/tests/fixtures/sample_bundle.json` — never real patient data

.github/workflows/fabric-deploy.yml renamed to .github/workflows/fabric-deploy.yml.disabled

File renamed without changes.

‎CHANGELOG.md‎

Lines changed: 113 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,8 +5,119 @@ Format: [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
55

66
## [Unreleased]
77

8-
### Session 5 (in progress) — Fabric end-to-end + dedup fix + Power BI
9-
Plan: [docs/roadmap/fabric-execution-plan.md](docs/roadmap/fabric-execution-plan.md). 7 phases; 1–3 complete, 4 in progress.
8+
### Session 5 (in progress) — Fabric Spark-native rewrite (ADR-022) + dedup fix + Power BI
9+
Plan: [docs/roadmap/fabric-execution-plan.md](docs/roadmap/fabric-execution-plan.md).
10+
11+
#### Milestone (2026-05-29 — first green cloud run)
12+
- **Notebooks 00–10 ran successfully end-to-end on Fabric F4 capacity**
13+
against `SAMPLE_SIZE=100` Coherent bundles. All 10 Silver tables +
14+
`gold.encounter_summary` + Bronze/Gold manifests materialized in the
15+
`scribe_iq_synthea_coherent` lakehouse.
16+
- Branch `feat/fabric-spark-native` pushed to **both** GitHub (canonical
17+
mirror) and Azure DevOps (Fabric Git Integration source) via
18+
multi-push origin. Single `git push` fans out to both.
19+
20+
#### Added (2026-05-29)
21+
- `fabric/notebooks/01_bronze_ingest.Notebook/` — self-contained Bronze
22+
ingest. Pulls Synthea Coherent from `s3://synthea-open-data/coherent/`
23+
via anonymous boto3, round-robin partitions into `cohort=A,B,C` under
24+
`Files/bronze/fhir/`, writes an `IngestManifest`-shaped JSON under
25+
`Files/bronze/_metadata/`. `SAMPLE_SIZE` knob for fast demo (`100`) vs
26+
full corpus (`None`).
27+
- `fabric/environments/public_libraries.yml` — pip-block file Fabric's
28+
Environment "Import .yml" UI accepts; pins `boto3==1.35.36` +
29+
`botocore==1.35.36` for reproducibility.
30+
- `FabricPlatform.files_path(subpath)` — Files/-rooted URI helper for
31+
non-table artifacts (Bronze JSON, Gold manifest). One place owns the
32+
GUID-vs-name path detail.
33+
34+
#### Changed (2026-05-29 — operational fixes from cloud run)
35+
- `FabricPlatform.ensure_env` now reads from Spark conf
36+
(`trident.workspace.id`, `trident.lakehouse.id`) instead of
37+
`mssparkutils.env.getWorkspaceId()` — the latter is a Synapse API
38+
not present on Fabric. Returns workspace + lakehouse GUIDs (not name);
39+
display name is best-effort, informational only.
40+
- OneLake paths now use lakehouse GUID throughout (drop `.Lakehouse`
41+
suffix). Required for tenants with `FriendlyNameSupportDisabled`
42+
(the trial tenant has this) — `<name>.Lakehouse` paths get HTTP 400.
43+
Notebooks 00, 01, 10 updated to use `platform.files_path()` instead
44+
of inline path construction.
45+
- `00_setup` Gate 1 reads `spark.conf.get("trident.workspace.id")`
46+
(drops the broken `mssparkutils.env.getWorkspaceId` call).
47+
- `01_bronze_ingest` validation cell uses `spark.read.text(wholetext=True)`
48+
to read the sample bundle — `mssparkutils.fs.head` silently truncates
49+
at ~100 KB even when a larger maxBytes is passed, breaking
50+
`json.loads`. Sample-histogram wrapped in try/except so a parse
51+
failure prints a one-liner instead of halting the cell (manifest
52+
write below it now always runs).
53+
- `fabric/environments/lakehouse_env.yml` — documentation-style spec
54+
updated to match ADR-022; drops `pyarrow`/`pydicom`/`python-dateutil`
55+
(not used by the pure-Spark Fabric tier — Fabric runtime supplies
56+
pyarrow; pydicom is local-only; date parsing is Spark-native).
57+
- `.github/workflows/fabric-deploy.yml` renamed
58+
`fabric-deploy.yml.disabled`. User removed the `fabric-prod` GitHub
59+
Environment; the workflow's `environment: fabric-prod` would fail on
60+
trigger. Matches the existing `aws-deploy.yml.disabled` /
61+
`databricks-deploy.yml.disabled` convention. Active deploy path is
62+
Azure DevOps Git Integration + manual UI wheel upload.
63+
64+
#### Tests (2026-05-29)
65+
- `test_fabric_platform.py` updated for GUID-based API:
66+
`test_storage_path_builds_onelake_uri` rewritten for the GUID shape
67+
(no `.Lakehouse` suffix). New `test_files_path_builds_onelake_uri`
68+
covers the helper. `FabricPlatform(lakehouse_id=...)` constructor
69+
arg replaces `lakehouse_name=...` for path-shape tests.
70+
- Full suite: 128 passed + 1 skipped (workspace-only).
71+
72+
#### Added (2026-05-29 — ADR-022 architecture pivot)
73+
- **ADR-022** (Independent per-platform implementations) — supersedes ADR-002
74+
(LakehousePlatform ABC as universal contract), ADR-004 (pa.Table as
75+
cross-platform interchange), and ADR-020 (applyInPandas bridge — same-day
76+
supersession). Each platform tier now owns its complete Silver + Gold +
77+
validation stack written engine-native; cross-platform compat is by
78+
schema parity + lockstep CONTRACT_VERSION bumps, not code sharing.
79+
- `fabric/transforms/` — Spark-native Silver layer (10 builders + union
80+
BUNDLE_SCHEMA + registry). Parses bundles via `from_json` and projects
81+
to Silver via Spark DataFrame ops; no Python bridge.
82+
- `fabric/gold/` — Spark-native `build_encounter_summary` + `corpus_manifest`.
83+
Output schema matches `core.gold.encounter_summary` field-for-field.
84+
Includes a UUIDv5 expression synthesized in Spark (SHA1 + RFC 4122 bit
85+
twiddling) so `summary_id` stays deterministic across rebuilds.
86+
- `fabric/validation/` — single `.agg()` per Silver table computes every
87+
metric in one pass; ingest_log schema matches core's.
88+
- `.claude/rules/fabric-transforms.md` — Fabric-tier transform rules.
89+
90+
#### Changed (2026-05-29)
91+
- `fabric/platform.py` slimmed: dropped `write_silver(pa.Table)` /
92+
`read_silver() → pa.Table` / `write_gold(pa.Table)` convenience wrappers,
93+
dropped legacy `_write_delta(pa.Table)`, dropped `LakehousePlatform`
94+
inheritance. Spark DataFrames are the only interchange type. Added
95+
`read_bronze_bundles_spark()` as the canonical Bronze entry point.
96+
- `core/platform/factory.py` PLATFORMS dict drops `fabric/databricks/aws/gcp`
97+
— independent tiers don't dispatch through the local factory.
98+
- All Fabric notebooks (00 + 02–10) rewritten: instantiate `FabricPlatform()`
99+
directly (no factory, no env var), import from `fabric.transforms` /
100+
`fabric.gold` / `fabric.validation`, no `applyInPandas`. Notebook 10
101+
rewritten against the actual manifest keys (`gold_table`,
102+
`silver_sources`, `row_count`) and Gold schema names (`soap_note_text`).
103+
- `CLAUDE.md` + `.claude/rules/transforms.md` + `.claude/rules/notebooks.md`
104+
updated for the independence model. ADR index README.md flags 002/004/020
105+
as Superseded with links into `docs/_archive/adr/`. ADR-017 amended in place.
106+
107+
#### Removed (2026-05-29)
108+
- `fabric/spark_helpers.py` (housed the `applyInPandas` bridge factory +
109+
pa→Spark schema converter; both dead under pure-Spark).
110+
111+
#### Tests (2026-05-29)
112+
- `fabric/tests/test_fabric_platform.py` — dropped subclass + abstract-method
113+
contract tests; rewrote the workspace round-trip to use Spark DataFrames
114+
against `fabric.transforms.registry`. Added `test_name_attribute`.
115+
- `core/tests/test_platform_factory.py` — added `test_fabric_not_in_factory`;
116+
updated unbuilt-platform test to use `local_spark` placeholder.
117+
- Full suite: 128 passed + 1 skipped (workspace-only).
118+
119+
### Session 5 — earlier phases (Fabric end-to-end + dedup fix + Power BI)
120+
Plan: [docs/roadmap/fabric-execution-plan.md](docs/roadmap/fabric-execution-plan.md). Phases 1–3 complete (pre-pivot).
10121

11122
#### Added
12123
- **ADR-019** (Silver MERGE idempotency) — pre-merge target-side dedup guard

0 commit comments

Comments
 (0)