Skip to content

Commit 10ea1df

Browse files
sdebruynclaude
andcommitted
Document remote Spark test execution in CONTRIBUTING.md
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1 parent 8dc317e commit 10ea1df

1 file changed

Lines changed: 27 additions & 0 deletions

File tree

CONTRIBUTING.md

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -87,6 +87,33 @@ uv run pytest --with-grants # Include GRANT/authorization tests
8787
uv run pytest -k "TestClassName" # A specific test class
8888
```
8989

90+
### Remote Spark test execution
91+
92+
FabricSpark tests normally run via a local Livy session. The `--remote` flag delegates test execution to a Spark Job Definition on Fabric infrastructure instead. Locally, pytest collects tests as usual, but instead of executing them it syncs the project to a lakehouse via azcopy and submits a Spark job that runs the tests remotely. Results are downloaded as junitxml and reported back in the local pytest session.
93+
94+
```shell
95+
uv run pytest --de --remote -k "TestClassName" # Run a specific test remotely
96+
uv run pytest --de --remote # Run all DE tests remotely
97+
```
98+
99+
`--remote` requires `--de` — it only applies to FabricSpark tests.
100+
101+
#### Prerequisites
102+
103+
1. [azcopy](https://learn.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-v10?WT.mc_id=MVP_310840) installed and on your `PATH`
104+
2. Azure credentials (`az login` or service principal configured in `test.env`)
105+
3. A Spark Job Definition in the Fabric workspace (the name is configurable via `FABRIC_TEST_SPARK_JOB_NAME`, default: `dbt-fabric-tests`)
106+
107+
#### Optional environment variables
108+
109+
| Variable | Description |
110+
|---|---|
111+
| `FABRIC_TEST_SPARK_EXEC_MODE` | `remote` or `mounted` (default: local execution when omitted) |
112+
| `FABRIC_TEST_ONELAKE_PATH` | Local mount path for OneLake (required for `mounted` mode) |
113+
| `FABRIC_TEST_SPARK_JOB_NAME` | Spark Job Definition display name (default: `dbt-fabric-tests`) |
114+
115+
The remote orchestrator reuses `FABRIC_TEST_WORKSPACE_NAME` and `FABRIC_TEST_LAKEHOUSE_NAME` from `test.env` and resolves their IDs via the Fabric API — no separate ID variables are needed.
116+
90117
### Test architecture
91118

92119
Tests use [dbt-tests-adapter](https://github.com/dbt-labs/dbt-adapters), dbt's official adapter test harness. It provides base test classes for standard adapter behavior. Our tests inherit from these and override fixtures where Fabric's SQL dialect differs:

0 commit comments

Comments
 (0)