Skip to content

Commit ca40269

Browse files
authored
Merge pull request #8760 from MicrosoftDocs/main
Publish to live, Tuesday 4 AM PST, 6/3
2 parents 70f3aa8 + 869877d commit ca40269

8 files changed

Lines changed: 203 additions & 58 deletions

File tree

docs/data-engineering/fabric-notebook-selection-guide.md

Lines changed: 7 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@ Use this high-level reference to quickly determine the most suitable notebook ty
3535

3636
Use this structured comparison table to understand the architectural and operational trade-offs between notebook types. Best suited for engineering evaluations or implementation planning.
3737

38-
**Execution & Performance**
38+
### **Execution & Performance**
3939

4040
| Scenario | Python Notebooks (2-core VM) | PySpark Notebooks (Spark Compute) |
4141
| --- | --- | --- |
@@ -46,27 +46,27 @@ Use this structured comparison table to understand the architectural and operati
4646
| High-Concurrency Execution | Manual FIFO-style parallelism per notebook | System-managed concurrency with support for parallel execution. |
4747
| Resource Customization & Scaling | Fixed compute (2-core VM); does not auto scale. Users can manually scale out using %%config within the notebook. | Flexible resource allocation; supports autoscaling and custom Spark configurations. |
4848

49-
**Workflow & Orchestration**
49+
### **Workflow & Orchestration**
5050

5151
| Scenario | Python Notebooks (2-core VM) | PySpark Notebooks (Spark Compute) |
5252
| --- | --- | --- |
5353
| API Orchestration | Effective for lightweight orchestration and control flows, especially REST/gRPC-based integrations | Less optimal for basic orchestration tasks due to longer start-up and distributed overhead. |
5454
| Complex ETL DAGs | Supports FIFO scheduling only | Supports both FAIR and FIFO scheduling with DAG customization. |
5555

56-
**Platform & Library Support**
56+
### **Platform & Library Support**
5757

5858
| Scenario | Python Notebooks (2-core VM) | PySpark Notebooks (Spark Compute) |
5959
| --- | --- | --- |
6060
| Library Access | Strong Python library support across multiple runtimes; however, limited access to Spark-native libraries may require manual integration. | Fully supports MLlib, Spark SQL, PySpark, and Spark Streaming. |
6161
| Delta Lake Handling | The Python Notebook runtime comes with pre-installed [deltas](https://delta-io.github.io/delta-rs/) and [duckdb](https://duckdb.org/) libraries, enabling both reading and writing of Delta Lake data. However, some Delta Lake features may still be unsupported. | Fully supported with native compatibility. |
6262

63-
**Production & Enterprise Readiness**
63+
### **Production & Enterprise Readiness**
6464

6565
| Scenario | Python Notebooks (2-core VM) | PySpark Notebooks (Spark Compute) |
6666
| --- | --- | --- |
6767
| Production Management | Limited production features; Does not support environment vars. | Support for production workflows with environment variables, library management through environment items, and item-based deployment. |
6868

69-
**Cost Considerations**
69+
### **Cost Considerations**
7070

7171
| Scenario | Python Notebooks (2-core VM) | PySpark Notebooks (Spark Compute) |
7272
| --- | --- | --- |
@@ -80,15 +80,15 @@ Use **Python Notebooks** for fast iteration, cost-effective analysis, and intera
8080

8181
Use **PySpark Notebooks** for distributed computing, production-grade ETL workflows, or scenarios where high concurrency and Spark-native APIs are essential.
8282

83-
**Choose Python Notebooks When:**
83+
### **Choose Python Notebooks When:**
8484

8585
- You need fast start-up (typically within seconds) on a lightweight 2-core container.
8686
- If minimizing compute cost is a priority - for interactive analysis or scheduled micro-jobs.
8787
- You want immediate access to pip-installable libraries and pre-installed DuckDB and Polars.
8888
- You need to test across different Python runtime versions.
8989
- Your data comfortably fits in the memory of a single node.
9090

91-
**Choose PySpark Notebooks When:**
91+
### **Choose PySpark Notebooks When:**
9292

9393
- Your workloads exceed the memory or compute limits of a single node.
9494
- You require high-concurrency pools to run parallel jobs across Notebooks.
@@ -109,8 +109,6 @@ This section provides a quick reference for the fundamental technical and archit
109109
| Cost Profile | Lower cost; ideal for short tasks and prototyping | Higher cost; suited for scalable, long-running workloads |
110110
| Python/Spark Versioning | Multiple Python versions available | Tied to specific Spark runtime version |
111111
| Custom Libraries | pip install + resource folders | pip install + resource folders + environments item |
112-
| ~~Environment Variables~~ | ~~Not supported~~ | ~~Supported~~ |
113-
| ~~Lakehouse Integration~~ | ~~Requires Lakehouse to be explicitly attached~~ | ~~Can read/write without attachment~~ |
114112
| Fabric Spark Capabilities | Limited access to Spark engine features | Full access: NEE, Autotune, VORDER, Vegas Cache |
115113
| Delta Lake Compatibility | Partially compatible; potential performance issues | Fully supported and optimized |
116114

docs/data-engineering/runtime.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,7 @@ Below, you find a comprehensive comparison of key components, including Apache S
3232
3333
| | **[Runtime 1.1](./runtime-1-1.md)** | **[Runtime 1.2](./runtime-1-2.md)** | **[Runtime 1.3](./runtime-1-3.md)** |
3434
|-----------------------|-------------------------------------|-------------------------------------|------------------------------------|
35-
| **Release Stage** | EOSA | GA | GA |
35+
| **Release Stage** | EOS | EOSA| GA |
3636
| **Apache Spark** | 3.3.1 | 3.4.1 | 3.5.0 |
3737
| **Operating System** | Ubuntu 18.04 | Mariner 2.0 | Mariner 2.0 |
3838
| **Java** | 8 | 11 | 11 |

0 commit comments

Comments
 (0)