You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/data-engineering/fabric-notebook-selection-guide.md
+7-9Lines changed: 7 additions & 9 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -35,7 +35,7 @@ Use this high-level reference to quickly determine the most suitable notebook ty
35
35
36
36
Use this structured comparison table to understand the architectural and operational trade-offs between notebook types. Best suited for engineering evaluations or implementation planning.
@@ -46,27 +46,27 @@ Use this structured comparison table to understand the architectural and operati
46
46
| High-Concurrency Execution | Manual FIFO-style parallelism per notebook | System-managed concurrency with support for parallel execution. |
47
47
| Resource Customization & Scaling | Fixed compute (2-core VM); does not auto scale. Users can manually scale out using %%config within the notebook. | Flexible resource allocation; supports autoscaling and custom Spark configurations. |
| API Orchestration | Effective for lightweight orchestration and control flows, especially REST/gRPC-based integrations | Less optimal for basic orchestration tasks due to longer start-up and distributed overhead. |
54
54
| Complex ETL DAGs | Supports FIFO scheduling only | Supports both FAIR and FIFO scheduling with DAG customization. |
| Library Access | Strong Python library support across multiple runtimes; however, limited access to Spark-native libraries may require manual integration. | Fully supports MLlib, Spark SQL, PySpark, and Spark Streaming. |
61
61
| Delta Lake Handling | The Python Notebook runtime comes with pre-installed [deltas](https://delta-io.github.io/delta-rs/) and [duckdb](https://duckdb.org/) libraries, enabling both reading and writing of Delta Lake data. However, some Delta Lake features may still be unsupported. | Fully supported with native compatibility. |
| Production Management | Limited production features; Does not support environment vars. | Support for production workflows with environment variables, library management through environment items, and item-based deployment. |
@@ -80,15 +80,15 @@ Use **Python Notebooks** for fast iteration, cost-effective analysis, and intera
80
80
81
81
Use **PySpark Notebooks** for distributed computing, production-grade ETL workflows, or scenarios where high concurrency and Spark-native APIs are essential.
82
82
83
-
**Choose Python Notebooks When:**
83
+
### **Choose Python Notebooks When:**
84
84
85
85
- You need fast start-up (typically within seconds) on a lightweight 2-core container.
86
86
- If minimizing compute cost is a priority - for interactive analysis or scheduled micro-jobs.
87
87
- You want immediate access to pip-installable libraries and pre-installed DuckDB and Polars.
88
88
- You need to test across different Python runtime versions.
89
89
- Your data comfortably fits in the memory of a single node.
90
90
91
-
**Choose PySpark Notebooks When:**
91
+
### **Choose PySpark Notebooks When:**
92
92
93
93
- Your workloads exceed the memory or compute limits of a single node.
94
94
- You require high-concurrency pools to run parallel jobs across Notebooks.
@@ -109,8 +109,6 @@ This section provides a quick reference for the fundamental technical and archit
109
109
| Cost Profile | Lower cost; ideal for short tasks and prototyping | Higher cost; suited for scalable, long-running workloads |
110
110
| Python/Spark Versioning | Multiple Python versions available | Tied to specific Spark runtime version |
0 commit comments