|
17 | 17 | under the License.
|
18 | 18 | -->
|
19 | 19 |
|
20 |
| -# RaySQL Design Documentation |
| 20 | +# DataFusion Ray Design Documentation |
21 | 21 |
|
22 |
| -RaySQL is a distributed SQL query engine that is powered by DataFusion. |
| 22 | +DataFusion Ray is a distributed SQL query engine that is powered by DataFusion and Ray. |
23 | 23 |
|
24 | 24 | DataFusion provides a high-performance query engine that is already partition-aware, with partitions being executed
|
25 |
| -in parallel in separate threads. RaySQL provides a distributed query planner that translates a DataFusion physical |
| 25 | +in parallel in separate threads. DataFusion Ray provides a distributed query planner that translates a DataFusion physical |
26 | 26 | plan into a distributed plan.
|
27 | 27 |
|
28 | 28 | Let's walk through an example to see how that works. We'll use [SQLBench-H](https://github.com/sql-benchmarks/sqlbench-h)
|
@@ -83,9 +83,6 @@ DataFusion's physical plan lists all the files to be queried, and they are organ
|
83 | 83 | parallel execution within a single process. In this example, the level of concurrency was configured to be four, so
|
84 | 84 | we see `partitions={4 groups: [[ ... ]]` in the leaf `ParquetExec` nodes, with the filenames listed in four groups.
|
85 | 85 |
|
86 |
| -_DataFusion will soon support parallel execution for single Parquet files but for now the parallelism is based on |
87 |
| -splitting the available files into separate groups, so RaySQL will not yet scale well for single-file inputs._ |
88 |
| - |
89 | 86 | Here is the full physical plan for query 3.
|
90 | 87 |
|
91 | 88 | ```text
|
@@ -123,7 +120,7 @@ GlobalLimitExec: skip=0, fetch=10
|
123 | 120 | ## Partitioning & Distribution
|
124 | 121 |
|
125 | 122 | The partitioning scheme changes throughout the plan and this is the most important concept to
|
126 |
| -understand in order to understand RaySQL's design. Changes in partitioning are implemented by the `RepartitionExec` |
| 123 | +understand in order to understand DataFusion Ray's design. Changes in partitioning are implemented by the `RepartitionExec` |
127 | 124 | operator in DataFusion and are happen in the following scenarios.
|
128 | 125 |
|
129 | 126 | ### Joins
|
@@ -155,7 +152,7 @@ Sort also has multiple approaches.
|
155 | 152 | - The input partitions can be collapsed down to a single partition and then sorted
|
156 | 153 | - Partitions can be sorted in parallel and then merged using a sort-preserving merge
|
157 | 154 |
|
158 |
| -DataFusion and RaySQL currently the first approach, but there is a DataFusion PR open for implementing the second. |
| 155 | +DataFusion and DataFusion Ray currently choose the first approach, but there is a DataFusion PR open for implementing the second. |
159 | 156 |
|
160 | 157 | ### Limit
|
161 | 158 |
|
@@ -260,13 +257,12 @@ child plans, building up a DAG of futures.
|
260 | 257 |
|
261 | 258 | ## Distributed Shuffle
|
262 | 259 |
|
263 |
| -The output of each query stage needs to be persisted somewhere so that the next query stage can read it. Currently, |
264 |
| -RaySQL is just writing the output to disk in Arrow IPC format, and this means that RaySQL is not truly distributed |
265 |
| -yet because it requires a shared file system. It would be better to use the Ray object store instead, as |
266 |
| -proposed [here](https://github.com/datafusion-contrib/ray-sql/issues/22). |
| 260 | +The output of each query stage needs to be persisted somewhere so that the next query stage can read it. |
| 261 | + |
| 262 | +DataFusion Ray uses the Ray object store as a shared file system, which was proposed [here](https://github.com/datafusion-contrib/ray-sql/issues/22) and implemented [here](https://github.com/datafusion-contrib/ray-sql/pull/33). |
267 | 263 |
|
268 | 264 | DataFusion's `RepartitionExec` uses threads and channels within a single process and is not suitable for a
|
269 |
| -distributed query engine, so RaySQL rewrites the physical plan and replaces the `RepartionExec` with a pair of |
| 265 | +distributed query engine, so DataFusion Ray rewrites the physical plan and replaces the `RepartionExec` with a pair of |
270 | 266 | operators to perform a "shuffle". These are the `ShuffleWriterExec` and `ShuffleReaderExec`.
|
271 | 267 |
|
272 | 268 | ### Shuffle Writes
|
|
0 commit comments