Remove legacy shuffle, add docs for distributed testing #19
Conversation
| @@ -77,7 +77,7 @@ def _get_worker_inputs( | |||
| plan_bytes = datafusion_ray.serialize_execution_plan(stage.get_execution_plan()) | |||
| futures = [] | |||
| opt = {} | |||
| opt["resources"] = {"worker": 1e-3} | |||
| # opt["resources"] = {"worker": 1e-3} | |||
There was a problem hiding this comment.
I had to remove this; otherwise, the Ray cluster could not find a suitable worker node.
@franklsf95 do we still need this?
There was a problem hiding this comment.
This only works when you start a Ray cluster with custom resources, e.g. the head node with ray start --head --resources='{"head":1}' and worker nodes each with ray start --resources='{"worker":1}'. I had this resource requirement to make sure the tasks run on worker nodes exclusively (for fair benchmarking). If we don't do this, the task could also run on the driver node. Depending on the use case, this may be harmless.
| @@ -77,7 +77,7 @@ def _get_worker_inputs( | |||
| plan_bytes = datafusion_ray.serialize_execution_plan(stage.get_execution_plan()) | |||
| futures = [] | |||
| opt = {} | |||
| opt["resources"] = {"worker": 1e-3} | |||
| # opt["resources"] = {"worker": 1e-3} | |||
There was a problem hiding this comment.
Maybe a comment here is useful?
| opt["resources"] = {"worker": 1e-3} | ||
| if use_ray_shuffle: | ||
| opt["num_returns"] = output_partitions_count | ||
| #opt["resources"] = {"worker": 1e-3} |
| ## Start Ray Worker Nodes(s) | ||
|
|
||
| ```shell | ||
| ray start --address=10.0.0.23:6379 --redis-password='5241590000000000' |
There was a problem hiding this comment.
Did you need to start redis?
There was a problem hiding this comment.
no, this is built into ray somehow and this is the password that it uses
This PR removes the legacy shuffle reader/writer that only work on a single node.
I also added brief documentation on distributed testing.