RayDP-0.4.0

Highlights

• Support conversion between Spark Dataframe and Ray Dataset
• Support Ray 1.7.0
• Support Spark 3.2.0
• Various bug fixes and improvements

Thanks @ConeyLiu @zuston @kira-lin @mjschock @edoakes @carsonwang for their contributions to the release!

RayDP-0.3.0

RayDP 0.3.0 includes the following key changes:

• Spark dynamic resource allocation support. This allows you to launch Spark external shuffle service on Ray and enable Spark dynamic resource allocation to maximize your resource utilization.
• Spark-submit support. A command line utility bin/raydp-submit is provided for you to submit a scala/java/python Spark application to a Ray cluster.
• MPI on Ray. This allows you to run MPI jobs on Ray. You can use this feature to construct pipelines like Spark + MPI on Ray.
• Ray 1.3.0 support.

RayDP-0.2.0

Several bug fixes. And also we have added some examples to show how RayDP works together with other libraries, such as PyTorch, Tensorflow, XGBoost, and Horovod.

RayDP-0.1.1

RayDP 0.1.1

RayDP-0.1.0

RayDP 0.1.0