[2.50 CHERRY-PICK][docs] Update SLURM docs with symmetric-run (#56775) (#57565)

We recently added `ray symmetric-run`. This helps greatly simplify our
slurm tutorial.

cherrypick #56775

Author: richardliaw

doc/source/cluster/doc_code/slurm-basic.sh

@@ -16,50 +16,30 @@ nodes=$(scontrol show hostnames "$SLURM_JOB_NODELIST")
 nodes_array=($nodes)
 
 head_node=${nodes_array[0]}
-head_node_ip=$(srun --nodes=1 --ntasks=1 -w "$head_node" hostname --ip-address)
 
-# if we detect a space character in the head node IP, we'll
-# convert it to an ipv4 address. This step is optional.
-if [[ "$head_node_ip" == *" "* ]]; then
-IFS=' ' read -ra ADDR <<<"$head_node_ip"
-if [[ ${#ADDR[0]} -gt 16 ]]; then
-  head_node_ip=${ADDR[1]}
-else
-  head_node_ip=${ADDR[0]}
-fi
-echo "IPV6 address detected. We split the IPV4 address as $head_node_ip"
-fi
-# __doc_head_address_end__
-
-# __doc_head_ray_start__
 port=6379
-ip_head=$head_node_ip:$port
+ip_head=$head_node:$port
 export ip_head
 echo "IP Head: $ip_head"
+# __doc_head_address_end__
 
-echo "Starting HEAD at $head_node"
-srun --nodes=1 --ntasks=1 -w "$head_node" \
-    ray start --head --node-ip-address="$head_node_ip" --port=$port \
-    --num-cpus "${SLURM_CPUS_PER_TASK}" --num-gpus "${SLURM_GPUS_PER_TASK}" --block &
-# __doc_head_ray_end__
-
-# __doc_worker_ray_start__
-# optional, though may be useful in certain versions of Ray < 1.0.
-sleep 10
-
-# number of nodes other than the head node
-worker_num=$((SLURM_JOB_NUM_NODES - 1))
-
-for ((i = 1; i <= worker_num; i++)); do
-    node_i=${nodes_array[$i]}
-    echo "Starting WORKER $i at $node_i"
-    srun --nodes=1 --ntasks=1 -w "$node_i" \
-        ray start --address "$ip_head" \
-        --num-cpus "${SLURM_CPUS_PER_TASK}" --num-gpus "${SLURM_GPUS_PER_TASK}" --block &
-    sleep 5
-done
-# __doc_worker_ray_end__
+# __doc_symmetric_run_start__
+# Start Ray cluster using symmetric_run.py on all nodes.
+# Symmetric run will automatically start Ray on all nodes and run the script ONLY the head node.
+# Use the '--' separator to separate Ray arguments and the entrypoint command.
+# The --min-nodes argument ensures all nodes join before running the script.
+
+# All nodes (including head and workers) will execute this block.
+# The entrypoint (simple-trainer.py) will only run on the head node.
+srun --nodes="$SLURM_JOB_NUM_NODES" --ntasks="$SLURM_JOB_NUM_NODES" \
+    ray symmetric-run \
+    --address "$ip_head" \
+    --min-nodes "$SLURM_JOB_NUM_NODES" \
+    --num-cpus="${SLURM_CPUS_PER_TASK}" \
+    --num-gpus="${SLURM_GPUS_PER_TASK}" \
+    -- \
+    python -u simple-trainer.py "$SLURM_CPUS_PER_TASK"
+# __doc_symmetric_run_end__
 
 # __doc_script_start__
-# ray/doc/source/cluster/doc_code/simple-trainer.py
-python -u simple-trainer.py "$SLURM_CPUS_PER_TASK"
+# The entrypoint script (simple-trainer.py) will be run on the head node by symmetric_run.

doc/source/cluster/vms/user-guides/community/slurm.rst

@@ -8,15 +8,9 @@ Slurm usage with Ray can be a little bit unintuitive.
 * SLURM requires multiple copies of the same program are submitted multiple times to the same cluster to do cluster programming. This is particularly well-suited for MPI-based workloads.
 * Ray, on the other hand, expects a head-worker architecture with a single point of entry. That is, you'll need to start a Ray head node, multiple Ray worker nodes, and run your Ray script on the head node.
 
-.. warning::
+To bridge this gap, Ray 2.49 and above introduces ``ray symmetric-run`` command, which will start a Ray cluster on all nodes with given CPU and GPU resources and run your entrypoint script ONLY the head node.
 
-    SLURM support is still a work in progress. SLURM users should be aware
-    of current limitations regarding networking.
-    See :ref:`here <slurm-network-ray>` for more explanations.
-
-    SLURM support is community-maintained. Maintainer GitHub handle: tupui.
-
-This document aims to clarify how to run Ray on SLURM.
+Below, we provide a walkthrough using ``ray symmetric-run`` to run Ray on SLURM.
 
 .. contents::
   :local:
@@ -107,46 +101,27 @@ Next, we'll want to obtain a hostname and a node IP address for the head node. T
    :start-after: __doc_head_address_start__
    :end-before: __doc_head_address_end__
 
+.. note:: In Ray 2.49 and above, you can use IPv6 addresses/hostnames.
 
 
-Starting the Ray head node
-~~~~~~~~~~~~~~~~~~~~~~~~~~
+Starting Ray and executing your script
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-After detecting the head node hostname and head node IP, we'll want to create
-a Ray head node runtime. We'll do this by using ``srun`` as a background task
-as a single task/node (recall that ``tasks-per-node=1``).
+.. note:: `ray symmetric-run` is available in Ray 2.49 and above. Check older versions of the documentation if you are using an older version of Ray.
+
+Now, we'll use `ray symmetric-run` to start Ray on all nodes with given CPU and GPU resources and run your entrypoint script ONLY the head node.
 
 Below, you'll see that we explicitly specify the number of CPUs (``num-cpus``)
 and number of GPUs (``num-gpus``) to Ray, as this will prevent Ray from using
 more resources than allocated. We also need to explicitly
-indicate the ``node-ip-address`` for the Ray head runtime:
-
-.. literalinclude:: /cluster/doc_code/slurm-basic.sh
-   :language: bash
-   :start-after: __doc_head_ray_start__
-   :end-before: __doc_head_ray_end__
-
-By backgrounding the above srun task, we can proceed to start the Ray worker runtimes.
-
-Starting the Ray worker nodes
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Below, we do the same thing, but for each worker. Make sure the Ray head and Ray worker processes are not started on the same node.
+indicate the ``address`` parameter for the head node to identify itself and other nodes to connect to:
 
 .. literalinclude:: /cluster/doc_code/slurm-basic.sh
    :language: bash
-   :start-after: __doc_worker_ray_start__
-   :end-before: __doc_worker_ray_end__
-
-Submitting your script
-~~~~~~~~~~~~~~~~~~~~~~
-
-Finally, you can invoke your Python script:
-
-.. literalinclude:: /cluster/doc_code/slurm-basic.sh
-   :language: bash
-   :start-after: __doc_script_start__
+   :start-after: __doc_symmetric_run_start__
+   :end-before: __doc_symmetric_run_end__
 
+After the training job is completed, the Ray cluster will be stopped automatically.
 
 .. note:: The -u argument tells python to print to stdout unbuffered, which is important with how slurm deals with rerouting output. If this argument is not included, you may get strange printing behavior such as printed statements not being logged by slurm until the program has terminated.
 
@@ -165,6 +140,7 @@ One common use of a SLURM cluster is to have multiple users running concurrent
 jobs on the same infrastructure. This can easily conflict with Ray due to the
 way the head node communicates with its workers.
 
+
 Considering 2 users, if they both schedule a SLURM job using Ray
 at the same time, they are both creating a head node. In the backend, Ray will
 assign some internal ports to a few services. The issue is that as soon as the
@@ -183,13 +159,12 @@ adjusted. For an explanation on ports, see :ref:`here <ray-ports>`::
     --ray-client-server-port
     --redis-shard-ports
 
-For instance, again with 2 users, they would have to adapt the instructions
-seen above to:
+For instance, again with 2 users, they would run the following commands. Note that we don't use symmetric-run here
+because it does not currently work in multi-tenant environments:
 
 .. code-block:: bash
 
   # user 1
-  # same as above
   ...
   srun --nodes=1 --ntasks=1 -w "$head_node" \
       ray start --head --node-ip-address="$head_node_ip" \
@@ -202,8 +177,9 @@ seen above to:
           --max-worker-port=19999 \
           --num-cpus "${SLURM_CPUS_PER_TASK}" --num-gpus "${SLURM_GPUS_PER_TASK}" --block &
 
+    python -u your_script.py
+
   # user 2
-  # same as above
   ...
   srun --nodes=1 --ntasks=1 -w "$head_node" \
       ray start --head --node-ip-address="$head_node_ip" \
@@ -216,12 +192,15 @@ seen above to:
           --max-worker-port=29999 \
           --num-cpus "${SLURM_CPUS_PER_TASK}" --num-gpus "${SLURM_GPUS_PER_TASK}" --block &
 
+    python -u your_script.py
+
 As for the IP binding, on some cluster architecture the network interfaces
 do not allow to use external IPs between nodes. Instead, there are internal
 network interfaces (`eth0`, `eth1`, etc.). Currently, it's difficult to
 set an internal IP
 (see the open `issue <https://github.com/ray-project/ray/issues/22732>`_).
 
+
 Python-interface SLURM scripts
 ------------------------------
 

python/ray/scripts/symmetric_run.py

@@ -84,7 +84,7 @@ def curate_and_validate_ray_start_args(run_and_start_args: List[str]) -> List[st
 
 USAGE:
 
-    python -m ray.scripts.symmetric_run --address ADDRESS
+    ray symmetric-run --address ADDRESS
 [--min-nodes NUM_NODES] [RAY_START_OPTIONS] -- [ENTRYPOINT_COMMAND]
 
 DESCRIPTION:
@@ -100,15 +100,15 @@ def curate_and_validate_ray_start_args(run_and_start_args: List[str]) -> List[st
 
     # Start Ray with default settings and run a Python script
 
-    python -m ray.scripts.symmetric_run --address 127.0.0.1:6379 -- python my_script.py
+    ray symmetric-run --address 127.0.0.1:6379 -- python my_script.py
 
     # Start Ray with specific head node and run a command
 
-    python -m ray.scripts.symmetric_run --address 127.0.0.1:6379 --min-nodes 4 -- python train_model.py --epochs=100
+    ray symmetric-run --address 127.0.0.1:6379 --min-nodes 4 -- python train_model.py --epochs=100
 
     # Start Ray and run a multi-word command
 
-    python -m ray.scripts.symmetric_run --address 127.0.0.1:6379 --min-nodes 4 --num-cpus=4 -- python -m my_module --config=prod
+    ray symmetric-run --address 127.0.0.1:6379 --min-nodes 4 --num-cpus=4 -- python -m my_module --config=prod
 
 RAY START OPTIONS: