Skip to content

Commit b8f6061

Browse files
ray-train: fix build (nvcc) + schedule, reuse shared training sanity test
Build fix (CI failure on PR): - runtime-base: install cuda-nvcc + cuda-cudart-devel (base runtime image has no CUDA compiler; nccl-tests build needs nvcc). Add global CUDA_VERSION ARG. - register ray-train.autorelease-gpu.yml cron in release-schedule.yml (moved to a free slot 00 23 * * 2,4 to avoid colliding with vllm-omni) Sanity test reuse (per repo convention — sanity scripts are shared by category, not per-framework): - add 'ray' to training_cluster_only in test_sanity_training.py so RayTrain runs the shared training contract (env/PATH/EFA/NCCL/CUDA/cuDNN/SSH/venv/OSS) - gate OpenMPI double-wrap + entrypoint.sh checks to pt_tf_only (RayTrain uses EFA's bundled OpenMPI and a passive/KubeRay entrypoint) - add ray-gated TestRayTrain class (ray version, extras import, serve absent) - delete standalone test_sanity_ray_train.py; wire the shared script in the workflow
1 parent 8f24076 commit b8f6061

7 files changed

Lines changed: 82 additions & 107 deletions

File tree

.github/release-schedule.yml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -89,3 +89,7 @@ schedules:
8989
- cron: "30 22 * * 2"
9090
workflow: openfold3.autorelease-sagemaker-amzn2023.yml
9191
gpu: heavy
92+
93+
- cron: "00 23 * * 2,4"
94+
workflow: ray-train.autorelease-gpu.yml
95+
gpu: heavy

.github/workflows/_reusable.sanity-tests.yml

Lines changed: 3 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -160,23 +160,16 @@ jobs:
160160
if: |
161161
needs.preflight.outputs.framework == 'pytorch_runtime' ||
162162
needs.preflight.outputs.framework == 'tensorflow' ||
163-
needs.preflight.outputs.framework == 'xgboost'
163+
needs.preflight.outputs.framework == 'xgboost' ||
164+
(needs.preflight.outputs.framework == 'ray' && needs.preflight.outputs.job-type == 'training')
164165
run: |
165166
docker exec \
166167
-e EXPECTED_FRAMEWORK=${{ needs.preflight.outputs.framework }} \
168+
-e EXPECTED_FRAMEWORK_VERSION=${{ needs.preflight.outputs.framework-version }} \
167169
-e EXPECTED_DEVICE=${{ needs.preflight.outputs.device-type }} \
168170
-e EXPECTED_CUSTOMER=${{ needs.preflight.outputs.customer-type }} \
169171
${CONTAINER_ID} python3 /workdir/test/sanity/scripts/test_sanity_training.py
170172
171-
- name: Run RayTrain image sanity checks
172-
if: |
173-
needs.preflight.outputs.framework == 'ray' &&
174-
needs.preflight.outputs.job-type == 'training'
175-
run: |
176-
docker exec \
177-
-e EXPECTED_FRAMEWORK_VERSION=${{ needs.preflight.outputs.framework-version }} \
178-
${CONTAINER_ID} python3 /workdir/test/sanity/scripts/test_sanity_ray_train.py
179-
180173
- name: Run OSS compliance check
181174
run: |
182175
docker exec ${CONTAINER_ID} /usr/local/bin/testOSSCompliance /root

.github/workflows/ray-train.autorelease-gpu.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@ name: "Auto Release - Ray Train GPU"
22

33
on:
44
schedule:
5-
- cron: '00 19 * * 2,4'
5+
- cron: '00 23 * * 2,4'
66
workflow_dispatch:
77

88
concurrency:

docker/ray-train/Dockerfile.cuda

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@ ARG PYTHON_VERSION=3.13
2525
ARG PYTHON_SHORT_VERSION=3.13
2626
ARG FRAMEWORK_VERSION=2.56.0
2727
ARG TORCH_VERSION=2.13.0
28+
ARG CUDA_VERSION=13.0.2
2829
ARG EFA_VERSION=1.47.0
2930
ARG GDRCOPY_VERSION=2.4.4
3031
# Base CUDA DLC (we own it). devel for builder stages, runtime for output.
@@ -69,6 +70,7 @@ ARG PYTHON_VERSION
6970
ARG PYTHON_SHORT_VERSION
7071
ARG TORCH_VERSION
7172
ARG FRAMEWORK_VERSION
73+
ARG CUDA_VERSION
7274
ARG DLC_MAJOR_VERSION
7375
ARG DLC_MINOR_VERSION
7476
ARG EFA_VERSION
@@ -93,9 +95,13 @@ ENV PYTHONDONTWRITEBYTECODE=1 \
9395
NCCL_SOCKET_IFNAME=eth0 \
9496
NCCL_DEBUG=INFO
9597

96-
# Build tools needed by the interconnect installers + nccl-tests.
97-
RUN dnf install -y --allowerasing \
98-
shadow-utils tar gzip curl which findutils util-linux gcc gcc-c++ make \
98+
# Build tools needed by the interconnect installers + nccl-tests. The base
99+
# runtime image has no CUDA compiler, so add cuda-nvcc + cudart-devel (nccl-tests
100+
# needs nvcc). Package suffix derives from CUDA version: 13.0.2 -> 13-0.
101+
RUN CUDA_MAJOR_MINOR=$(echo "${CUDA_VERSION}" | cut -d. -f1,2 | tr '.' '-') \
102+
&& dnf install -y --allowerasing \
103+
shadow-utils tar gzip curl which findutils util-linux gcc gcc-c++ make \
104+
cuda-nvcc-${CUDA_MAJOR_MINOR} cuda-cudart-devel-${CUDA_MAJOR_MINOR} \
99105
&& dnf clean all
100106

101107
# Python venv (base + torch/ray/HF deps from builder-base)

test/ray-train/README.md

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -2,10 +2,14 @@
22

33
Current coverage (basic):
44

5-
- **Sanity**`test/sanity/scripts/test_sanity_ray_train.py`, wired into
6-
`_reusable.sanity-tests.yml` (guarded on `framework == ray && job_type == training`).
7-
Import/version-level: verifies Ray 2.56.0, a CUDA torch build, the HF/Lightning
8-
training stack, and that the Ray Serve extra is absent. Runs on a CPU sanity runner.
5+
- **Sanity** — reuses the shared training sanity test
6+
`test/sanity/scripts/test_sanity_training.py` (RayTrain is added to
7+
`training_cluster_only`, like the other training frameworks — no per-framework
8+
file, matching repo convention). It verifies the shared training contract
9+
(env, PATH, EFA/NCCL, CUDA, cuDNN, SSH, venv, OSS) plus a RayTrain-only
10+
`TestRayTrain` class (Ray version, `ray[train,tune,data]` importable, CUDA torch
11+
build, Ray Serve extra absent). Wired in `_reusable.sanity-tests.yml` on
12+
`framework == ray && job_type == training`. Runs on a CPU sanity runner.
913
- **Security** — shared Ray ECR scan allowlist at
1014
`test/security/data/ecr_scan_allowlist/ray/` (RayTrain uses `framework: ray`, so it
1115
shares the allowlist with the Ray Serve DLC).

test/sanity/scripts/test_sanity_ray_train.py

Lines changed: 0 additions & 82 deletions
This file was deleted.

test/sanity/scripts/test_sanity_training.py

Lines changed: 57 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -47,13 +47,21 @@
4747
cpu_only = unittest.skipIf(DEVICE != "cpu", "CPU-only test")
4848
sagemaker_only = unittest.skipIf(CUSTOMER != "sagemaker", "SageMaker-only test")
4949
tensorflow_only = unittest.skipIf(FRAMEWORK != "tensorflow", "TF-only test")
50-
# Tests gated with this decorator assume PT/TF-style training DLC
51-
# (SSH cluster, MPI, entrypoint script). xgboost is a SageMaker
52-
# algorithm container and does not honor this contract.
50+
# Tests gated with this decorator assume a training DLC with the SSH+MPI cluster
51+
# stack and /opt/venv. Covers PyTorch, TensorFlow, and RayTrain (all reuse the
52+
# PyTorch EFA/NCCL/SSH install scripts). xgboost is a SageMaker algorithm
53+
# container and does not honor this contract.
5354
training_cluster_only = unittest.skipUnless(
54-
FRAMEWORK in {"tensorflow", "pytorch_runtime"},
55+
FRAMEWORK in {"tensorflow", "pytorch_runtime", "ray"},
5556
"training-cluster-only test (requires SSH+MPI stack; xgboost is algorithm container)",
5657
)
58+
# RayTrain reuses EFA's bundled OpenMPI and a passive/KubeRay entrypoint — it does
59+
# NOT rebuild OpenMPI from source and has no /usr/local/bin/entrypoint.sh, so the
60+
# double-wrap and entrypoint-script checks below do not apply to it.
61+
pt_tf_only = unittest.skipUnless(
62+
FRAMEWORK in {"tensorflow", "pytorch_runtime"},
63+
"PyTorch/TensorFlow-only test (source-built OpenMPI + entrypoint.sh contract)",
64+
)
5765

5866

5967
class TestContainerEnv(unittest.TestCase):
@@ -204,11 +212,13 @@ class TestOpenMPI(unittest.TestCase):
204212
def test_openmpi_binary_exists(self):
205213
self.assertTrue(os.access("/opt/amazon/openmpi/bin/mpirun", os.X_OK))
206214

215+
@pt_tf_only
207216
def test_openmpi_double_wrap(self):
208217
"""`mpirun` is a wrapper that exec's `mpirun.real --allow-run-as-root`. Verify the
209218
wrapper-vs-real split is in place — single grep -c match means it's NOT
210219
double-wrapped (which would happen if EFA's bundled OMPI wasn't wiped before
211-
the from-source build)."""
220+
the from-source build). RayTrain uses EFA's bundled OpenMPI as-is, so this
221+
source-build-specific check does not apply."""
212222
self.assertTrue(os.access("/opt/amazon/openmpi/bin/mpirun.real", os.X_OK))
213223
out = subprocess.check_output(
214224
["grep", "-c", "mpirun.real", "/opt/amazon/openmpi/bin/mpirun"], text=True
@@ -315,13 +325,53 @@ def test_venv_bin_exists(self):
315325
self.assertTrue(os.path.isdir("/opt/venv/bin"))
316326

317327

318-
@training_cluster_only
328+
@pt_tf_only
319329
class TestEntrypoint(unittest.TestCase):
320-
"""Entrypoint script is executable (universal)."""
330+
"""Entrypoint script is executable. PyTorch/TensorFlow ship
331+
/usr/local/bin/entrypoint.sh; RayTrain uses a passive/KubeRay entrypoint and
332+
does not, so this is gated to PT/TF only."""
321333

322334
def test_entrypoint_executable(self):
323335
self.assertTrue(os.access("/usr/local/bin/entrypoint.sh", os.X_OK))
324336

325337

338+
ray_only = unittest.skipUnless(FRAMEWORK == "ray", "RayTrain-only test")
339+
340+
341+
@ray_only
342+
class TestRayTrain(unittest.TestCase):
343+
"""RayTrain-specific contract: Ray training stack present, training-scoped
344+
(no Ray Serve extra). Gated on EXPECTED_FRAMEWORK == 'ray'."""
345+
346+
def test_ray_train_stack_imports(self):
347+
import accelerate # noqa: F401
348+
import datasets # noqa: F401
349+
import pytorch_lightning # noqa: F401
350+
import ray.data # noqa: F401
351+
import ray.train # noqa: F401
352+
import ray.tune # noqa: F401
353+
import transformers # noqa: F401
354+
355+
def test_ray_version_matches_expected(self):
356+
expected = os.environ.get("EXPECTED_FRAMEWORK_VERSION", "")
357+
if not expected:
358+
self.skipTest("EXPECTED_FRAMEWORK_VERSION not set")
359+
import ray
360+
361+
self.assertEqual(ray.__version__, expected)
362+
363+
def test_torch_is_cuda_build(self):
364+
import torch
365+
366+
self.assertIn("cu", torch.__version__)
367+
368+
def test_ray_serve_extra_absent(self):
369+
"""RayTrain is training-scoped. The `ray.serve` module always ships in the
370+
ray package, but without the `serve` extra its deps (e.g. starlette) are
371+
missing and the import fails — a successful import means serve leaked in."""
372+
with self.assertRaises(ImportError):
373+
import ray.serve # noqa: F401
374+
375+
326376
if __name__ == "__main__":
327377
unittest.main(verbosity=2)

0 commit comments

Comments
 (0)