[CI] Limit PyTorch testing to a single Python version

[ScottTodd]

We currently run PyTorch tests nightly across the full matrix of

• GPUs we have test runners for: [gfx942, gfx950, gfx110X, gfx1151, gfx120X, gfx90a, gfx103X, etc.]
• Operating systems: [windows, linux]
• PyTorch release versions: [release/2.12, release/2.13, release/2.14, nightly]
• Python versions: [3.10, 3.11, 3.12, 3.13, 3.14]

For example: https://github.com/ROCm/rockrel/actions/runs/32806241769

Some of this test coverage is redundant and puts excess pressure on runners that are in limited supply.

Upstream PyTorch only tests on python 3.12 at the moment, so we can limit to that also to cut our runner load in 5 (separately we can also drop support for python 3.10 now).

cc @HereThereBeDragons @Melantha-S @pragupta @jithunnair-amd @ethanwee1 @pruthvistony

Keeping some coverage

We should still run wheel sanity checks on CPU builders and possibly GPU sanity checks, at least running import torch and perhaps https://github.com/ROCm/TheRock/blob/main/external-builds/pytorch/smoke-tests/pytorch_smoke_test.py instead of the 20+ minutes of pytorch unit tests that we run normally. For smoketests we probably also don't need to checkout pytorch source code or install full test requirements.

Potential savings

Metrics and analysis compiled with assistance from an agent

How much does this save within PyTorch nightly CI?

The example PyTorch run consumed 70.80 GPU runner-hours across 180 jobs. Limiting the full tests to Python 3.12 would have retained 36 jobs and consumed 14.31 GPU runner-hours, avoiding 56.49 GPU runner-hours (79.8%) in this run.

The ROCm artifact tests column puts each PyTorch number next to a representative daily Linux artifact-test run on the same family-specific runner pool.

Linux gfx family	PyTorch: all 5 Python versions	PyTorch: Python 3.12 only	PyTorch time avoided	ROCm artifact tests
gfx103X-all	2.28 h	0.04 h	2.24 h	1.97 h
gfx110X-all	14.61 h	3.17 h	11.44 h	11.47 h
gfx1150	7.32 h	1.46 h	5.86 h	16.33 h
gfx1151	4.18 h	0.84 h	3.34 h	10.43 h
gfx1153	1.50 h	0.10 h	1.41 h	33.64 h
gfx120X-all	5.71 h	1.13 h	4.58 h	13.73 h
gfx90a	2.08 h	0.41 h	1.67 h	not in August 28 dispatch
gfx94X-dcgpu	17.82 h	3.50 h	14.31 h	31.55 h
gfx950-dcgpu	15.30 h	3.65 h	11.65 h	not in August 28 dispatch
Total	70.80 h	14.31 h	56.49 h	119.13 h across 7 families

How much does this save within overall nightly CI?

For the measured Linux GPU scope in the table, PyTorch and ROCm artifact tests currently consume 189.92 runner-hours per nightly cycle:

• PyTorch tests: 70.80 hours (37.3%)
• ROCm artifact tests: 119.13 hours (62.7%) across seven families

Running full PyTorch tests only on Python 3.12 and moving the other wheel checks to CPU/scalable runners saves 56.49 GPU runner-hours. That is 79.8% of the PyTorch test load and 29.7% of this measured overall Linux nightly GPU load. Retaining GPU sanity jobs lowers the overall saving to 26.6%-27.0%.

[ScottTodd]

I looked at a few different implementation options given how we have different "test filter level" settings across

• ROCm artifacts: https://github.com/ROCm/TheRock/blob/main/docs/development/test_filtering.md
• PyTorch "full" tests:

  TheRock/.github/workflows/multi_arch_build_portable_linux_pytorch_wheels.yml

  Lines 564 to 571 in a84a743

  	dispatch_pytorch_wheels_full_test:
  	name: Dispatch PyTorch Full Test | torch ${{ inputs.pytorch_git_ref }}
  	if: >-
  	${{
  	inputs.run_full_pytorch_tests &&
  	inputs.python_version == '3.12' &&
  	contains(inputs.amdgpu_families, 'gfx94X-dcgpu')
  	}}

• JAX test levels: https://github.com/ROCm/TheRock/blob/main/build_tools/github_actions/configure_jax_test_matrix.py

What I think makes the most sense is:

1. build_tools/github_actions/configure_pytorch_release_matrix.py in .github/workflows/multi_arch_release_linux_pytorch_wheels.yml sets a test level on each [torch version, python version] matrix entry from 'sanity', 'standard', 'full'
   • Python 3.12 gets 'standard' or 'full' depending on the schedule and release type
   • Other Python versions get 'sanity' (or --release-type=prerelease runs could use 'standard')
   • The existing run_full_pytorch_tests boolean can control whether 'full' is allowed/used or just 'standard'
2. build_tools/github_actions/configure_pytorch_test_matrix.py in .github/workflows/multi_arch_build_portable_linux_pytorch_wheels.yml expands that level into actual test_runs_on child jobs
   • 'sanity' tests run either just as part of the CPU build (already exists) or on GPU runners with a narrower set of tests and setup steps (keeping overhead low)
   • if the build workflow is run directly with workflow_dispatch for a python version other than 3.12 it should still default to the 'standard' test level