Fix CUDA plugin CI.

[ysiraichi]

Fix: #8577

This PR reverts #8286, and bumps CUDA version to 12.3. The latter is needed for successfully compiling GPU dependent source code that makes use of CUgraphConditionalHandle (not available in 12.1) driver API typedef.

[amjames]

Looks like the failing jobs are due to a failed clone from kleidiai's gitlab. Is that a widespread issue or spurious failure?

[ysiraichi]

It doesn't look widespread (haven't seen in other PRs). I will try rebasing this PR.

[tengyifei]

@ysiraichi from pytorch/pytorch#138609 (comment), it looks like PyTorch upstream decided to release with some specific set of CUDA versions (see issue). Can we use one of their chosen versions, for example CUDA 12.4 instead of CUDA 12.3?

[ysiraichi]

Problem is: I didn't find a docker image with CUDA 12.4. Also, I'm not sure how to create one, since it seems something internal.

[tengyifei]

Problem is: I didn't find a docker image with CUDA 12.4. Also, I'm not sure how to create one, since it seems something internal.

Could you clarify this challenge? Do you mean that you were hoping to find a torch_xla CUDA 12.4 docker build?

[ysiraichi]

As far as I understand, PyTorch/XLA CI relies on docker images (see dev-image). My point is that there is no docker image with CUDA 12.4 in that registry.

xla/.github/workflows/build_and_test.yml

Lines 44 to 52 in fbbdfca

	# build-torch-with-cuda:
	# name: "Build PyTorch with CUDA"
	# uses: ./.github/workflows/_build_torch_with_cuda.yml
	# needs: get-torch-commit
	# with:
	# # note that to build a torch wheel with CUDA enabled, we do not need a GPU runner.
	# dev-image: us-central1-docker.pkg.dev/tpu-pytorch-releases/docker/development:3.10_cuda_12.1
	# torch-commit: ${{needs.get-torch-commit.outputs.torch_commit}}
	# runner: linux.24xlarge

[tengyifei]

As far as I understand, PyTorch/XLA CI relies on docker images (see dev-image). My point is that there is no docker image with CUDA 12.4 in that registry.

@ysiraichi got it. thanks for the explanation. I think using CUDA 12.3 for now is a-okay. IIUC, most of the time we're only using torch CPU + torch_xla GPU in any case.

[tengyifei]

LMK when I should review. It looks like there are still some failed tests.