fix(ci): cache heavy CUDA wheels in install-test (#1796)

[KevinSailema]

Reduce Python+CUDA installation time in CI by prebuilding and caching heavyweight CUDA dependency wheels, then reusing those wheels in install jobs.

Changelog

• Add a dedicated CUDA wheelhouse job to build wheels for heavy dependencies used by CUDA extras.
• Add GitHub Actions cache for the wheelhouse, keyed by OS, Python version, dependency lock inputs, and workflow file hash.
• Add fail-fast validation so the workflow errors early if expected heavyweight wheels are not produced.
• Upload built wheels as an artifact for downstream jobs in the same workflow run.
• Update the CUDA pip install job to depend on wheelhouse generation and download wheel artifacts before install.
• Update pip install behavior to prefer local prebuilt wheels first, while preserving fallback behavior when cache/artifact is cold.
• Update the install summary job dependencies to include the new wheelhouse job so gating stays consistent.

Before your PR is "Ready for review"

Pre checks:

• Make sure you read and followed Contributor guidelines
• Did you write any new necessary tests?
• Did you add or update any necessary documentation?

Notes:

• This is a CI workflow optimization change; no application/runtime logic was modified.
• Validation is based on workflow execution success and install-time reduction in CI runs.

Additional Information

• Related to Optimize causal-conv1d/transformer engine installation in CI #1796
• Cold-cache behavior: first run may still compile once.
• Warm-cache behavior: subsequent runs should reuse cached wheels and reduce install time significantly.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[akoumpa]

Thanks a lot for making these @KevinSailema ! I'm not an area expert, so I wanted to ask you, can this also be used when the docker container is made that run on the gha ci workers? because initially, it tests installation on a few targets [20 min], then builds the docker container [20 min], so I was wondering if your fix could be reused?

Hi, @thomasdhc can you review as the domain expert? Thank you.

[thomasdhc]

/ok to test cc8cb2d

[thomasdhc]

@KevinSailema Please review failures

[akoumpa]

/ok to test 5368547