[Bug]: E2E test should be smarter about GPU saturation

[MikeSpreitzer]

Contact Details

No response

What happened?

The E2E test on OpenShift failed for #420 . Because the assumption identified below was not actually true at that time and place.

The E2E test suite currently assumes (in the "Same-Node Port Collision Creates New Launcher" test case) that it can use 2 GPUs on the node where the server-requesting Pod is initially assigned. That is simply not always true.

It would be great if the test case could detect that situation and react in some sensible manner. It would be good if the debug dumping steps at the end of the job included one that exposed this situation.

This is really just a special case of the larger problem that GPU availability is dynamic in the shared cluster. Every test step that requires a GPU to be allocated is making an assumption of GPU availability that might not be true.

Version

main (please specify commit below)

Branch name

No response

Commit SHA

No response

Relevant log output

[MikeSpreitzer]

Discussed in a group meeting this morning. Consensus is that since this failure case is rare now, the best approach is the simplest: retry the test when it fails this way. Maybe add some logic to the test case to detect this failure mode and print an explanatory message.

[MikeSpreitzer]

I created PR #426 to implement the suggested solution, but the manual retry strategy is not working well. There is currently a Node that seems to be an attractive nuisance. That node has 1 GPU free, when my PR is not being tested. Every try of the test gets scheduled to that Node and fails. The screen snapshot below shows two nuisance nodes.

This is consistent with my usual experience with this cluster. While the GPU usage can change at any time, it is often steady for long periods of time.

[MikeSpreitzer]

OTOH, the retry at https://github.com/llm-d-incubation/llm-d-fast-model-actuation/actions/runs/24434942280/job/71456786539 did NOT go to a nuisance node, even though the inventory (shown below) was the same as for the previous failures.