CI execution over Slurm managed clusters

[MrBr-github]

What

This PR adds execution over scctl managed slurm clusters

slurm_cmd.sh - provides abstraction to execute slurm based commands
run_slurm_tests_ucc.sh - basic test file
fix_enroot.sh - comes to mitigate enroot issue to handle errors with anonymous image access`

Issue: HPCINFRA-3983

[greptile-apps]

Skipped: This PR does not contain any of your configured labels: (Ready-For-Review)

[MrBr-github]

Working example https://nbuprod.blsm.nvidia.com/hpcx-01/blue/organizations/jenkins/UCC%2Fucc_slurm_cluster_tests/detail/ucc_slurm_cluster_tests/44/pipeline

[dpressle]

The whole design is not clear, the main work here is to run NVLS tests, not SLURM tests, slurm is just a way to run these tests on specific HW/clusters but it is used as the main subject in the code.

[dpressle]

use the existing jjb yaml, we dont want to duplicate as many variables are common to all jobs

[mike-dubman]

very cool
imho - it should use opportunistic approach, i,e. - try to use scctl to submit (check if desired amount of nodes available like dry run) use salloc --immediate=sec flag that if salloc waits for nodes more than sec - it will bail and go for fallback

[MrBr-github]

very cool imho - it should use opportunistic approach, i,e. - try to use scctl to submit (check if desired amount of nodes available like dry run) use salloc --immediate=sec flag that if salloc waits for nodes more than sec - it will bail and go for fallback

@mike-dubman
Opportunism (like in clusterminder) is a great idea
However I don't quite understand what do you mean by fallback?
Only alternative I see right now for opportunism is to approach multiple partitions with salloc -p jazz.rock -N 2
In later stages, when we'll have multiple managed slurm clusters with suitable HW (not partitions), we can loop through them (like hpchead partition and scctl)

[mike-dubman]

very cool imho - it should use opportunistic approach, i,e. - try to use scctl to submit (check if desired amount of nodes available like dry run) use salloc --immediate=sec flag that if salloc waits for nodes more than sec - it will bail and go for fallback

@mike-dubman Opportunism (like in clusterminder) is a great idea However I don't quite understand what do you mean by fallback? Only alternative I see right now for opportunism is to approach multiple partitions with salloc -p jazz.rock -N 2 In later stages, when we'll have multiple managed slurm clusters with suitable HW (not partitions), we can loop through them (like hpchead partition and scctl)

fallback to static servers you have today?

[MrBr-github]

very cool imho - it should use opportunistic approach, i,e. - try to use scctl to submit (check if desired amount of nodes available like dry run) use salloc --immediate=sec flag that if salloc waits for nodes more than sec - it will bail and go for fallback

@mike-dubman Opportunism (like in clusterminder) is a great idea However I don't quite understand what do you mean by fallback? Only alternative I see right now for opportunism is to approach multiple partitions with salloc -p jazz.rock -N 2 In later stages, when we'll have multiple managed slurm clusters with suitable HW (not partitions), we can loop through them (like hpchead partition and scctl)

fallback to static servers you have today?

Oh, got you
Main idea of this PR was to execute tests over HW not available on static servers, in particular NVLS test
But opportunistic approach opens new use cases for the same tests
Will discuss possibilities internally