Commit cd3d50c
committed
fix(validator): isolate each NCCL benchmark run in its own namespace
The NCCL all-reduce TrainJob/TrainingRuntime/ComputeDomain/RoCE-claim
resources are applied by fixed name in the shared, persistent
aicr-validation namespace, so two concurrent `aicr validate` runs (or
a crashed run and its retry) could delete or adopt each other's
resources. This affects every NCCL leaf in the catalog (EFA, TCPXO,
RoCE, NVLS across A100/H100/B200/GB200/GB300), not just one
accelerator or fabric.
Give each run its own namespace instead, mirroring the pattern
inference-perf already uses (deriveRunID/ensureNamespace): every fixed
resource name only has to be unique within its own namespace, so
concurrent or crashed runs can never collide, adopt, or delete each
other's resources. Cleanup becomes a single namespace delete relying
on Kubernetes cascading garbage collection, instead of tracking and
UID-pinning a delete for each individual resource.
cleanupNCCLResources also only logged delete failures rather than
returning them, letting a real cleanup failure disappear silently.
Return them instead, via foldCleanupError (now parameterized on its
fallback message instead of hardcoding the Trainer-specific one) so a
genuine cleanup failure fails the check.
Signed-off-by: Mike Cook <micook@nvidia.com>1 parent f457427 commit cd3d50c
5 files changed
Lines changed: 156 additions & 153 deletions
File tree
- validators/performance
0 commit comments