Skip to content

Commit cd3d50c

Browse files
committed
fix(validator): isolate each NCCL benchmark run in its own namespace
The NCCL all-reduce TrainJob/TrainingRuntime/ComputeDomain/RoCE-claim resources are applied by fixed name in the shared, persistent aicr-validation namespace, so two concurrent `aicr validate` runs (or a crashed run and its retry) could delete or adopt each other's resources. This affects every NCCL leaf in the catalog (EFA, TCPXO, RoCE, NVLS across A100/H100/B200/GB200/GB300), not just one accelerator or fabric. Give each run its own namespace instead, mirroring the pattern inference-perf already uses (deriveRunID/ensureNamespace): every fixed resource name only has to be unique within its own namespace, so concurrent or crashed runs can never collide, adopt, or delete each other's resources. Cleanup becomes a single namespace delete relying on Kubernetes cascading garbage collection, instead of tracking and UID-pinning a delete for each individual resource. cleanupNCCLResources also only logged delete failures rather than returning them, letting a real cleanup failure disappear silently. Return them instead, via foldCleanupError (now parameterized on its fallback message instead of hardcoding the Trainer-specific one) so a genuine cleanup failure fails the check. Signed-off-by: Mike Cook <micook@nvidia.com>
1 parent f457427 commit cd3d50c

5 files changed

Lines changed: 156 additions & 153 deletions

File tree

0 commit comments

Comments
 (0)