Skip to content

Track standalone GFD and CUDA validator E2E enablement #446

Description

@giuliocalzo

Summary

The legacy nvml-mock-e2e.yaml workflow keeps the standalone GFD and CUDA validator steps disabled by default with if: false because the standalone images are pulled from nvcr.io and may require NGC authentication.

The Go/Ginkgo E2E harness has parity scaffolding for these steps, but the specs should remain skipped by default until the image/auth story is resolved.

Legacy reference

Disabled legacy workflow section pinned to the current default-branch commit:

# GFD and CUDA validator standalone images from nvcr.io may require NGC auth.
# The GPU Operator path (e2e-gpu-operator job) bundles these publicly and works
# without credentials. These standalone steps are disabled until verified public.
# See tests/e2e/README.md for local testing instructions.
- name: Deploy GPU Feature Discovery
if: false
run: |
docker pull nvcr.io/nvidia/gpu-feature-discovery:v0.17.0
kind load docker-image nvcr.io/nvidia/gpu-feature-discovery:v0.17.0 --name nvml-mock-e2e-${{ matrix.gpu-profile }}
kubectl apply -f tests/e2e/gfd-mock.yaml
kubectl -n kube-system wait --for=condition=ready \
pod -l name=nvidia-gfd-mock --timeout=120s
- name: Verify GFD node labels
if: false
run: |
NODE=$(kubectl get nodes -o jsonpath='{.items[0].metadata.name}')
for i in $(seq 1 30); do
PRODUCT=$(kubectl get node "$NODE" -o jsonpath='{.metadata.labels.nvidia\.com/gpu\.product}' 2>/dev/null || true)
if [ -n "$PRODUCT" ]; then break; fi
echo "Attempt $i: waiting for GFD labels..."
sleep 2
done
echo "Checking GFD labels on node $NODE..."
for LABEL in nvidia.com/gpu.product nvidia.com/gpu.memory nvidia.com/gpu.compute.major; do
VALUE=$(kubectl get node "$NODE" -o json | jq -r ".metadata.labels[\"$LABEL\"] // empty")
if [ -z "$VALUE" ]; then
echo "FAIL: GFD label $LABEL not set"
kubectl get node "$NODE" -o json | jq '.metadata.labels | to_entries[] | select(.key | startswith("nvidia.com"))' || true
exit 1
fi
echo " $LABEL=$VALUE"
done
echo "PASS: GFD labels verified"
- name: Run CUDA validator job
if: false
run: |
docker pull nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda12.5.0
kind load docker-image nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda12.5.0 --name nvml-mock-e2e-${{ matrix.gpu-profile }}
kubectl apply -f tests/e2e/validator-mock.yaml
kubectl wait --for=condition=complete job/gpu-validator-mock --timeout=120s || {
echo "Validator job did not complete. Checking status..."
kubectl describe job/gpu-validator-mock
kubectl logs -l name=gpu-validator-mock --tail=50 || true
exit 1
}
echo "PASS: CUDA validator job completed successfully"

Disabled legacy steps

  • Deploy GPU Feature Discovery
  • Verify GFD node labels
  • Run CUDA validator job

Acceptance criteria

  • Decide how CI should access the standalone nvcr.io images, or replace them with images that are public/fork-safe.
  • Enable the Go validator/GFD specs in CI only when credentials or public images are available.
  • Keep fork PR behavior safe and deterministic.
  • Update tests/e2e/README.md and .github/workflows/nvml-mock-e2e-go.yaml with the final enablement path.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions