Add Dynamic MIG tests to Lambda CI with nvmm no-op stub

Enable the existing DynMIG test suite on Lambda by adding a no-op
nvmm stub and wiring up GPU-type-aware filtering.

Lambda nvmm (tests/bats/lib/lambda/nvmm):
- No-op stub that accepts and discards nvmm arguments. On Lambda,
  MIG cleanup is handled by e2e-test.sh on the host via SSH (where
  nvidia-smi is available).
- NVMM_PATH env var selects which nvmm to use. Defaults to the
  original (tests/bats/lib). Lambda CI sets it to the lambda
  version (tests/bats/lib/lambda).

Dynamic MIG on Lambda:
- test_gpu_dynmig.bats tagged 'dynmig' for auto-filtering
- e2e-test.sh: skip dynmig on non-MIG GPUs (V100, A10); run on
  A100, H100, GH200, B200
- test_gpu_dynmig.bats added to tests-gpu-single Makefile target
- cleanup-from-previous-run.sh: make nvmm MIG cleanup non-fatal

Also removes SKIP_CLEANUP by handling MIG pre-cleanup on the host
via SSH and using the lambda nvmm no-op for in-container calls.

Signed-off-by: Davanum Srinivas <[email protected]>

Author: dims

hack/ci/lambda/e2e-test.sh

@@ -84,12 +84,17 @@ fi
 case "${LAMBDA_GPU_TYPE}" in
   *v100*|*a10) FILTER="${FILTER},!gpu-busgrind" ;;
 esac
+# Dynamic MIG requires MIG-capable GPUs (A100/H100/GH200/B200).
+case "${LAMBDA_GPU_TYPE}" in
+  *a100*|*h100*|*gh200*|*b200*) ;;
+  *) FILTER="${FILTER},!dynmig" ;;
+esac
 echo "Test filter: ${FILTER}"
 
 # --- Pre-cleanup: MIG teardown on host ---
 # Run MIG cleanup directly on the host where nvidia-smi is available.
-# The BATS Docker container doesn't have nvidia-smi, so cleanup-from-previous-run.sh
-# uses the lambda/nvmm no-op stub. We handle MIG cleanup here instead.
+# The BATS Docker container uses the lambda nvmm stub (no-op).
+# We handle MIG cleanup here instead.
 # IMPORTANT: Skip on A100 — disabling MIG mode on cloud VM A100s can put the
 # GPU in an unrecoverable state (#883).
 case "${LAMBDA_GPU_TYPE}" in
@@ -117,7 +122,7 @@ export TEST_CHART_LOCAL=true
 export DISABLE_COMPUTE_DOMAINS=true
 export TEST_FILTER_TAGS='${FILTER}'
 export GIT_COMMIT_SHORT=${GIT_COMMIT_SHORT}
-# Use lambda nvmm stub (no GPU Operator). MIG cleanup is handled above on the host.
+# Use lambda nvmm stub (no GPU Operator). MIG cleanup handled above.
 export NVMM_PATH=/cwd/tests/bats/lib/lambda
 
 make -f tests/bats/Makefile tests-gpu-single GIT_COMMIT_SHORT=${GIT_COMMIT_SHORT}

tests/bats/Makefile

@@ -170,13 +170,14 @@ runner-image:
 # cmdline args).
 .PHONY: tests-gpu tests-gpu-single tests-cd
 
-# Single-GPU-safe subset. Suitable for CI environments with one GPU and no
-# GPU Operator (e.g., Lambda Cloud). Excludes test_basics.bats (expects GPU
-# Operator + clean state), MIG, stress, and upgrade tests.
+# Lambda CI subset. Excludes test_basics.bats (expects GPU Operator + clean
+# state), static MIG, stress, and upgrade tests. DynMIG tests are included
+# but tagged 'dynmig' — auto-skipped on non-MIG GPUs via TEST_FILTER_TAGS.
 tests-gpu-single: runner-image
 	$(call RUN_BATS, \
 		tests/bats/test_gpu_basic.bats \
 		tests/bats/test_gpu_cuda_workloads.bats \
+		tests/bats/test_gpu_dynmig.bats \
 		tests/bats/test_gpu_sharing.bats)
 
 # Run a subset covering mainly the GPU plugin

tests/bats/cleanup-from-previous-run.sh

@@ -104,7 +104,8 @@ set -e
 bash tests/bats/clean-state-dirs-all-nodes.sh
 
 # Remove any stray MIG devices and disable MIG mode on all nodes.
-nvmm all sh -c 'nvidia-smi mig -dci; nvidia-smi mig -dgi; nvidia-smi -mig 0'
+# Non-fatal: MIG may not be supported on all GPU types (V100, A10).
+nvmm all sh -c 'nvidia-smi mig -dci; nvidia-smi mig -dgi; nvidia-smi -mig 0' || echo "nvmm MIG cleanup skipped (non-fatal)"
 
 set +x
 echo "cleanup: done"

tests/bats/test_gpu_dynmig.bats

@@ -13,6 +13,9 @@ setup_file () {
   kubectl delete resourceslices.resource.k8s.io --all
 
   local _iargs=("--set" "logVerbosity=6" "--set" "featureGates.DynamicMIG=true")
+  if [ "${DISABLE_COMPUTE_DOMAINS:-}" = "true" ]; then
+    _iargs+=("--set" "resources.computeDomains.enabled=false")
+  fi
   iupgrade_wait "${TEST_CHART_REPO}" "${TEST_CHART_VERSION}" _iargs
   run kubectl logs \
     -l dra-driver-nvidia-gpu-component=kubelet-plugin \
@@ -57,7 +60,7 @@ confirm_mig_mode_disabled_all_nodes() {
 }
 
 
-# bats test_tags=fastfeedback
+# bats test_tags=fastfeedback,dynmig,version-specific
 @test "DynMIG: inspect device attributes in resource slice (gpu)" {
   local reference=(
     "architecture"
@@ -80,7 +83,7 @@ confirm_mig_mode_disabled_all_nodes() {
 }
 
 
-# bats test_tags=fastfeedback
+# bats test_tags=fastfeedback,dynmig,version-specific
 @test "DynMIG: inspect device attributes in resource slice (mig)" {
   local reference=(
     "architecture"
@@ -104,7 +107,7 @@ confirm_mig_mode_disabled_all_nodes() {
 }
 
 
-# bats test_tags=fastfeedback
+# bats test_tags=fastfeedback,dynmig
 @test "DynMIG: 1 pod, 1 MIG" {
   confirm_mig_mode_disabled_all_nodes
   kubectl apply -f tests/bats/specs/gpu-simple-mig.yaml
@@ -127,7 +130,7 @@ confirm_mig_mode_disabled_all_nodes() {
 }
 
 
-# bats test_tags=fastfeedback
+# bats test_tags=fastfeedback,dynmig
 @test "DynMIG: 1 pod, 2 containers (1 MIG each)" {
   confirm_mig_mode_disabled_all_nodes
 
@@ -159,9 +162,12 @@ confirm_mig_mode_disabled_all_nodes() {
 }
 
 
-# bats test_tags=fastfeedback
+# bats test_tags=fastfeedback,dynmig
 @test "DynMIG: 1 pod, 1 MIG + TimeSlicing config" {
   local _iargs=("--set" "logVerbosity=6" "--set" "featureGates.DynamicMIG=true" "--set" "featureGates.TimeSlicingSettings=true")
+  if [ "${DISABLE_COMPUTE_DOMAINS:-}" = "true" ]; then
+    _iargs+=("--set" "resources.computeDomains.enabled=false")
+  fi
   iupgrade_wait "${TEST_CHART_REPO}" "${TEST_CHART_VERSION}" _iargs
 
   confirm_mig_mode_disabled_all_nodes