Skip to content

troubleshooting.md

Latest commit

 

History

History
388 lines (281 loc) · 7.68 KB

troubleshooting.md

File metadata and controls

388 lines (281 loc) · 7.68 KB
Raw

Troubleshooting Guide

Common issues and solutions for Mock NVML.

Build Issues

CGo Not Enabled

Error:

cgo: C compiler "gcc" not found: exec: "gcc": executable file not found in $PATH

Solution:

# Ubuntu/Debian
sudo apt-get install build-essential

# RHEL/CentOS
sudo yum groupinstall "Development Tools"

# macOS
xcode-select --install

Go Version Too Old

Error:

go: go.mod file indicates go 1.25

Solution:

# Install Go 1.25+
wget https://go.dev/dl/go1.25.0.linux-amd64.tar.gz
sudo tar -C /usr/local -xzf go1.25.0.linux-amd64.tar.gz
export PATH=$PATH:/usr/local/go/bin

Build on macOS

Problem: Cannot build shared library on macOS for Linux.

Solution: Use Docker build:

make docker-build

Runtime Issues

Library Not Found

Error:

nvidia-smi: error while loading shared libraries: libnvidia-ml.so.1: cannot open shared object file

Solution:

# Verify library exists
ls -la pkg/gpu/mocknvml/libnvidia-ml.so*

# Set library path correctly
export LD_LIBRARY_PATH=$(pwd)/pkg/gpu/mocknvml:$LD_LIBRARY_PATH

# Or use absolute path
LD_LIBRARY_PATH=/full/path/to/pkg/gpu/mocknvml nvidia-smi

Symbol Not Found

Error:

nvidia-smi: symbol lookup error: nvidia-smi: undefined symbol: nvmlSomeFunction

Solution:

  1. Check if symbol is exported:
nm -D pkg/gpu/mocknvml/libnvidia-ml.so | grep nvmlSomeFunction
  1. If missing, check if it's a stub or needs implementation:
# Check if it's in stubs
grep "nvmlSomeFunction" pkg/gpu/mocknvml/bridge/stubs_generated.go

# If stub, implement it in appropriate bridge file, then:
go generate ./pkg/gpu/mocknvml/bridge/
make -C pkg/gpu/mocknvml clean
make -C pkg/gpu/mocknvml

Segmentation Fault

Error:

Segmentation fault (core dumped)

Causes and Solutions:

  1. Handle dereferencing issue:

    • Enable debug mode: MOCK_NVML_DEBUG=1
    • Check which function crashes
    • Verify handle table is working

  2. CGo memory issue:

    • Run with address sanitizer:
    CGO_CFLAGS="-fsanitize=address" go build -buildmode=c-shared ...

  3. nvidia-smi version mismatch:

    • Ensure driver version in config matches nvidia-smi expectation
    • Try different nvidia-smi versions

YAML Config Not Loading

Error:

[CONFIG] Failed to load YAML config from /path/to/config.yaml: ...

Solution:

  1. Verify file exists and is readable:
cat $MOCK_NVML_CONFIG
  1. Validate YAML syntax:
python3 -c "import yaml; yaml.safe_load(open('$MOCK_NVML_CONFIG'))"
  1. Check required fields:
version: "1.0"                    # Required
system:
  driver_version: "550.163.01"    # Required
  1. Enable debug to see specific error:
MOCK_NVML_DEBUG=1 LD_LIBRARY_PATH=. nvidia-smi

nvidia-smi Shows Wrong Values

Problem: Values don't match YAML config.

Solution:

  1. Verify config is loaded:
MOCK_NVML_DEBUG=1 LD_LIBRARY_PATH=. nvidia-smi 2>&1 | head -5
# Should show: [CONFIG] Loaded YAML config: N devices, driver X.Y.Z

  1. Check device override vs defaults:

    • Per-device settings override defaults
    • Check devices: section in YAML

  2. Verify field mapping:

NOT_SUPPORTED Errors

Problem: nvidia-smi shows "N/A" or errors for some queries.

Explanation: Functions return NOT_SUPPORTED when:

  • YAML config doesn't provide the value
  • Function is not implemented (stub)

Solution:

  1. Add missing config values:
device_defaults:
  thermal:
    temperature_gpu_c: 33    # Add this for temperature
  1. Check if function is implemented:
# Check engine implementation
grep "GetXxx" pkg/gpu/mocknvml/engine/device.go

# Check bridge implementation
grep "nvmlDeviceGetXxx" pkg/gpu/mocknvml/bridge/*.go

# Check if it's just a stub
grep "nvmlDeviceGetXxx" pkg/gpu/mocknvml/bridge/stubs_generated.go

Testing Issues

Integration Test Fails

Error:

Error: No GPU found

Solution:

  1. Ensure mock library is built:
make -C pkg/gpu/mocknvml
  1. Check library is accessible in Docker:
docker run -v $(pwd)/pkg/gpu/mocknvml:/mock -e LD_LIBRARY_PATH=/mock ...

Race Conditions

Error:

WARNING: DATA RACE

Solution:

  • All Engine methods should use mutex
  • Check that new code acquires locks properly
  • Run tests with race detector: go test -race ./...

Performance Issues

Slow Startup

Problem: nvidia-smi takes long to start.

Causes:

  1. Large YAML config parsing
  2. Debug logging enabled

Solution:

# Disable debug
unset MOCK_NVML_DEBUG

# Use simpler config for testing

Memory Usage

Problem: High memory usage.

Explanation:

  • Each handle allocates ~40 bytes of C memory
  • Error string cache grows with unique errors

This is typically not a problem for normal usage.

Environment Issues

LD_LIBRARY_PATH Not Working

Problem: System uses real NVML despite setting LD_LIBRARY_PATH.

Solutions:

  1. Check library search order:
LD_DEBUG=libs nvidia-smi 2>&1 | grep nvml
  1. Verify no cached libraries:
sudo ldconfig  # Refresh cache
  1. Use absolute path:
LD_LIBRARY_PATH=/absolute/path/to/mocknvml nvidia-smi
  1. Check for ld.so.conf entries:
grep nvml /etc/ld.so.conf.d/*

Docker Issues

Problem: Library doesn't work in Docker container.

Solutions:

  1. Mount library correctly:
docker run -v $(pwd)/pkg/gpu/mocknvml:/mock:ro \
           -e LD_LIBRARY_PATH=/mock \
           ubuntu nvidia-smi
  1. Ensure nvidia-smi is available:
docker run ... which nvidia-smi
  1. Use correct architecture:
# Build for container architecture
GOARCH=amd64 make -C pkg/gpu/mocknvml

GPU Operator Issues

Validator Pod in CrashLoopBackOff

Problem: The GPU Operator validator pod repeatedly crashes.

Explanation: The GPU Operator validator (/usr/bin/nvidia-validator) is a statically-linked Go binary with no shell. It probes the driver root for expected files.

Solution: Ensure the mock driver root has all required files:

# Use the specific Kind cluster and node name (adjust cluster name as needed)
NODE_CONTAINER="nvml-mock-operator-control-plane"
docker exec "$NODE_CONTAINER" ls -la /run/nvidia/driver/usr/lib64/libnvidia-ml.so*
docker exec "$NODE_CONTAINER" cat /var/lib/nvml-mock/driver/config/config.yaml

CDI Spec Not Generated

Problem: CDI specs are not appearing in /var/run/cdi/.

Solution: Check nvml-mock DaemonSet logs for CDI generation errors:

kubectl logs -l app.kubernetes.io/name=nvml-mock | grep -i cdi

Device Plugin Shows 0 GPUs with GPU Operator

Problem: The GPU Operator-managed device plugin reports 0 GPUs.

Solution: Verify the device plugin is using the mock driver root:

kubectl -n gpu-operator logs -l app=nvidia-device-plugin-daemonset | head -20

The GPU Operator must be installed with --set driver.enabled=false and --set toolkit.enabled=false when using mock GPUs, since there is no real NVIDIA driver to manage.

Getting Help

Debug Information to Collect

When reporting issues, include:

  1. Environment:
uname -a
go version
gcc --version
  1. Debug output:
MOCK_NVML_DEBUG=1 LD_LIBRARY_PATH=. nvidia-smi 2>&1
  1. Library info:
file pkg/gpu/mocknvml/libnvidia-ml.so
ldd pkg/gpu/mocknvml/libnvidia-ml.so

  1. Config file (if using YAML)

  2. Expected vs actual output

Filing Issues

Include:

  • Steps to reproduce
  • Expected behavior
  • Actual behavior
  • Debug output
  • Environment info