nvdia-cuda-mps-server consistently hangs at the "creating worker thread" log

[yangcheng-dev]

I am using nvidia-cuda-mps-server for GPU virtualization (GPU is V100), and the plugin comes from Nebuly-NVIDIA. The CUDA client is k8s.gcr.io/cuda-vector-add:v0.1. After the CUDA client starts as a container, the nvidia-cuda-mps-server process consistently hangs at the "creating worker thread" log, and the client does not print any logs. Where could the problem be? Is it possible that my GPU card does not support MPS, or is it an issue with the client?

Steps to reproduce the issue
1.install Nebuly-NVIDIA plugin：https://github.com/nebuly-ai/k8s-device-plugin
2.start a pod which image is “k8s.gcr.io/cuda-vector-add:v0.1”
3. client does not print any logs,and nvidia-cuda-mps-server hangs at the "creating worker thread" log