vllm container crashes on standup

[shashwatj07]

Component

I don't know

Describe the bug

Error upon launching vllm inside containers:

kubectl logs <prefill/decode_pod_name>

(EngineCore_DP0 pid=316) Process EngineCore_DP0:                                                                                       
(EngineCore_DP0 pid=316) ERROR 02-24 00:11:22 [core.py:936] EngineCore failed to start.                                                
(EngineCore_DP0 pid=316) ERROR 02-24 00:11:22 [core.py:936] Traceback (most recent call last):                                         
(EngineCore_DP0 pid=316) ERROR 02-24 00:11:22 [core.py:936]   File "/opt/vllm-source/vllm/v1/engine/core.py", line 927, in run_engine_c
ore                                                                                                                                    
(EngineCore_DP0 pid=316) ERROR 02-24 00:11:22 [core.py:936]     engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)    
(EngineCore_DP0 pid=316) ERROR 02-24 00:11:22 [core.py:936]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^    
(EngineCore_DP0 pid=316) ERROR 02-24 00:11:22 [core.py:936]   File "/opt/vllm-source/vllm/v1/engine/core.py", line 692, in __init__    
(EngineCore_DP0 pid=316) ERROR 02-24 00:11:22 [core.py:936]     super().__init__(                                                      
(EngineCore_DP0 pid=316) ERROR 02-24 00:11:22 [core.py:936]   File "/opt/vllm-source/vllm/v1/engine/core.py", line 106, in __init__    
(EngineCore_DP0 pid=316) ERROR 02-24 00:11:22 [core.py:936]     self.model_executor = executor_class(vllm_config)                      
(EngineCore_DP0 pid=316) ERROR 02-24 00:11:22 [core.py:936]                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^                      
(EngineCore_DP0 pid=316) ERROR 02-24 00:11:22 [core.py:936]   File "/opt/vllm-source/vllm/v1/executor/abstract.py", line 101, in __init
__
(EngineCore_DP0 pid=316) ERROR 02-24 00:11:22 [core.py:936]     self._init_executor()
(EngineCore_DP0 pid=316) ERROR 02-24 00:11:22 [core.py:936]   File "/opt/vllm-source/vllm/v1/executor/uniproc_executor.py", line 47, in
 _init_executor
(EngineCore_DP0 pid=316) ERROR 02-24 00:11:22 [core.py:936]     self.driver_worker.init_device()
(EngineCore_DP0 pid=316) ERROR 02-24 00:11:22 [core.py:936]   File "/opt/vllm-source/vllm/v1/worker/worker_base.py", line 326, in init_
device
(EngineCore_DP0 pid=316) ERROR 02-24 00:11:22 [core.py:936]     self.worker.init_device()  # type: ignore
(EngineCore_DP0 pid=316) ERROR 02-24 00:11:22 [core.py:936]     ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=316) ERROR 02-24 00:11:22 [core.py:936]   File "/opt/vllm-source/vllm/v1/worker/gpu_worker.py", line 209, in init_d
evice
(EngineCore_DP0 pid=316) ERROR 02-24 00:11:22 [core.py:936]     current_platform.set_device(self.device)
(EngineCore_DP0 pid=316) ERROR 02-24 00:11:22 [core.py:936]   File "/opt/vllm-source/vllm/platforms/cuda.py", line 123, in set_device
(EngineCore_DP0 pid=316) ERROR 02-24 00:11:22 [core.py:936]     torch.cuda.set_device(device)
(EngineCore_DP0 pid=316) ERROR 02-24 00:11:22 [core.py:936]   File "/opt/vllm/lib64/python3.12/site-packages/torch/cuda/__init__.py", l
ine 567, in set_device
(EngineCore_DP0 pid=316) ERROR 02-24 00:11:22 [core.py:936]     torch._C._cuda_setDevice(device)
(EngineCore_DP0 pid=316) ERROR 02-24 00:11:22 [core.py:936]   File "/opt/vllm/lib64/python3.12/site-packages/torch/cuda/__init__.py", l
ine 410, in _lazy_init
(EngineCore_DP0 pid=316) ERROR 02-24 00:11:22 [core.py:936]     torch._C._cuda_init()
(EngineCore_DP0 pid=316) ERROR 02-24 00:11:22 [core.py:936] RuntimeError: Unexpected error from cudaGetDeviceCount(). Did you run some 
cuda functions before calling NumCudaDevices() that might have already set an error? Error 803: system has unsupported display driver /
 cuda driver combination

Steps to reproduce

./scripts/standup.sh -c gpu

Additional context or screenshots

No response

[maugustosilva]

Please try to follow the example from https://github.com/llm-d/llm-d-benchmark/blob/main/scenarios/cicd/gke_H100_fb.sh

export _LD_LIBRARY_PATH="\${LD_LIBRARY_PATH}:/usr/local/nvidia/lib64"

export LLMDBENCH_VLLM_COMMON_ENVVARS_TO_YAML=_LD_LIBRARY_PATH

(please note that I have an underscore as a prefix, _LD_LIBRARY_PATH, so as to not clash with the locally define LD_LIBRARY_PATH)

[shashwatj07]

@maugustosilva Thanks for your note. I tried it but ended up receiving another error inside the pod.

Defaulted container "vllm" out of: vllm, routing-proxy (init)
Traceback (most recent call last):
  File "/opt/vllm/bin/vllm", line 4, in <module>
    from vllm.entrypoints.cli.main import main
  File "/opt/vllm-source/vllm/__init__.py", line 14, in <module>
    import vllm.env_override  # noqa: F401
    ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/vllm-source/vllm/env_override.py", line 5, in <module>
    import torch
  File "/opt/vllm/lib64/python3.12/site-packages/torch/__init__.py", line 427, in <module>
    from torch._C import *  # noqa: F403
    ^^^^^^^^^^^^^^^^^^^^^^
ImportError: libnvshmem_host.so.3: cannot open shared object file: No such file or directory

I'm able to successfully run https://docs.vllm.ai/en/stable/deployment/k8s/#deployment-with-gpus after passing LD_LIBRARY_PATH in the deployment yaml. Earlier, I was seeing the same cuda error in this toy example too.