How to run vllm on jetpack 6.0

[zhiweit]

Search before asking

• I have searched the jetson-containers issues and found no similar feature requests.

Question

I am looking for a way to run vllm via docker in jetson orin jetpack 6.0.

I have a Advantech Orin that has jetpack 6.0 built in but the current vllm containers from nvidia are only support for jetpack 6.2 above (r.36.4). Is there any other way to run vllm on jetpack 6.0 (r36.3)? I have changed the etc/apt/sources.list.d/nvidia-l4t-apt-source.list to point to the 36.4 repo and I have run

sudo apt update

sudo apt install nvidia-jetpack

sudo reboot

but i am unable to run the vllm docker container image from the Nvidia AI IOT ghcr here getting

vlm-1  | (EngineCore_DP0 pid=64) torch.AcceleratorError: CUDA error: device kernel image is invalid
vlm-1  | (EngineCore_DP0 pid=64) Search for `cudaErrorInvalidKernelImage' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
vlm-1  | (EngineCore_DP0 pid=64) CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
vlm-1  | (EngineCore_DP0 pid=64) For debugging consider passing CUDA_LAUNCH_BLOCKING=1
vlm-1  | (EngineCore_DP0 pid=64) Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
vlm-1  | (EngineCore_DP0 pid=64)

this is when i run jetson-release

Software part of jetson-stats 4.3.2 - (c) 2024, Raffaello Bonghi
Model: NVIDIA Jetson AGX Orin Developer Kit - Jetpack 6.0 [L4T 36.3.0]
NV Power Mode[0]: MAXN
Serial Number: [XXX Show with: jetson_release -s XXX]
Hardware:
 - Module: Check with sudo
Platform:
 - Distribution: Ubuntu 22.04 Jammy Jellyfish
 - Release: 5.15.136-tegra
jtop:
 - Version: 4.3.2
 - Service: Active
Libraries:
 - CUDA: 12.6.68
 - cuDNN: 8.9.4.25
 - TensorRT: 10.3.0.30
 - VPI: 3.2.4
 - OpenCV: 4.8.0 - with CUDA: NO

How can I run vllm on jetpack 6.0?

Additional

No response