Llm large reference

[pgmpablo157321]

No description provided.

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

[nvzhihanj]

This torch version doesn't exist

[nvzhihanj]

WARNING 11-28 01:43:11 multiproc_gpu_executor.py:53] Reducing Torch parallelism from 112 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
INFO 11-28 01:43:11 custom_cache_manager.py:17] Setting Triton cache manager to: vllm.triton_utils.custom_cache_manager:CustomCacheManager
(VllmWorkerProcess pid=899) INFO 11-28 01:43:11 multiproc_worker_utils.py:216] Worker ready; awaiting tasks
(VllmWorkerProcess pid=899) ERROR 11-28 01:43:11 multiproc_worker_utils.py:231] Exception in worker VllmWorkerProcess while processing method init_device: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method, Traceback (most recent call last):
(VllmWorkerProcess pid=899) ERROR 11-28 01:43:11 multiproc_worker_utils.py:231]   File "/home/zhihanj/.local/lib/python3.10/site-packages/vllm/executor/multiproc_worker_utils.py", line 224, in _run_worker_process
(VllmWorkerProcess pid=899) ERROR 11-28 01:43:11 multiproc_worker_utils.py:231]     output = executor(*args, **kwargs)
(VllmWorkerProcess pid=899) ERROR 11-28 01:43:11 multiproc_worker_utils.py:231]   File "/home/zhihanj/.local/lib/python3.10/site-packages/vllm/worker/worker.py", line 166, in init_device
(VllmWorkerProcess pid=899) ERROR 11-28 01:43:11 multiproc_worker_utils.py:231]     torch.cuda.set_device(self.device)
(VllmWorkerProcess pid=899) ERROR 11-28 01:43:11 multiproc_worker_utils.py:231]   File "/home/zhihanj/.local/lib/python3.10/site-packages/torch/cuda/__init__.py", line 420, in set_device
(VllmWorkerProcess pid=899) ERROR 11-28 01:43:11 multiproc_worker_utils.py:231]     torch._C._cuda_setDevice(device)
(VllmWorkerProcess pid=899) ERROR 11-28 01:43:11 multiproc_worker_utils.py:231]   File "/home/zhihanj/.local/lib/python3.10/site-packages/torch/cuda/__init__.py", line 300, in _lazy_init
(VllmWorkerProcess pid=899) ERROR 11-28 01:43:11 multiproc_worker_utils.py:231]     raise RuntimeError(
(VllmWorkerProcess pid=899) ERROR 11-28 01:43:11 multiproc_worker_utils.py:231] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
(VllmWorkerProcess pid=899) ERROR 11-28 01:43:11 multiproc_worker_utils.py:231] 

[nvzhihanj]

Please add the commit as of today (there has been no update since 9/25) so we are fixed on one version of the checkpoint