Runtime Error

[2437218051]

/usr/local/IMOD/pythonLink/python /home/ethan/TxAgent-main/run_example.py
INFO 03-31 16:54:26 [init.py:239] Automatically detected platform cuda.
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 4.64it/s]
model_name: mims-harvard/TxAgent-T1-Llama-3.1-8B
tokenizer: None
terminators: None
rag_model_name: mims-harvard/ToolRAG-T1-GTE-Qwen2-1.5B
tool_files_dict: None
model: None
rag_model: <txagent.toolrag.ToolRAGModel object at 0x7f4891eafe50>
tooluniverse: None
prompt_multi_step: You are a helpful assistant that will solve problems through detailed, step-by-step reasoning and actions based on your reasoning. Typically, your actions will use the provided functions. You have access to the following functions.
self_prompt: Strictly follow the instruction.
chat_prompt: You are helpful assistant to chat with the user.
enable_finish: True
enable_rag: True
enable_summary: False
summary_mode: step
summary_skip_last_k: 0
summary_context_length: None
init_rag_num: 0
step_rag_num: 10
force_finish: True
avoid_repeat: True
seed: None
enable_checker: False
additional_default_tools: None
INFO 03-31 16:54:36 [config.py:2610] Downcasting torch.float32 to torch.float16.
INFO 03-31 16:54:43 [config.py:585] This model supports multiple tasks: {'classify', 'score', 'generate', 'reward', 'embed'}. Defaulting to 'generate'.
INFO 03-31 16:54:43 [config.py:1697] Chunked prefill is enabled with max_num_batched_tokens=8192.
WARNING 03-31 16:54:43 [config.py:1193] Possibly too large swap space. 4.00 GiB out of the 7.60 GiB total CPU memory is allocated for the swap space.
WARNING 03-31 16:54:44 [utils.py:2181] We must use the spawn multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/getting_started/troubleshooting.html#python-multiprocessing for more information. Reason: CUDA is initialized
INFO 03-31 16:54:48 [init.py:239] Automatically detected platform cuda.
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 4.25it/s]
model_name: mims-harvard/TxAgent-T1-Llama-3.1-8B
tokenizer: None
terminators: None
rag_model_name: mims-harvard/ToolRAG-T1-GTE-Qwen2-1.5B
tool_files_dict: None
model: None
rag_model: <txagent.toolrag.ToolRAGModel object at 0x7f1e5f066290>
tooluniverse: None
prompt_multi_step: You are a helpful assistant that will solve problems through detailed, step-by-step reasoning and actions based on your reasoning. Typically, your actions will use the provided functions. You have access to the following functions.
self_prompt: Strictly follow the instruction.
chat_prompt: You are helpful assistant to chat with the user.
enable_finish: True
enable_rag: True
enable_summary: False
summary_mode: step
summary_skip_last_k: 0
summary_context_length: None
init_rag_num: 0
step_rag_num: 10
force_finish: True
avoid_repeat: True
seed: None
enable_checker: False
additional_default_tools: None
INFO 03-31 16:55:32 [config.py:2610] Downcasting torch.float32 to torch.float16.
INFO 03-31 16:55:42 [config.py:585] This model supports multiple tasks: {'generate', 'reward', 'score', 'classify', 'embed'}. Defaulting to 'generate'.
INFO 03-31 16:55:43 [config.py:1697] Chunked prefill is enabled with max_num_batched_tokens=8192.
WARNING 03-31 16:55:43 [config.py:1193] Possibly too large swap space. 4.00 GiB out of the 7.60 GiB total CPU memory is allocated for the swap space.
Traceback (most recent call last):
File "", line 1, in
File "/usr/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "/usr/lib/python3.10/multiprocessing/spawn.py", line 125, in _main
prepare(preparation_data)
File "/usr/lib/python3.10/multiprocessing/spawn.py", line 236, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/usr/lib/python3.10/multiprocessing/spawn.py", line 287, in _fixup_main_from_path
main_content = runpy.run_path(main_path,
File "/usr/lib/python3.10/runpy.py", line 289, in run_path
return _run_module_code(code, init_globals, run_name,
File "/usr/lib/python3.10/runpy.py", line 96, in _run_module_code
_run_code(code, mod_globals, init_globals,
File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/home/ethan/TxAgent-main/run_example.py", line 15, in
agent.init_model()
File "/home/ethan/.local/lib/python3.10/site-packages/txagent/txagent.py", line 65, in init_model
self.load_models()
File "/home/ethan/.local/lib/python3.10/site-packages/txagent/txagent.py", line 79, in load_models
self.model = LLM(model=self.model_name)
File "/home/ethan/.local/lib/python3.10/site-packages/vllm/utils.py", line 1037, in inner
return fn(*args, **kwargs)
File "/home/ethan/.local/lib/python3.10/site-packages/vllm/entrypoints/llm.py", line 243, in init
self.llm_engine = LLMEngine.from_engine_args(
File "/home/ethan/.local/lib/python3.10/site-packages/vllm/v1/engine/llm_engine.py", line 139, in from_engine_args
return cls(vllm_config=vllm_config,
File "/home/ethan/.local/lib/python3.10/site-packages/vllm/v1/engine/llm_engine.py", line 88, in init
self.engine_core = EngineCoreClient.make_client(
File "/home/ethan/.local/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 66, in make_client
return SyncMPClient(vllm_config, executor_class, log_stats)
File "/home/ethan/.local/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 337, in init
super().init(
File "/home/ethan/.local/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 301, in init
self.resources.proc_handle = BackgroundProcHandle(
File "/home/ethan/.local/lib/python3.10/site-packages/vllm/v1/utils.py", line 121, in init
self.proc.start()
File "/usr/lib/python3.10/multiprocessing/process.py", line 121, in start
self._popen = self._Popen(self)
File "/usr/lib/python3.10/multiprocessing/context.py", line 288, in _Popen
return Popen(process_obj)
File "/usr/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 32, in init
super().init(process_obj)
File "/usr/lib/python3.10/multiprocessing/popen_fork.py", line 19, in init
self._launch(process_obj)
File "/usr/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 42, in _launch
prep_data = spawn.get_preparation_data(process_obj._name)
File "/usr/lib/python3.10/multiprocessing/spawn.py", line 154, in get_preparation_data
_check_not_importing_main()
File "/usr/lib/python3.10/multiprocessing/spawn.py", line 134, in _check_not_importing_main
raise RuntimeError('''
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.

    This probably means that you are not using fork to start your
    child processes and you have forgotten to use the proper idiom
    in the main module:

        if __name__ == '__main__':
            freeze_support()
            ...

    The "freeze_support()" line can be omitted if the program
    is not going to be frozen to produce an executable.

please help me, i can't understand why this happened

[3H-Gene]

The Same problem

[gasvn]

Can you provide your env info? GPU, vllm version. It seems an issue related to vllm.

[2437218051]

Can you provide your env info? GPU, vllm version. It seems an issue related to vllm.

thank you for your reply, i solved this problem, i put the all code in the" if name == 'main': " ,it worked.
but now i meet anthoer problem,
↓↓↓↓
ERROR 03-31 22:15:01 [core.py:343] torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 1002.00 MiB. GPU 0 has a total capacity of 8.00 GiB of which 0 bytes is free. Of the allocated memory 14.12 GiB is allocated by PyTorch, and 1.28 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
ERROR 03-31 22:15:01 [core.py:343]
CRITICAL 03-31 22:15:01 [core_client.py:269] Got fatal signal from worker processes, shutting down. See stack trace above for root cause issue.
↑↑↑↑

i think it is the memory problem， and i tried to changed the ""PYTORCH_CUDA_ALLOC_CONF" param

↓↓↓↓↓↓
os.environ["PYTORCH_CUDA_ALLOC_CONF"] = "max_split_size_mb:32"
↑↑↑↑

but it don't work

[2437218051]

Can you provide your env info? GPU, vllm version. It seems an issue related to vllm.

vllm 0.8.2
txagent 0.1.2
here are some version

[2437218051]

The Same problem

https://docs.vllm.ai/en/latest/getting_started/troubleshooting.html#python-multiprocessing
here is the solusion
put the run code under the "if name == 'main"

[gasvn]

Thank you! I will update the code for this.

os.environ["PYTORCH_CUDA_ALLOC_CONF"] = "max_split_size_mb:32"
For this problem, I was using a GPU with 80GB of memory. The model size is 8B, so a GPU with 8GB of memory might not be sufficient. You can consider using multiple GPUs or finding a way to store the model on the CPU.

[2437218051]

Thank you! I will update the code for this.

os.environ["PYTORCH_CUDA_ALLOC_CONF"] = "max_split_size_mb:32" For this problem, I was using a GPU with 80GB of memory. The model size is 8B, so a GPU with 8GB of memory might not be sufficient. You can consider using multiple GPUs or finding a way to store the model on the CPU.

i see, thank you so much

[3H-Gene]

I used H20 and successfully finished the test task(run_example).

1. if name == 'main"
   2.os.environ["VLLM_USE_V1"] = "0"

[2437218051]

I used H20 and successfully finished the test task(run_example).

1. if name == 'main"
   2.os.environ["VLLM_USE_V1"] = "0"

thanks,i tried,but it didn't work.I am pretty sure it can't work with my 8GB GPU

[sarveswara-rao]

Hi, while the run_example.py I got the following error,

ValueError: vLLM V1 does not support per request user provided logits processors.

versions:
vLLM = 0.8.4
txagent = 0.1.3
cuda = 12.4

I have two RTX 6000 ada GPUs with 48GB VRAM.

if i use os.environ['VLLM_USE_V1'] = '0', then its giving cuda out of memory error.

if i comment it (vllm-project/vllm#15636), then i am getting the above error.