Can not open the tool server

[wkfdb]

For simply, running the following code, I always get the following error:

host=$(hostname -i | awk '{print $1}')
port=$(shuf -i 30000-31000 -n 1)
tool_server_url=http://$host:$port/get_observation
server_pid=$!
echo "Server (pid=$server_pid) started at $tool_server_url"
python -m verl_tool.servers.serve --host $host --port $port --tool_type "pixel_reasoner" --workers_per_tool 4

Server (pid=) started at http://22.13.220.123:30142/get_observation
2026-01-22 22:55:32,123 - __main__ - WARNING - Adjusted uvi_workers to 4 to handle max_concurrent_requests=1024
2026-01-22 22:55:32,123 - __main__ - WARNING - Adjusted router_workers to 8 to handle max_concurrent_requests=1024
2026-01-22 22:55:32,180 - __main__ - INFO - [MAIN] Using uvloop event loop for router
2026-01-22 22:55:32,181 - __main__ - INFO - [MAIN] Starting 4 backend worker subprocesses...
2026-01-22 22:55:32,181 - __main__ - INFO - [WORKER_MANAGER] Starting 4 workers with max_concurrent_requests=1024 each
2026-01-22 22:55:32,182 - __main__ - INFO - [BACKEND 0] Starting with command: /mnt/videodata_nas/group/envs/verl-tool/bin/python -m verl_tool.servers.tool_server --tool_type pixel_reasoner --host 127.0.0.1 --port 42067 --workers_per_tool 1 --max_concurrent_requests 1024 --log_level info --log_interval 30
2026-01-22 22:55:32,182 - __main__ - INFO - [BACKEND 0] Spawned PID=2448 on 127.0.0.1:42067
2026-01-22 22:55:32,183 - __main__ - INFO - [BACKEND 1] Starting with command: /mnt/videodata_nas/group/envs/verl-tool/bin/python -m verl_tool.servers.tool_server --tool_type pixel_reasoner --host 127.0.0.1 --port 57243 --workers_per_tool 1 --max_concurrent_requests 1024 --log_level info --log_interval 30
2026-01-22 22:55:32,183 - __main__ - INFO - [BACKEND 1] Spawned PID=2449 on 127.0.0.1:57243
2026-01-22 22:55:32,184 - __main__ - INFO - [BACKEND 2] Starting with command: /mnt/videodata_nas/group/envs/verl-tool/bin/python -m verl_tool.servers.tool_server --tool_type pixel_reasoner --host 127.0.0.1 --port 60589 --workers_per_tool 1 --max_concurrent_requests 1024 --log_level info --log_interval 30
2026-01-22 22:55:32,184 - __main__ - INFO - [BACKEND 2] Spawned PID=2450 on 127.0.0.1:60589
2026-01-22 22:55:32,185 - __main__ - INFO - [BACKEND 3] Starting with command: /mnt/videodata_nas/group/envs/verl-tool/bin/python -m verl_tool.servers.tool_server --tool_type pixel_reasoner --host 127.0.0.1 --port 35765 --workers_per_tool 1 --max_concurrent_requests 1024 --log_level info --log_interval 30
2026-01-22 22:55:32,185 - __main__ - INFO - [BACKEND 3] Spawned PID=2451 on 127.0.0.1:35765
2026-01-22 22:55:32,185 - __main__ - INFO - [MAIN] Backend workers started:
2026-01-22 22:55:32,185 - __main__ - INFO -   backend[0]: http://127.0.0.1:42067
2026-01-22 22:55:32,185 - __main__ - INFO -   backend[1]: http://127.0.0.1:57243
2026-01-22 22:55:32,185 - __main__ - INFO -   backend[2]: http://127.0.0.1:60589
2026-01-22 22:55:32,185 - __main__ - INFO -   backend[3]: http://127.0.0.1:35765
2026-01-22 22:55:32,185 - __main__ - INFO - [HEALTH] Waiting for 4 workers to become healthy...
2026-01-22 22:56:32,550 - __main__ - ERROR - [HEALTH] 4 workers not healthy after 60.0s: http://127.0.0.1:60589, http://127.0.0.1:35765, http://127.0.0.1:57243, http://127.0.0.1:42067
2026-01-22 22:56:32,551 - __main__ - ERROR - [MAIN] Some workers failed to start, aborting...
2026-01-22 22:56:32,551 - __main__ - INFO - [WORKER_MANAGER] Shutting down 4 workers...
2026-01-22 22:56:32,568 - __main__ - INFO - [WORKER_MANAGER] Worker PID=2448 terminated gracefully
2026-01-22 22:56:32,568 - __main__ - INFO - [WORKER_MANAGER] Worker PID=2449 terminated gracefully
2026-01-22 22:56:32,568 - __main__ - INFO - [WORKER_MANAGER] Worker PID=2450 terminated gracefully
2026-01-22 22:56:32,568 - __main__ - INFO - [WORKER_MANAGER] Worker PID=2451 terminated gracefully
2026-01-22 22:56:32,568 - __main__ - INFO - [MAIN] Shutdown complete

Any ideas about this error? I even cant start the basic tool-server.

[chendy25]

Maybe you can show the version of your ray library.

[wkfdb]

Still can not open the tool server.....

[chendy25]

Ok，I think you should be fine if you downgrade ray to version 2.48.0.

[wkfdb]

By trying so many times I successfully started the server, but the success rate is really low. This doesn't seem to be a problem with the ray version. What does it mean of "waiting for 4 workers to become healthy"? I'm running a program on a NAS-based server. I suspect the child processes are starting too slowly; 60 seconds isn't enough time for them to complete. As a developer, do you have any idea why the server might be having trouble starting?

[chendy25]

Your problem might be slightly different from what I've encountered before. However, I can see that if your startup time exceeds 60 seconds, it will shut down. The successful attempt took 51 seconds, so there's a certain success rate, albeit very low. I think the slow startup of the child process might be due to I/O latency on the NAS storage, exceeding the 60-second health check threshold set by the main process. Perhaps you could try changing the configuration parameters in /verl-tool/verl_tool/servers/serve.py, for example, increasing HEALTH_CHECK_TIMEOUT to 120 seconds or trying other values.

[wkfdb]

I also got another problem. By running the train_qwen25_vl.sh，it seems that the CUDA memory cannot be properly released. By debugging the code I got an error which means that I have run out of CUDA memory. However, by ending the training script can not release the memory. The CUDA memory is being occupied by zombie processes and cannot be released. Do you even seen this problem?

[chendy25]

Using ray stop should solve the problem.