2025-04-24 18:57:03,337 ERR cli.py:71 -- ^[[31m---------------------------------------^[[39m
2025-04-24 18:57:03,337 ERR cli.py:72 -- ^[[31mJob 'raysubmit_LKFSP4hfiq2xPuz6' failed^[[39m
2025-04-24 18:57:03,337 ERR cli.py:73 -- ^[[31m---------------------------------------^[[39m
2025-04-24 18:57:03,337 INFO cli.py:86 -- Status message: Job entrypoint command failed with exit code 1, last available logs (truncated to 20,000 chars):
95412 14.05 ray::LLMRayActor
95410 13.86 ray::LLMRayActor
95414 13.85 ray::LLMRayActor
95411 13.61 ray::LLMRayActor
95415 13.49 ray::LLMRayActor
95409 13.43 ray::LLMRayActor
Refer to the documentation on how to address the out of memory issue: https://docs.ray.io/en/latest/ray-core/scheduling/ray-oom-prevention.html. Consider provisioning more memory on this node or reducing task parallelism by requesting more CPUs per task. Set max_restarts and max_task_retries to enable retry when the task crashes due to OOM. To adjust the kill threshold, set the environment variable `RAY_memory_usage_threshold` when starting Ray. To disable worker killing, set the environment variable `RAY_memory_monitor_refresh_ms` to zero.
^[[36m(LLMRayActor pid=95409)^[[0m INFO 04-24 18:22:16 [block_pool.py:264] Successfully reset prefix cache^[[32m [repeated 7x across cluster]^[[0m
^[[36m(LLMRayActor pid=95409)^[[0m INFO 04-24 18:22:18 [gpu_worker.py:81] Sleep mode freed 35.71 GiB memory, 12.82 GiB memory is still in use.^[[32m [repeated 7x across cluster]^[[0m
^[[36m(LLMRayActor pid=95409)^[[0m INFO 04-24 18:22:18 [executor_base.py:210] It took 1.396755 seconds to fall asleep.^[[32m [repeated 7x across cluster]^[[0m
最新版本的代码跑 Qwen2.5VL的RL训练，CPU 内存 OOM， 之前的分支 v0.7.0a 就没有这个问题。
另外：train_fre_text.sh 脚本训练出来的模型，train_mgt_percereas.sh 接着训练，vllm好像inference会有问题。
(LLMRayActor pid=52671) RuntimeError: Expected there to be 1 prompt updates corresponding to 1 image items, but instead found 0 prompt updates! Either the prompt text has missing/incorrect tokens for multi-modal inputs, or there is a problem with your implementation of merged multi-modal processor for this model (usually arising from an inconsistency between `_call_hf_processor` and `_get_prompt_updates`).
最新版本的代码跑 Qwen2.5VL的RL训练，CPU 内存 OOM， 之前的分支 v0.7.0a 就没有这个问题。
另外：train_fre_text.sh 脚本训练出来的模型，train_mgt_percereas.sh 接着训练，vllm好像inference会有问题。