已经训了16个step，突然出现：
forward: 91%|█████████ | 29/32 [00:23<00:01, 1.64it/s]
�[36m(ActorModelRayActor pid=396186)�[0m
forward: 94%|█████████▍| 30/32 [00:23<00:01, 1.64it/s]
�[36m(ActorModelRayActor pid=396186)�[0m
forward: 97%|█████████▋| 31/32 [00:24<00:00, 1.65it/s]
�[36m(ActorModelRayActor pid=396186)�[0m
forward: 100%|██████████| 32/32 [00:24<00:00, 1.65it/s]
forward: 100%|██████████| 32/32 [00:24<00:00, 1.29it/s]
�[36m(PPOTrainer pid=396792)�[0m INFO 06-10 23:24:11 experience_maker.py:446] ✨ Experience making completed in 0:00:29
�[36m(LLMRayActor pid=387182)�[0m INFO 06-10 23:23:41 [gpu_worker.py:81] Sleep mode freed 33.23 GiB memory, 41.97 GiB memory is still in use.�[32m [repeated 4x across cluster]�[0m
�[36m(LLMRayActor pid=387182)�[0m INFO 06-10 23:23:41 [executor_base.py:208] It took 1.504741 seconds to fall asleep.�[32m [repeated 4x across cluster]�[0m
�[36m(LLMRayActor pid=387185)�[0m INFO 06-10 23:23:40 [block_pool.py:255] Successfully reset prefix cache�[32m [repeated 2x across cluster]�[0m
�[36m(PPOTrainer pid=396792)�[0m
Steps [Episode 1]: 11%|█ | 16/150 [3:38:05<30:26:30, 817.84s/it, policy_loss=-0.0976, actor_lr=9.88e-7, kl=0, reward=0.301, response_length=604, total_length=2.84e+3, return=-4.66e-9]
Traceback (most recent call last):
File "/root/miniconda3/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/root/miniconda3/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/root/.cache/ray/session_2025-06-10_19-43-09_008202_379889/runtime_resources/working_dir_files/_ray_pkg_2a493473a90f7883/openrlhf/cli/train_ppo_ray.py", line 500, in
train(args)
File "/root/.cache/ray/session_2025-06-10_19-43-09_008202_379889/runtime_resources/working_dir_files/_ray_pkg_2a493473a90f7883/openrlhf/cli/train_ppo_ray.py", line 167, in train
ray.get(ppo_trainer.fit.remote())
File "/root/miniconda3/lib/python3.10/site-packages/ray/_private/auto_init_hook.py", line 21, in auto_init_wrapper
return fn(*args, **kwargs)
File "/root/miniconda3/lib/python3.10/site-packages/ray/_private/client_mode_hook.py", line 103, in wrapper
return func(*args, **kwargs)
File "/root/miniconda3/lib/python3.10/site-packages/ray/_private/worker.py", line 2771, in get
values, debugger_breakpoint = worker.get_objects(object_refs, timeout=timeout)
File "/root/miniconda3/lib/python3.10/site-packages/ray/_private/worker.py", line 919, in get_objects
raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(AssertionError): �[36mray::PPOTrainer.fit()�[39m (pid=396792, ip=0.0.0.0, actor_id=ab0436db2d6f11d97921d33502000000, repr=<openrlhf.trainer.ppo_trainer.PPOTrainer object at 0x7efa5e688490>)
File "/root/.cache/ray/session_2025-06-10_19-43-09_008202_379889/runtime_resources/working_dir_files/_ray_pkg_2a493473a90f7883/openrlhf/trainer/ppo_trainer.py", line 194, in fit
self.replay_buffer.extend(experiences)
File "/root/.cache/ray/session_2025-06-10_19-43-09_008202_379889/runtime_resources/working_dir_files/_ray_pkg_2a493473a90f7883/openrlhf/trainer/ppo_utils/replay_buffer.py", line 219, in extend
self.append(experience)
File "/root/miniconda3/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
File "/root/.cache/ray/session_2025-06-10_19-43-09_008202_379889/runtime_resources/working_dir_files/_ray_pkg_2a493473a90f7883/openrlhf/trainer/ppo_utils/replay_buffer.py", line 207, in append
items = split_experience_batch(experience, self.data_processor)
File "/root/.cache/ray/session_2025-06-10_19-43-09_008202_379889/runtime_resources/working_dir_files/_ray_pkg_2a493473a90f7883/openrlhf/trainer/ppo_utils/replay_buffer.py", line 70, in split_experience_batch
visual_inputs_chunks = data_processor.split_input_batch(visual_inputs_batch)
File "/root/.cache/ray/session_2025-06-10_19-43-09_008202_379889/runtime_resources/working_dir_files/_ray_pkg_2a493473a90f7883/openrlhf/models/lmm_kits/qwen2_5_vl/data_processor.py", line 104, in split_input_batch
assert vision_start_num == vision_end_num
AssertionError
2025-06-10 23:24:22,770 ERR cli.py:71 -- �[31m---------------------------------------�[39m
2025-06-10 23:24:22,770 ERR cli.py:72 -- �[31mJob 'raysubmit_niv53yJ6Jf2ygH2q' failed�[39m
2025-06-10 23:24:22,770 ERR cli.py:73 -- �[31m---------------------------------------�[39m
2025-06-10 23:24:22,770 INFO cli.py:86 -- Status message: Job entrypoint command failed with exit code 1, last available logs (truncated to 20,000 chars):
self.append(experience)
File "/root/miniconda3/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
File "/root/.cache/ray/session_2025-06-10_19-43-09_008202_379889/runtime_resources/working_dir_files/_ray_pkg_2a493473a90f7883/openrlhf/trainer/ppo_utils/replay_buffer.py", line 207, in append
items = split_experience_batch(experience, self.data_processor)
File "/root/.cache/ray/session_2025-06-10_19-43-09_008202_379889/runtime_resources/working_dir_files/_ray_pkg_2a493473a90f7883/openrlhf/trainer/ppo_utils/replay_buffer.py", line 70, in split_experience_batch
visual_inputs_chunks = data_processor.split_input_batch(visual_inputs_batch)
File "/root/.cache/ray/session_2025-06-10_19-43-09_008202_379889/runtime_resources/working_dir_files/_ray_pkg_2a493473a90f7883/openrlhf/models/lmm_kits/qwen2_5_vl/data_processor.py", line 104, in split_input_batch
assert vision_start_num == vision_end_num
AssertionError
Cleaning up...
2025-06-10 23:24:24,289 VINFO scripts.py:1215 -- Send termination request to �[1m/root/miniconda3/lib/python3.10/site-packages/ray/core/src/ray/raylet/raylet --raylet_socket_name=/root/.cache/ray/session_2025-06-10_19-43-09_008202_379889/sockets/raylet --store_socket_name=/root/.cache/ray/session_2025-06-10_19-43-09_008202_379889/sockets/plasma_store --object_manager_port=0 --min_worker_port=10002 --max_worker_port=19999 --node_manager_port=0 --node_id=9d90b598a59412f4dc246c871f94773ee14f756d11303a46ed864060 --node_ip_address=0.0.0.0 --maximum_startup_concurrency=51 --static_resource_list=node:0.0.0.0,1.0,node:__internal_head__,1.0,accelerator_type:A100,1,CPU,51,GPU,4,memory,288094970880,object_store_memory,127754987520 "--python_worker_command=/root/miniconda3/bin/python3.10 /root/miniconda3/lib/python3.10/site-packages/ray/_private/workers/setup_worker.py /root/miniconda3/lib/python3.10/site-packages/ray/_private/workers/default_worker.py --node-ip-address=0.0.0.0 --node-manager-port=RAY_NODE_MANAGER_PORT_PLACEHOLDER --object-store-name=/root/.cache/ray/session_2025-06-10_19-43-09_008202_379889/sockets/plasma_store --raylet-name=/root/.cache/ray/session_2025-06-10_19-43-09_008202_379889/sockets/raylet --redis-address=None --metrics-agent-port=51747 --logging-rotate-bytes=536870912 --logging-rotate-backup-count=5 --runtime-env-agent-port=55185 --gcs-address=0.0.0.0:6379 --session-name=session_2025-06-10_19-43-09_008202_379889 --temp-dir=/root/.cache/ray --webui=127.0.0.1:8265 --cluster-id=0707a2661d1a594d509db90ebae786e8a78c65b121cb85272b553f37 RAY_WORKER_DYNAMIC_OPTION_PLACEHOLDER" "--java_worker_command=/root/miniconda3/bin/python3.10 /root/miniconda3/lib/python3.10/site-packages/ray/_private/workers/setup_worker.py -Dray.address=0.0.0.0:6379 -Dray.raylet.node-manager-port=RAY_NODE_MANAGER_PORT_PLACEHOLDER -Dray.object-store.socket-name=/root/.cache/ray/session_2025-06-10_19-43-09_008202_379889/sockets/plasma_store -Dray.raylet.socket-name=/root/.cache/ray/session_2025-06-10_19-43-09_008202_379889/sockets/raylet -Dray.redis.username= -Dray.redis.password= -Dray.node-ip=0.0.0.0 -Dray.home=/root/miniconda3/lib/python3.10/site-packages/ray/../.. -Dray.logging.dir=/root/.cache/ray/session_2025-06-10_19-43-09_008202_379889/logs -Dray.session-dir=/root/.cache/ray/session_2025-06-10_19-43-09_008202_379889 RAY_WORKER_DYNAMIC_OPTION_PLACEHOLDER io.ray.runtime.runner.worker.DefaultWorker" --cpp_worker_command= --native_library_path=/root/miniconda3/lib/python3.10/site-packages/ray/cpp/lib --temp_dir=/root/.cache/ray --session_dir=/root/.cache/ray/session_2025-06-10_19-43-09_008202_379889 --log_dir=/root/.cache/ray/session_2025-06-10_19-43-09_008202_379889/logs --resource_dir=/root/.cache/ray/session_2025-06-10_19-43-09_008202_379889/runtime_resources --metrics-agent-port=51747 --metrics_export_port=64658 --runtime_env_agent_port=55185 --object_store_memory=127754987520 --plasma_directory=/dev/shm --ray-debugger-external=0 --gcs-address=0.0.0.0:6379 --session-name=session_2025-06-10_19-43-09_008202_379889 --labels= --cluster-id=0707a2661d1a594d509db90ebae786e8a78c65b121cb85272b553f37 --ray_log_filepath=/root/.cache/ray/session_2025-06-10_19-43-09_008202_379889/logs/raylet.out --ray_err_log_filepath=/root/.cache/ray/session_2025-06-10_19-43-09_008202_379889/logs/raylet.err --head --num_prestart_python_workers=51 "--dashboard_agent_command=/root/miniconda3/bin/python3.10 -u /root/miniconda3/lib/python3.10/site-packages/ray/dashboard/agent.py --node-ip-address=0.0.0.0 --metrics-export-port=64658 --dashboard-agent-port=51747 --listen-port=52365 --node-manager-port=RAY_NODE_MANAGER_PORT_PLACEHOLDER --object-store-name=/root/.cache/ray/session_2025-06-10_19-43-09_008202_379889/sockets/plasma_store --raylet-name=/root/.cache/ray/session_2025-06-10_19-43-09_008202_379889/sockets/raylet --temp-dir=/root/.cache/ray --session-dir=/root/.cache/ray/session_2025-06-10_19-43-09_008202_379889 --log-dir=/root/.cache/ray/session_2025-06-10_19-43-09_008202_379889/logs --logging-rotate-bytes=536870912 --logging-rotate-backup-count=5 --session-name=session_2025-06-10_19-43-09_008202_379889 --gcs-address=0.0.0.0:6379 --cluster-id-hex=0707a2661d1a594d509db90ebae786e8a78c65b121cb85272b553f37" "--runtime_env_agent_command=/root/miniconda3/bin/python3.10 -u /root/miniconda3/lib/python3.10/site-packages/ray/_private/runtime_env/agent/main.py --node-ip-address=0.0.0.0 --runtime-env-agent-port=55185 --gcs-address=0.0.0.0:6379 --cluster-id-hex=0707a2661d1a594d509db90ebae786e8a78c65b121cb85272b553f37 --runtime-env-dir=/root/.cache/ray/session_2025-06-10_19-43-09_008202_379889/runtime_resources --logging-rotate-bytes=536870912 --logging-rotate-backup-count=5 --log-dir=/root/.cache/ray/session_2025-06-10_19-43-09_008202_379889/logs --temp-dir=/root/.cache/ray" --node-name=0.0.0.0�[22m�[26m �[2m(via SIGTERM)�[22m�[26m
2025-06-10 23:24:29,694 VINFO scripts.py:1215 -- Send termination request to �[1m/root/miniconda3/bin/python3.10 -u /root/miniconda3/lib/python3.10/site-packages/ray/autoscaler/_private/monitor.py --logs-dir=/root/.cache/ray/session_2025-06-10_19-43-09_008202_379889/logs --logging-rotate-bytes=536870912 --logging-rotate-backup-count=5 --gcs-address=0.0.0.0:6379 --monitor-ip=0.0.0.0�[22m�[26m �[2m(via SIGTERM)�[22m�[26m
2025-06-10 23:24:29,694 VINFO scripts.py:1215 -- Se
已经训了16个step，突然出现：
forward: 91%|█████████ | 29/32 [00:23<00:01, 1.64it/s]
�[36m(ActorModelRayActor pid=396186)�[0m
forward: 94%|█████████▍| 30/32 [00:23<00:01, 1.64it/s]
�[36m(ActorModelRayActor pid=396186)�[0m
forward: 97%|█████████▋| 31/32 [00:24<00:00, 1.65it/s]
�[36m(ActorModelRayActor pid=396186)�[0m
forward: 100%|██████████| 32/32 [00:24<00:00, 1.65it/s]
forward: 100%|██████████| 32/32 [00:24<00:00, 1.29it/s]
�[36m(PPOTrainer pid=396792)�[0m INFO 06-10 23:24:11 experience_maker.py:446] ✨ Experience making completed in 0:00:29
�[36m(LLMRayActor pid=387182)�[0m INFO 06-10 23:23:41 [gpu_worker.py:81] Sleep mode freed 33.23 GiB memory, 41.97 GiB memory is still in use.�[32m [repeated 4x across cluster]�[0m
�[36m(LLMRayActor pid=387182)�[0m INFO 06-10 23:23:41 [executor_base.py:208] It took 1.504741 seconds to fall asleep.�[32m [repeated 4x across cluster]�[0m
�[36m(LLMRayActor pid=387185)�[0m INFO 06-10 23:23:40 [block_pool.py:255] Successfully reset prefix cache�[32m [repeated 2x across cluster]�[0m
�[36m(PPOTrainer pid=396792)�[0m
Steps [Episode 1]: 11%|█ | 16/150 [3:38:05<30:26:30, 817.84s/it, policy_loss=-0.0976, actor_lr=9.88e-7, kl=0, reward=0.301, response_length=604, total_length=2.84e+3, return=-4.66e-9]
Traceback (most recent call last):
File "/root/miniconda3/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/root/miniconda3/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/root/.cache/ray/session_2025-06-10_19-43-09_008202_379889/runtime_resources/working_dir_files/_ray_pkg_2a493473a90f7883/openrlhf/cli/train_ppo_ray.py", line 500, in
train(args)
File "/root/.cache/ray/session_2025-06-10_19-43-09_008202_379889/runtime_resources/working_dir_files/_ray_pkg_2a493473a90f7883/openrlhf/cli/train_ppo_ray.py", line 167, in train
ray.get(ppo_trainer.fit.remote())
File "/root/miniconda3/lib/python3.10/site-packages/ray/_private/auto_init_hook.py", line 21, in auto_init_wrapper
return fn(*args, **kwargs)
File "/root/miniconda3/lib/python3.10/site-packages/ray/_private/client_mode_hook.py", line 103, in wrapper
return func(*args, **kwargs)
File "/root/miniconda3/lib/python3.10/site-packages/ray/_private/worker.py", line 2771, in get
values, debugger_breakpoint = worker.get_objects(object_refs, timeout=timeout)
File "/root/miniconda3/lib/python3.10/site-packages/ray/_private/worker.py", line 919, in get_objects
raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(AssertionError): �[36mray::PPOTrainer.fit()�[39m (pid=396792, ip=0.0.0.0, actor_id=ab0436db2d6f11d97921d33502000000, repr=<openrlhf.trainer.ppo_trainer.PPOTrainer object at 0x7efa5e688490>)
File "/root/.cache/ray/session_2025-06-10_19-43-09_008202_379889/runtime_resources/working_dir_files/_ray_pkg_2a493473a90f7883/openrlhf/trainer/ppo_trainer.py", line 194, in fit
self.replay_buffer.extend(experiences)
File "/root/.cache/ray/session_2025-06-10_19-43-09_008202_379889/runtime_resources/working_dir_files/_ray_pkg_2a493473a90f7883/openrlhf/trainer/ppo_utils/replay_buffer.py", line 219, in extend
self.append(experience)
File "/root/miniconda3/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
File "/root/.cache/ray/session_2025-06-10_19-43-09_008202_379889/runtime_resources/working_dir_files/_ray_pkg_2a493473a90f7883/openrlhf/trainer/ppo_utils/replay_buffer.py", line 207, in append
items = split_experience_batch(experience, self.data_processor)
File "/root/.cache/ray/session_2025-06-10_19-43-09_008202_379889/runtime_resources/working_dir_files/_ray_pkg_2a493473a90f7883/openrlhf/trainer/ppo_utils/replay_buffer.py", line 70, in split_experience_batch
visual_inputs_chunks = data_processor.split_input_batch(visual_inputs_batch)
File "/root/.cache/ray/session_2025-06-10_19-43-09_008202_379889/runtime_resources/working_dir_files/_ray_pkg_2a493473a90f7883/openrlhf/models/lmm_kits/qwen2_5_vl/data_processor.py", line 104, in split_input_batch
assert vision_start_num == vision_end_num
AssertionError
2025-06-10 23:24:22,770 ERR cli.py:71 -- �[31m---------------------------------------�[39m
2025-06-10 23:24:22,770 ERR cli.py:72 -- �[31mJob 'raysubmit_niv53yJ6Jf2ygH2q' failed�[39m
2025-06-10 23:24:22,770 ERR cli.py:73 -- �[31m---------------------------------------�[39m
2025-06-10 23:24:22,770 INFO cli.py:86 -- Status message: Job entrypoint command failed with exit code 1, last available logs (truncated to 20,000 chars):
self.append(experience)
File "/root/miniconda3/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
File "/root/.cache/ray/session_2025-06-10_19-43-09_008202_379889/runtime_resources/working_dir_files/_ray_pkg_2a493473a90f7883/openrlhf/trainer/ppo_utils/replay_buffer.py", line 207, in append
items = split_experience_batch(experience, self.data_processor)
File "/root/.cache/ray/session_2025-06-10_19-43-09_008202_379889/runtime_resources/working_dir_files/_ray_pkg_2a493473a90f7883/openrlhf/trainer/ppo_utils/replay_buffer.py", line 70, in split_experience_batch
visual_inputs_chunks = data_processor.split_input_batch(visual_inputs_batch)
File "/root/.cache/ray/session_2025-06-10_19-43-09_008202_379889/runtime_resources/working_dir_files/_ray_pkg_2a493473a90f7883/openrlhf/models/lmm_kits/qwen2_5_vl/data_processor.py", line 104, in split_input_batch
assert vision_start_num == vision_end_num
AssertionError
Cleaning up...
2025-06-10 23:24:24,289 VINFO scripts.py:1215 -- Send termination request to
�[1m/root/miniconda3/lib/python3.10/site-packages/ray/core/src/ray/raylet/raylet --raylet_socket_name=/root/.cache/ray/session_2025-06-10_19-43-09_008202_379889/sockets/raylet --store_socket_name=/root/.cache/ray/session_2025-06-10_19-43-09_008202_379889/sockets/plasma_store --object_manager_port=0 --min_worker_port=10002 --max_worker_port=19999 --node_manager_port=0 --node_id=9d90b598a59412f4dc246c871f94773ee14f756d11303a46ed864060 --node_ip_address=0.0.0.0 --maximum_startup_concurrency=51 --static_resource_list=node:0.0.0.0,1.0,node:__internal_head__,1.0,accelerator_type:A100,1,CPU,51,GPU,4,memory,288094970880,object_store_memory,127754987520 "--python_worker_command=/root/miniconda3/bin/python3.10 /root/miniconda3/lib/python3.10/site-packages/ray/_private/workers/setup_worker.py /root/miniconda3/lib/python3.10/site-packages/ray/_private/workers/default_worker.py --node-ip-address=0.0.0.0 --node-manager-port=RAY_NODE_MANAGER_PORT_PLACEHOLDER --object-store-name=/root/.cache/ray/session_2025-06-10_19-43-09_008202_379889/sockets/plasma_store --raylet-name=/root/.cache/ray/session_2025-06-10_19-43-09_008202_379889/sockets/raylet --redis-address=None --metrics-agent-port=51747 --logging-rotate-bytes=536870912 --logging-rotate-backup-count=5 --runtime-env-agent-port=55185 --gcs-address=0.0.0.0:6379 --session-name=session_2025-06-10_19-43-09_008202_379889 --temp-dir=/root/.cache/ray --webui=127.0.0.1:8265 --cluster-id=0707a2661d1a594d509db90ebae786e8a78c65b121cb85272b553f37 RAY_WORKER_DYNAMIC_OPTION_PLACEHOLDER" "--java_worker_command=/root/miniconda3/bin/python3.10 /root/miniconda3/lib/python3.10/site-packages/ray/_private/workers/setup_worker.py -Dray.address=0.0.0.0:6379 -Dray.raylet.node-manager-port=RAY_NODE_MANAGER_PORT_PLACEHOLDER -Dray.object-store.socket-name=/root/.cache/ray/session_2025-06-10_19-43-09_008202_379889/sockets/plasma_store -Dray.raylet.socket-name=/root/.cache/ray/session_2025-06-10_19-43-09_008202_379889/sockets/raylet -Dray.redis.username= -Dray.redis.password= -Dray.node-ip=0.0.0.0 -Dray.home=/root/miniconda3/lib/python3.10/site-packages/ray/../.. -Dray.logging.dir=/root/.cache/ray/session_2025-06-10_19-43-09_008202_379889/logs -Dray.session-dir=/root/.cache/ray/session_2025-06-10_19-43-09_008202_379889 RAY_WORKER_DYNAMIC_OPTION_PLACEHOLDER io.ray.runtime.runner.worker.DefaultWorker" --cpp_worker_command= --native_library_path=/root/miniconda3/lib/python3.10/site-packages/ray/cpp/lib --temp_dir=/root/.cache/ray --session_dir=/root/.cache/ray/session_2025-06-10_19-43-09_008202_379889 --log_dir=/root/.cache/ray/session_2025-06-10_19-43-09_008202_379889/logs --resource_dir=/root/.cache/ray/session_2025-06-10_19-43-09_008202_379889/runtime_resources --metrics-agent-port=51747 --metrics_export_port=64658 --runtime_env_agent_port=55185 --object_store_memory=127754987520 --plasma_directory=/dev/shm --ray-debugger-external=0 --gcs-address=0.0.0.0:6379 --session-name=session_2025-06-10_19-43-09_008202_379889 --labels= --cluster-id=0707a2661d1a594d509db90ebae786e8a78c65b121cb85272b553f37 --ray_log_filepath=/root/.cache/ray/session_2025-06-10_19-43-09_008202_379889/logs/raylet.out --ray_err_log_filepath=/root/.cache/ray/session_2025-06-10_19-43-09_008202_379889/logs/raylet.err --head --num_prestart_python_workers=51 "--dashboard_agent_command=/root/miniconda3/bin/python3.10 -u /root/miniconda3/lib/python3.10/site-packages/ray/dashboard/agent.py --node-ip-address=0.0.0.0 --metrics-export-port=64658 --dashboard-agent-port=51747 --listen-port=52365 --node-manager-port=RAY_NODE_MANAGER_PORT_PLACEHOLDER --object-store-name=/root/.cache/ray/session_2025-06-10_19-43-09_008202_379889/sockets/plasma_store --raylet-name=/root/.cache/ray/session_2025-06-10_19-43-09_008202_379889/sockets/raylet --temp-dir=/root/.cache/ray --session-dir=/root/.cache/ray/session_2025-06-10_19-43-09_008202_379889 --log-dir=/root/.cache/ray/session_2025-06-10_19-43-09_008202_379889/logs --logging-rotate-bytes=536870912 --logging-rotate-backup-count=5 --session-name=session_2025-06-10_19-43-09_008202_379889 --gcs-address=0.0.0.0:6379 --cluster-id-hex=0707a2661d1a594d509db90ebae786e8a78c65b121cb85272b553f37" "--runtime_env_agent_command=/root/miniconda3/bin/python3.10 -u /root/miniconda3/lib/python3.10/site-packages/ray/_private/runtime_env/agent/main.py --node-ip-address=0.0.0.0 --runtime-env-agent-port=55185 --gcs-address=0.0.0.0:6379 --cluster-id-hex=0707a2661d1a594d509db90ebae786e8a78c65b121cb85272b553f37 --runtime-env-dir=/root/.cache/ray/session_2025-06-10_19-43-09_008202_379889/runtime_resources --logging-rotate-bytes=536870912 --logging-rotate-backup-count=5 --log-dir=/root/.cache/ray/session_2025-06-10_19-43-09_008202_379889/logs --temp-dir=/root/.cache/ray" --node-name=0.0.0.0�[22m�[26m�[2m(via SIGTERM)�[22m�[26m2025-06-10 23:24:29,694 VINFO scripts.py:1215 -- Send termination request to
�[1m/root/miniconda3/bin/python3.10 -u /root/miniconda3/lib/python3.10/site-packages/ray/autoscaler/_private/monitor.py --logs-dir=/root/.cache/ray/session_2025-06-10_19-43-09_008202_379889/logs --logging-rotate-bytes=536870912 --logging-rotate-backup-count=5 --gcs-address=0.0.0.0:6379 --monitor-ip=0.0.0.0�[22m�[26m�[2m(via SIGTERM)�[22m�[26m2025-06-10 23:24:29,694 VINFO scripts.py:1215 -- Se