model can not saved

[glennccc]

when setting --save_steps 4, even though training after 1 epoch. I can't see anything in the save_path.
the training process doesn't meet any error.

[glennccc]

@HYZ17 @Zeng-WH @jxhe @SivilTaram did you miss writing code to save checkpoint? I can't see any use about save_path and save_steps in train_ppo_ray_box.py

[HYZ17]

Hi, the code should be correct, and the checkpoint during training should be stored under ckpt_path. According to the implementation, saving occurs every save_steps; however, please note that one step here refers to a PPO step rather than a gradient step.

[glennccc]

Hi, the code should be correct, and the checkpoint during training should be stored under ckpt_path. According to the implementation, saving occurs every save_steps; however, please note that one step here refers to a PPO step rather than a gradient step.

python3 openrlhf/cli/train_ppo_ray_box.py
--ref_num_nodes 1
--ref_num_gpus_per_node 2
--reward_num_nodes 0
--reward_num_gpus_per_node 0
--critic_num_nodes 1
--critic_num_gpus_per_node 2
--actor_num_nodes 1
--actor_num_gpus_per_node 2
--vllm_num_engines 2
--vllm_tensor_parallel_size 1
--colocate_actor_ref
--pretrain {path}/Qwen2.5-Math-7B
--save_path {path}/simplerl/checkpoints/Qwen2.5-Math-7B_ppo_from_base_math_lv35
--micro_train_batch_size 2
--train_batch_size 128
--micro_rollout_batch_size 2
--rollout_batch_size 1024
--temperature 0.6
--n_samples_per_prompt 8
--max_samples 100000
--max_epochs 1
--num_episodes 20
--prompt_max_len 1024
--generate_max_len 3000
--zero_stage 3
--bf16
--actor_learning_rate 5e-7
--critic_learning_rate 9e-6
--init_kl_coef 0.01
--prompt_data {path}/simpleRL-reason/train/data/math_level3to5_data_processed_with_qwen_prompt.json
--input_key input
--normalize_reward
--flash_attn
--gradient_checkpointing
--save_steps 4
--logging_steps 1
--adam_offload

I use it to train on one A100 gpu, do i get some problems with these scripts?
I can't see anything in save_path.