When I was conducting RL training, I found that Cosmos RL uses VLLM for training, and the current code trajectory is obtained through inverse normalization instead of expert models. However, VLLM is unable to obtain KV cache for expert model trajectory generation, which seems inconsistent with the description in the paper. May I ask, if the complete version of the code cannot be used for RL training without VLLM, how can we use expert models to generate trajectories and perform reward calculations?
When I was conducting RL training, I found that Cosmos RL uses VLLM for training, and the current code trajectory is obtained through inverse normalization instead of expert models. However, VLLM is unable to obtain KV cache for expert model trajectory generation, which seems inconsistent with the description in the paper. May I ask, if the complete version of the code cannot be used for RL training without VLLM, how can we use expert models to generate trajectories and perform reward calculations?