In the paper, MIS-PO only considers filtering samples with large discrepancies between training and inference(vllm,sglang). However, as mentioned in the paper, each rollout is trained four times. Has the impact of off-policy been considered here? Is importance sampling applied?
In the paper, MIS-PO only considers filtering samples with large discrepancies between training and inference(vllm,sglang). However, as mentioned in the paper, each rollout is trained four times. Has the impact of off-policy been considered here? Is importance sampling applied?