What's the use case for ref_model_adapters?

[piedrahitapablo]

I'm trying to do a DPO training after doing an SFT using LoRA, and I want to do the DPO without merging the adapter back into the model (like it's described in the trl docs).

Checking the LlaMa-Factory docs and some of the existing issues, I see there are two parameters that I can use for this:

• adapter_name_or_path
• ref_model_adapters

But it's not completely clear to me how I should be using them. Checking the issues I only see adapter_name_or_path being mentioned, but in the docs ref_model_adapters is mentioned. Can you please tell me if I should be using both of them?

Existing issues:

• sft以后的lora可以继续通过dpo训练吗，不额外merge #6235
• 如何对Qwen的量化版本模型做sft及dpo #6573