How is RLHN Stage2 Handled if `<worse>` Tag is Empty in Stage1?

[whybe-choi]

Dear Authors,

First of all, I would like to thank you for providing such excellent insights through your research.

I have a question regarding the process of RLHN. As I understand it, in stage 1, you use gpt-4o-mini and assume that the documents within the <better> </better> tags are treated as positives, while those within the <worse> </worse> tags are considered hard negatives. Based on this, stage 2 RLHN is conducted. My question is: if there are no documents inside the <worse> </worse> tags at stage 1, it seems that there would be no values passed to the <documents> </documents> tag in stage 2. I am curious how you handled such cases.

Regards,
Yongbin Choi

[thakur-nandan]

Hi Yongbin,

Thanks for your question.

For the decision to use the gpt-4o judge in stage 2 after gpt-4o-mini in stage 1 is if we find any document in either <worse>...</worse> or <better>...</better> in the gpt-4o-mini response, we take it to the next stage and rerun the prompt completely with gpt-4o.

Only for stage-1 baselines mentioned in the paper, we focus on the negatives within the <better>...</better> tags.

[whybe-choi]

Thanks for your kindness!

In other words, when you use gpt-4o-mini as the judge in stage1 and there are documents inside <worse>...</worse> or <better>...</better>, are you saying that the results of stage1 are not reflected separately and you judge again in stage2 using gpt-4o?

Then, I have two additional questions:

1. I understood that the documents in <better>...</better> are false negatives, and the documents in <worse>...</worse> are hard negatives. Is my understanding correct?
2. If there are no documents inside <better>...</better> or <worse>...</worse> after going through stage2, can I consider that there are no false negatives or hard negatives, and therefore, if I want to utilize hard negatives, should I remove those samples?