Thanks for open-sourcing such an unique benchmark dataset for RAG. It's the only publicly available multi-hop RAG benchmark for a while.
When reading the qa_evaluate.py, I was pretty confused for the algorithm.
My understanding is, len(gold_list) == len(pred_list) == number of queries. Since in this evaluation task, fp=number of queries - tp, we would have fp==fn==tn. Under this circumstance, I do not think the calculated precision, recall, and F1 score makes more sense than a simple Accuracy.
Also, for how the tn is calculated ---- tn = len(pred_list) - tp, it just doesn't hold up under scrutiny.
Thanks for open-sourcing such an unique benchmark dataset for RAG. It's the only publicly available multi-hop RAG benchmark for a while.
When reading the
qa_evaluate.py, I was pretty confused for the algorithm.My understanding is,
len(gold_list)==len(pred_list)==number of queries. Since in this evaluation task,fp=number of queries-tp, we would havefp==fn==tn. Under this circumstance, I do not think the calculated precision, recall, and F1 score makes more sense than a simple Accuracy.Also, for how the
tnis calculated ----tn = len(pred_list) - tp, it just doesn't hold up under scrutiny.