Evaluation data of rule_qa for GPT4, GPT3.5, and Claude

[alirezamshi]

Hi,

Thanks for the cool resource. According to the publication, " rule_qa was also manually evaluated by a law-trained individual". Do you plan to release the annotations for this evaluation? Thanks

[neelguha]

The answers are available here: https://huggingface.co/datasets/nguha/legalbench/viewer/rule_qa/test.

Is this what you're looking for?

[alirezamshi]

Thanks for your response. I meant the evaluation of rule-based application: "Rule-application tasks were evaluated manually by a law-trained individual, who analyzed LLM responses for both correctness and analysis"

[neelguha]

Ah sorry I misunderstood.

• rule_qa is a "rule-application" task. rule_qa was manually evaluated by a legally trained individual, because it is an open-generation task. That individual examined a model's generation and compared it to the answers in the column of the above-linked dataset
• The answers for the rule-application tasks that were used for evaluation can be found on this page: https://hazyresearch.stanford.edu/legalbench/getting-started/

[alirezamshi]

Thanks for the answer. Do you plan to release that human judgement for the evaluation?

[alirezamshi]

Following up on this...