Open
Description
Is your feature request related to a problem? Please describe.
we have a single rater / mulitple rater support, but generally always pick multiple rather
Describe the solution you'd like
@david-fisher would like to instead have a default number, say 3, but you can tweak it. Max is the total number of judges. (show a warning message). So 10 judges, over 10 querys, would be featisdble. And good training.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
A use case would be "training new judges." Let's get ten judges to judge five queires five deep, and then we do IRR and talk about how to be better judges!
Metadata
Assignees
Labels
No labels