Skip to content

Replace the SelectionStrategy with a "Number of Judgements" #1171

Open
@epugh

Description

Is your feature request related to a problem? Please describe.
we have a single rater / mulitple rater support, but generally always pick multiple rather

Describe the solution you'd like
@david-fisher would like to instead have a default number, say 3, but you can tweak it. Max is the total number of judges. (show a warning message). So 10 judges, over 10 querys, would be featisdble. And good training.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
A use case would be "training new judges." Let's get ten judges to judge five queires five deep, and then we do IRR and talk about how to be better judges!

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions