Link to agenda where we discussed this.
The process for interviews is that we will set up the Session Evaluator to mark all sessions as unacceptable or acceptable, and then we want to be able to create an annotation queue based on all unacceptable sessions + a random 20% sample of the acceptable sessions.
There are actually a few requests within this one, let me know if you want to separate them about:
1. most urgently, the ability to select based on either Eval outputs or session level tags when creating the annotation queue (even if just manual)
2. Ability for an Eval to create a session-level tag (what we think is actually the best approach for above)
3. ability to set up automated annotation queue based on the criteria mentioned above (nice to have for efficiency, not a blocker)
simon, let me know how you'd like to handle those interrelated but distinct requests. happy to create other tickets
Link to agenda where we discussed this.
The process for interviews is that we will set up the Session Evaluator to mark all sessions as unacceptable or acceptable, and then we want to be able to create an annotation queue based on all unacceptable sessions + a random 20% sample of the acceptable sessions.
There are actually a few requests within this one, let me know if you want to separate them about:
1. most urgently, the ability to select based on either Eval outputs or session level tags when creating the annotation queue (even if just manual)
2. Ability for an Eval to create a session-level tag (what we think is actually the best approach for above)
3. ability to set up automated annotation queue based on the criteria mentioned above (nice to have for efficiency, not a blocker)
simon, let me know how you'd like to handle those interrelated but distinct requests. happy to create other tickets