TL;DR
Allow a second reviewer to weigh in on a labeled trace, either to resolve a contested verdict or to measure inter-annotator agreement.
Current State
Single-reviewer flow only. When a label feels uncertain, there is no way to get a second pair of eyes inside the tool. Disagreements between reviewers cannot be surfaced or reconciled.
Desired State
A reviewer can request (or be assigned) a second pass on a trace. The tool surfaces both labels side by side, flags disagreement, and supports a resolution path. Inter-annotator agreement is visible at the session or dataset level so reviewers can calibrate.
TL;DR
Allow a second reviewer to weigh in on a labeled trace, either to resolve a contested verdict or to measure inter-annotator agreement.
Current State
Single-reviewer flow only. When a label feels uncertain, there is no way to get a second pair of eyes inside the tool. Disagreements between reviewers cannot be surfaced or reconciled.
Desired State
A reviewer can request (or be assigned) a second pass on a trace. The tool surfaces both labels side by side, flags disagreement, and supports a resolution path. Inter-annotator agreement is visible at the session or dataset level so reviewers can calibrate.