Annotation happens outside the repo, in per-rater spreadsheets, and the labeled results have to be exported into the committed CSVs (tests/gold_set.csv and tests/verified_set.csv). That export used to be hand-rolled every time, with nothing in the repo to do it.
Keeping the export a deliberate, human-triggered step is intentional and should not change: the annotation source stays outside the repo and its location is supplied locally, never committed. What is worth fixing is that the mechanics were improvised on each run.
Status
The verified arm is done. export/verified_set.py produces tests/verified_set.csv from the two annotator sheets and is committed. It satisfies every bullet under "What It Should Do" below, and adds two things this issue did not anticipate:
- It fetches the sheets live rather than reading an exported dump. A stale export cannot detect that its source moved, and on its first live run it found a
CommitId blanked in one sheet that a snapshot would have hidden indefinitely.
- Sheet identifiers, the annotator handle map, and the adjudications file are all passed on the command line rather than committed, since this repository is public.
The gold arm is not. It is tracked outside this repository and is not blocked on anything in it.
Why This Matters
tests/gold_set.csv went unchanged from Jul 7 to Jul 21 while fifteen labels changed upstream, because the export was an implicit step nobody owned. That has since been made explicit in #38's definition of done, but the gold-arm run is still ad hoc, which invites two failure modes that have already occurred:
- Wrong row survives deduplication. Where the same dataset, commit and label appear twice with different
verified_by provenance, the row to keep is the annotator-verified one, not the seed row. A hand-rolled pass picked the seed row on the first attempt because it happened to come first.
- Incomplete rows get published. A row with a
DatasetID and CommitId but no label should never reach the oracle; it has to be excluded and reported, not silently written.
What It Should Do
- Read the annotation source for a given arm, with the source identifier taken from an environment variable or local config file that is not committed.
- Emit exactly the five gold-set columns, identically named (
DatasetID, CommitId, label, verified_by, notes), dropping any extra annotation-only columns.
- Preserve one row per refactoring, including legitimately repeated
DatasetID/CommitId pairs.
- Drop exact
(DatasetID, CommitId, label) duplicates, preferring the annotator-verified row over a seed row, and report what was dropped.
- Refuse to emit rows missing
label, verified_by, or notes, and report them rather than failing silently.
- Print a summary: rows written, unique commits, per-label counts, and everything excluded with the reason.
- End with a trailing newline (see the earlier fix in 32e7611).
Scope Note
Both arms need this, so the gold-arm export should reuse export/verified_set.py's shape rather than being written from scratch. The parts that differ are the label translation and the Action column, and both are reconciliation decisions rather than export mechanics.
One caveat worth stating: an export moves values only. Annotator commentary that lives alongside the cells in the annotation source has no column in the five-column schema and will not survive. If that commentary is worth keeping, it needs a schema decision first; this issue does not solve it.
Annotation happens outside the repo, in per-rater spreadsheets, and the labeled results have to be exported into the committed CSVs (
tests/gold_set.csvandtests/verified_set.csv). That export used to be hand-rolled every time, with nothing in the repo to do it.Keeping the export a deliberate, human-triggered step is intentional and should not change: the annotation source stays outside the repo and its location is supplied locally, never committed. What is worth fixing is that the mechanics were improvised on each run.
Status
The verified arm is done.
export/verified_set.pyproducestests/verified_set.csvfrom the two annotator sheets and is committed. It satisfies every bullet under "What It Should Do" below, and adds two things this issue did not anticipate:CommitIdblanked in one sheet that a snapshot would have hidden indefinitely.The gold arm is not. It is tracked outside this repository and is not blocked on anything in it.
Why This Matters
tests/gold_set.csvwent unchanged from Jul 7 to Jul 21 while fifteen labels changed upstream, because the export was an implicit step nobody owned. That has since been made explicit in #38's definition of done, but the gold-arm run is still ad hoc, which invites two failure modes that have already occurred:verified_byprovenance, the row to keep is the annotator-verified one, not the seed row. A hand-rolled pass picked the seed row on the first attempt because it happened to come first.DatasetIDandCommitIdbut nolabelshould never reach the oracle; it has to be excluded and reported, not silently written.What It Should Do
DatasetID,CommitId,label,verified_by,notes), dropping any extra annotation-only columns.DatasetID/CommitIdpairs.(DatasetID, CommitId, label)duplicates, preferring the annotator-verified row over a seed row, and report what was dropped.label,verified_by, ornotes, and report them rather than failing silently.Scope Note
Both arms need this, so the gold-arm export should reuse
export/verified_set.py's shape rather than being written from scratch. The parts that differ are the label translation and theActioncolumn, and both are reconciliation decisions rather than export mechanics.One caveat worth stating: an export moves values only. Annotator commentary that lives alongside the cells in the annotation source has no column in the five-column schema and will not survive. If that commentary is worth keeping, it needs a schema decision first; this issue does not solve it.