Re-Verify `tests/gold_set.csv`: Confirm Each Label Against Its Diff

[khatchad]

tests/gold_set.csv is the oracle for the classifier (#11) and its evaluation, so its labels need to be tight. A verification pass with analysis/inspect_commit.py (including the fixed --download from #37) surfaced a set of rows to re-check. This is normal QA in the wake of the #34 tool bug plus a move to judging every label from the diff rather than the filename, not a reflection on any one annotator.

Multiple Refactorings per Commit (Not a Bug, One Row per Refactoring): 37 rows/33 commits is expected where a commit does several things (e.g. imageomics @38a17b32 = add column(s) + standardize; syntaxnoob @a2061d30 = add + rename/split). No action beyond confirming each co-refactoring is real. Only genuine cleanup: gtfintechlab/finer-ord @7067c8ac has two identical not-a-refactoring rows → drop one.

Label Not Supported by the Diff:

• sheryylli/utr_total_reads (delete column(s) → the commit is an add)
• dragosnicolae555/RoITD (change file type → only adds an xlsx)
• scikit-learn/student-alcohol-consumption (merge → adds an R script)
• arize-ai/movie_reviews_with_context_drift (delete file → no changes found)

Labeled From the Filename on a First-Upload Add (Re-Judge From the Change or Mark Unverifiable):

• Rapidata/multilingual-llm-jokes-4o
• Abdulahad79/Crop_Classification
• AI-Growth-Lab/patents_claims_1.5m (×2)
• electricsheepafrica/age-standardized...

delete file Needs the Intent (dedup/cleanup/not-a-refactoring):

• pauri32/fiqa-2018
• hagara/labelled-PubMedQA
• Allen0307/LLMAgent
• hanR07/safe_packets
• joonhok-exo-ai/korean_law

Not Verifiable by inspect_commit.py (xlsx/value-level/row-level), Confirm Manually:

• hafsteinn/faroese_sentiment (xlsx)
• stacked-summaries/stacked-samsum (row-level dedup)
• phonemetransformers/IPA-CHILDES (value-level)
• BramVanroy/xlwic_wn (txt)

LFS Column-Change Rows, Redo With the Fixed --download: tracked in #38. First pass done: TheoTsio confirms delete column(s); the imageomics/nus-yam add-column and syntaxnoob rename sides confirm.

Definition of Done: one row per refactoring (a commit may legitimately have several); each label supported by its diff or marked with an explicit "unverifiable" note; the redundant finer-ord duplicate dropped; provenance recorded in notes; label vocabulary reconciled (the protocol seed vs. the finer file-level codes here).

[khatchad]

@miridonoso, heads-up, and definitely not a criticism of your labeling. Now that the --download bug (#34) is fixed in #37, I re-ran inspect_commit.py over the whole gold set to double-check labels against the actual diffs, and it surfaced the checklist above.

Most of it is benign: rows on first-upload commits where the change wasn't visible before the fix, plus a few worth a second look against the diff. And to be clear, the "37 rows/33 commits" thing is not a problem. That's your one-row-per-refactoring convention working exactly as intended for commits that do more than one thing.

Could you take a pass through these when you have a chance? Happy to go through the trickier ones together and adjudicate. Thanks for building all of this. The tooling made this re-check a five-minute job.

[khatchad]

Gold Set Synced

tests/gold_set.csv now carries the re-verification results from this issue, as of 429b3be. Fifteen commits had corrected labels.

Duplicate Dropped

The redundant gtfintechlab/finer-ord @ 7067c8ac row is gone, keeping the miridonoso-verified one. That is the cleanup this issue called for.

One Row Still Open

saraprice/OpenHermes-FN-headlines-SA-ihateyou @ b339c8e6 is still unlabeled. @miridonoso hit an LFS fetch failure there, and she was right that it fails. I tracked down why and filed #44: the repo has migrated to Xet storage, which defeats the smudge path, but the file is still fetchable by revision over HTTPS. The before/after column diff is in that issue, so the row can be labeled without LFS working.

Leaving this open until that row lands.