Re-verify gold-set rows labeled from LFS-tracked files

[khatchad]

Fallout from #34. Before the fix in #37, inspect_commit.py --download could print file changes and then report "no column change" on an LFS-tracked file whose columns it never read. git lfs pull exits 0 when it fetches nothing, so there was no crash, no warning, and no way for the person labeling to tell "this commit changed no columns" apart from "we could not read either side."

Any row in tests/gold_set.csv whose label was informed by that output could be wrong, specifically in the direction of a false negative: a real add column(s), delete column(s), or rename column(s) recorded as something weaker, or as not-a-refactoring.

This is not hypothetical for the corpus. TheoTsio/Health_Misinfo tracks test/train/val.csv under LFS, and it is in the gold set.

What Needs Doing

1. Identify which gold-set commits touch LFS-tracked files. Nothing in gold_set.csv records this (it has only DatasetID, CommitId, label, verified_by, notes), so it has to be recovered by re-running the inspection.
2. Re-inspect those commits with the fixed --download from Fix --download silently misreporting LFS-tracked files #37 and confirm the label.
3. Where a label was reached by other means (reading the diff on the Hub, a manual download), note that in notes so the provenance is recoverable next time.
4. Add the saraprice/OpenHermes-FN-headlines-SA-ihateyou rows. They exist in the annotation source but not in the committed CSV: the 2026-07-21 export left them out because the rows could not be verified until the Xet fetch fix landed. The flagged column-reorder row is adjudicated as restructure/reorg/consolidate and that label stands, so these need finalizing rather than re-judging.
5. Land the corrected labels in tests/gold_set.csv and push. This issue is not done until the committed file reflects the re-verification; a corrected label that has not reached main is not a corrected label as far as the classifier, its evaluation, or the second rater are concerned.

Why It Matters Beyond The Gold Set

The gold set is the development oracle for the classifier (#11) and the basis for its evaluation. A systematic false-negative bias in the oracle would show up as the classifier looking worse than it is on exactly the column-change types, and would quietly corrupt the per-type precision/recall numbers.

Worth doing before the classifier is scored, not after.

Sequencing

Unblocked as of 2026-07-21. This needed fixes to two separate LFS read paths, both now on main: #37 for --download under classic LFS, and #44 (PR #45) for Xet-backed repositories. Pull main before starting.

[khatchad]

A second instance of this failure mode, from a different cause. Reopening the scope rather than filing a new issue, since the rows and the remedy are the same.

#38 as written is fallout from #34: git lfs pull exiting 0 having fetched nothing. There is now a second path to the identical symptom. load_lfs_pointer() resolves a pointer by piping it to git lfs smudge, and as its own docstring notes, smudge "exits 0 and echoes the pointer straight back when it cannot fetch the object." For repositories migrated to Hugging Face's Xet storage, that is exactly what happens: the pointer comes back, is_lfs_pointer() is true, and the caller returns UNRESOLVED. Same silence, same false-negative direction, different cause. Details and the fix are in #44.

So the re-inspection in step 2 needs the fix from #44 as well as the one from #37 that it was originally blocked on.

Rows In Scope

Seven of the 35 rows in the current tests/gold_set.csv carry an LFS marker in notes:

• TheoTsio/Health_Misinfo — delete column(s)
• imageomics/Heliconius-Collection_Cambridge-Butterfly — add column(s)
• nus-yam/ex-repair — add column(s)
• syntaxnoob/weather-prediction-prototype-aws — add column(s)
• syntaxnoob/weather-prediction-prototype-aws — rename column(s)
• science-of-finetuning/diffing-stats-Meta-Llama-3.1-8B — not-a-refactoring
• jinnovation/generative-ai-red-teaming — delete column(s)

Three more record a manual download and are worth a second pass now that the tool can fetch directly: stacked-summaries/stacked-samsum-1024, BramVanroy/xlwic_wn, hafsteinn/faroese_sentiment_analysis.

This also partly discharges step 1: the notes field turns out to record LFS-ness in practice even though the schema does not, so the rows did not have to be recovered by re-running everything.

One More Reason To Finish This Before The Classifier

Beyond the oracle-corruption argument already in the description, these rows are about 20% of the gold set. Leaving labels in place that were produced by tooling known to return a false negative means any later comparison against them measures the tooling defect rather than the thing being compared. Re-verify before anything is scored against this file.

[khatchad]

Sequencing note: #52 should land first. --show_rows is the only signal for a commit that changes values while leaving the column set intact, and it currently diffs the local LFS pointer, or silently downloads the full payload, or prints real rows, depending on whether the object happens to sit in the local LFS cache. Re-verifying an LFS-tracked row against that output would repeat the failure this issue exists to undo. Details in #52.

[khatchad]

@miridonoso heads up: I marked #52 as blocking this issue and #40, but it is a partial block, so here is what it does and does not stop.

Still fine to do now: steps 1 and 2, identifying which gold-set commits touch LFS-tracked files and re-inspecting them with --download. The column-diff path is fixed and trustworthy, so any row whose label turns on a column being added, deleted, or renamed can be confirmed today. Step 4, finalizing the saraprice rows, is also unaffected since that one is already adjudicated.

Worth holding: rows where --download reports "no column-set change (values may still differ)" and the label turns on whether the values actually changed. --show_rows is the only signal there and it is currently unreliable, so a row judged on it now may need judging twice. I am working the fix on fix/row-sample-over-resolve and will post here when it lands.

If you hit one of those rows in the meantime, note it in the row's notes and move on rather than guessing.