Build the evidence-grounded rule classifier

[khatchad]

Build the baseline classifier that predicts a commit's data-refactoring type from its change, not just its message. A message-only keyword pass false-positives: a commit whose message says "split" but whose diff only edits the README is not a refactoring. Keep the message anyway, since when the observed change is consistent with it the message supplies intent (a file deleted with a message calling it a duplicate is deduplication, not a plain delete).

Step 0: Featurization

Turn a commit into its evidence: message tokens, change-op types, file counts, column delta, file extensions. All three stages below need this, so it is a prerequisite rather than part of the ML stage, and it is most of the work in this issue. tests/gold_set.csv carries labels only, no features, so something has to resolve each DatasetID/CommitId pair to its evidence.

analysis/classify_commit.py should not consume the output of inspect() in analysis/inspect_commit.py. That function prints and returns None, and its display is partial by design: column diffs only for modified non-LFS CSVs, word diff truncated at 50 lines, failures sent to stderr. Share the underlying pieces instead. Factor clone, lfs_status, header and parse_csv_header into a module both files call, leaving inspect_commit.py as a thin CLI over it and giving classify_commit.py structured evidence rather than printed text. Leave the LFS and Xet fetch logic as it is; it took #37 and #45 to get right.

Stages

Stage 1 is the whole scope of this issue. Do not start 2 or 3 before 1 is scored.

1. Evidence-Grounded Rules. Combine message keywords with the step-0 signals. The bar it has to clear is the message-only keyword pass, especially on false positives.
2. Feature-Based ML. Train something simple on the step-0 features: logistic regression, a decision tree, or k-nearest neighbours.
3. Optional LLM Classifier. Only if it beats the rule baseline, and disclosed as a threat to validity.

This Is Multi-Label

The gold set records one row per refactoring, so a single commit can carry several types (for example add column(s) and standardize in the same commit). Predict a set of types per commit, not a single class. Scoring follows from that: per-label precision/recall/F1, micro- and macro-averaged, plus per-label binary confusion or a type co-occurrence matrix. A single-label confusion matrix does not apply.

Scoring

Developed and scored against the hand-labeled gold set, using the multi-label metrics described above. The evaluation step itself is tracked outside this repo.

Notes

• New label types are expected. Labeling the gold set is what defines the taxonomy, so a commit that fits none of the existing types is a finding, not an error.
• The column delta can come back empty for reasons unrelated to the commit, so do not write a rule that treats empty as evidence of no change. The LFS defects behind that (inspect_commit.py --download: crash on LFS files, and a silent "no column change" when the pull does not materialize #34, LFS content unreadable on Xet-backed dataset repos; fetch by revision instead of git lfs smudge #44) are fixed, but the caution stands.

Tracked Outside This Repo

Labeling the gold set precedes this, and evaluating the classifier follows it. Both are tracked outside this repo, so neither appears in this issue's blocked-by relations.

[khatchad]

@miridonoso Re: where the classifier should live: put it in this repo (Hugging-Face-Dataset-Mining), not the paper repo and not a new one.

The task is defined in terms of two files that already live here: it is scored against tests/gold_set.csv, and it reads the same commits analysis/inspect_commit.py reads. Splitting it out would mean copying or importing across a repo boundary, and the gold set would drift away from the classifier it is supposed to evaluate. ML-Data-Refactoring-Study-Paper stays prose-only: it cites results, it does not produce them.

Suggested layout:

• analysis/classify_commit.py, next to inspect_commit.py.
• tests/test_classify_commit.py, mirroring tests/test_inspect_commit.py.

That also keeps the follow-on evaluation reading the same gold set from the same place.

[khatchad]

The verification pass surfaced a false-positive class that stage 1 can remove with a rule, and it is much larger than the three commits that exposed it.

Keyword In A Filename Is Not An Action

Three commits, each verified not-a-refactoring, each typed by the keyword pass:

• deduplication: souljaboi6801/non_duplicates, "Upload non_duplicate_expense_dataset.csv"
• merge: hingu/chungjudam, "Upload merged_s.csv"
• cleaning: Allen0307/LLMAgent, "Upload clean_data_0.csv"

In each one the keyword sits inside the name of the file being uploaded, so it describes the artifact rather than an action applied to existing data. The first is the sharpest case: the dataset itself is named non_duplicates, so the keyword filter is typing a commit off a noun.

The rule. A keyword that occurs only inside a filename token, on a commit whose sole file operation is an add, is not evidence of a refactoring. It reads the file-operation list and nothing else, so it costs one pass over the diff summary and no download.

How Big The Class Is

Counting rows of data/message_refactoring_candidates.csv where the assigned type's keyword appears inside a filename token and nowhere else in the message:

• 152 of 374 candidates, 40.6%
• 121 of those have an upload or add message shape
• cleaning 72 of 123, merge 45 of 55, normalize/standardize/encode 15 of 41, deduplication 9 of 24, rename/split-standardization 7 of 37, column/field add-remove 4 of 92

merge is the most affected proportionally at 82%. cleaning is the largest in absolute terms.

One caveat, and it matters. This counts the shape, not ground truth. A commit can modify an existing file whose name happens to carry the keyword, and those are real refactorings. So the figure bounds the class rather than measuring the baseline's error rate, and the gold set is what turns it into a precision number. It is still the largest mechanically removable false-positive family found so far, which is what makes it the first rule worth implementing.

A Second Class, Not Covered By That Rule

Mostafa3zazi/wonders_testing_dataset is titled "remove index column", and the diff shows an unnamed column receiving a name, which is the opposite operation. Here the keyword is the action and the diff contradicts it. No filename rule catches this, because it needs the column delta.

Worth scoring the two separately. The first class is removable by a cheap rule with no download; the second is what sets the floor on how far rules alone can go, and is the case that justifies the featurization in step 0.