Non-ASCII paths report as 'opaque'/'unanalyzed' due to git's escaped path output, not missing or unreadable data

[arishajohri]

What happened:
A mined commit touches six files inside a non-ASCII (Chinese-named) folder. The analysis scripts determine which files a commit touched by reading git's per-file plan/diff output and splitting each line on a tab character, then checking whether the resulting path string ends in .csv.

Git does not print non-ASCII bytes in a path as-is by default (core.quotePath=true): it wraps the path in double quotes and backslash-escapes every non-ASCII byte. So the string the scripts actually parse is a quoted, escaped stand-in for the real path, not the real path.

Reproduced locally, no network fetch involved: built a repository with the same real folder name and a genuine column-rename commit, then ran it through the tool's own code.

• Requesting the file from git using the garbled/escaped path fails outright — git reports the path does not exist.
• Requesting the exact same file using its real, correct UTF-8 name succeeds and returns the actual contents.
• Run through the tool, this mismatch caused one script to report the file as "not analyzed, wrong format" and the other to report the commit had nothing to compare — both because the .csv-suffix check is applied to the garbled string, which ends in a stray quote mark instead of .csv.

Why none of the existing dispositions fit:

• Absent — false. The file was uploaded; it was read successfully once queried by its real name.
• Inaccessible — false. The repository isn't gone or locked; it answers normally to a correctly-formed request.
• Opaque — false. It's an ordinary spreadsheet the tool proved it can read, given the correct path.

All three would be false statements about the dataset. The part that is not working is in path handling, not in the data.

Consequence if this isn't distinguished:
Without a separate label, this case gets folded into "opaque," and a later reader concludes the dataset simply isn't usable — when the data was fine and only the tool's path handling broke

[khatchad]

Triaged. The defect is real and I reproduced it against the tool's own code. Two corrections to the framing, though, and the scope is wider than reported.

The Diagnosis Holds

Confirmed end to end. A repository with a genuine column rename inside a non-ASCII folder, run through analysis/inspect_commit.py, reports:

not analyzed:
  "\346\225\260\346\215\256\351\233\206/\345\255\220/train.csv"   [format this tool does not read, see #43]
  Nothing above says whether the contents of these changed.

The file is an ordinary comma-separated table whose header changed from label to target. Nothing about the format was unreadable. Calling delimiter() on the escaped string returns None and on the real name returns ,, which is the whole mechanism: the suffix test lands on a string ending in a quote mark.

That output is also misdirecting, independent of the parsing error. It cites #43, which is about formats this tool genuinely does not read. A reader following that pointer goes to the wrong place.

The Requested Remedy Is Not The Right One

The issue asks for a fourth disposition alongside absent, inaccessible, and opaque. I would not add one. A disposition records something true about the data, and here there is nothing to record: once the path is decoded, the file opens and the column change reads out normally. A new category would make a tool defect look like a property of the corpus, which is the same confusion the issue is trying to prevent, just relocated. The reasoning about why the existing three are all false statements is correct, and the conclusion it supports is that the parsing gets repaired so no disposition is needed.

Scope Is Three Call Sites, Not One

The report covers inspect_commit.py. The same pattern appears in preservation_check.py, and its failure mode there is quieter:

• analysis/inspect_commit.py:713 builds the listing that both loops below split on a tab.
• analysis/preservation_check.py:130 reads every path at a revision.
• analysis/preservation_check.py:297 reads the changed paths, filters on is_tabular(), and drops the escaped name outright.

That last one is the one to watch. The file never reaches a skipped list and is never named in the output, so the row and column arithmetic runs over a partial file set and prints as though it had covered everything. In inspect_commit.py the file at least appears under "not analyzed"; here it vanishes.

Why Nobody Hit This Locally

My first attempt to reproduce it failed, and the reason is worth recording. This machine carries core.quotePath=false in its global git configuration, which suppresses the escaping. Git's own default is true. So the affected runs are anyone with a stock configuration, continuous integration, and a fresh checkout, while a local run looks clean. Expect "cannot reproduce" from anyone who shares that global setting, and neutralize it with an empty GIT_CONFIG_GLOBAL when checking.

Suggested Repair

Passing -z handles this and also survives a path containing a literal newline, which core.quotePath=false alone does not. One caveat for whoever picks it up: -z changes the shape of the output, it is not only a flag. Under --name-status the status and the path arrive as separate NUL-delimited records, and a detected rename emits three records rather than two, so the tab-splitting loop has to be restructured. The --name-only sites are a straight swap of the separator.

Worth a regression test with a non-ASCII path under a pinned core.quotePath=true, since the repository has none today and the default configuration is exactly what hides the bug.

Scope note: #43 stays open and is unrelated. That issue is about formats the tool genuinely cannot read, whereas every file here is one it reads fine once handed the correct name.

[khatchad]

Good find, and reproducing it from a constructed repository rather than from a failed run is the right way to establish it.

Filling in what the archived evidence already shows, so it does not have to be re-derived. The commit is sinkcup/openspd @ fcb01e8f: six CSVs under Chinese-named directories, 开关/train/0000.csv and five siblings. data/evidence/sinkcup__openspd__fcb01e8fab.txt renders those paths correctly and reports the column change for all six.

Do not read that archived file as a script that got it right. Nothing in archive_evidence.py handles paths; it shells out to inspect_commit.py, which is one of the scripts carrying the defect. The archive renders correctly because it was generated on a machine whose global git configuration carries core.quotePath=false, suppressing the escaping. Git's own default is true. So the committed evidence is correct on disk and not reproducible on a stock configuration: regenerate it there and those six files drop to "not analyzed" with no error raised. That is worth knowing before anyone regenerates the archive, and it is also why the bug went unseen here for so long.

Scope: of the archived commits in data/evidence, this is the only one whose file changes contain non-ASCII paths. One commit in the current set rather than a systematic loss, which is worth saying in the issue so a later reader does not assume a large silent gap.

On the repair, correcting what I first wrote here: this is not one line. There are three call sites across two scripts and no shared point that covers them, since preservation_check.py builds its git arguments inline at each one. Passing -c core.quotePath=false is also only a partial repair, because git still quotes a name containing a double quote or a backslash with that setting off. Passing -z covers those too, at the cost of changing the shape of the output rather than only its escaping, so the parsing loop has to be restructured.

One thing only you have: whether the repository you built for the reproduction is saved anywhere, or whether it should become a test fixture. A fixture would need its own pinned core.quotePath=true, since the default configuration on this machine is exactly what hides the failure.

No rush before Thursday. If you want it, the fix is a good first change to this repository after your talk.