Split out of #10, which bundled two separable deliverables. The LFS half shipped (#23, #32); this is the parquet half.
Blocked, and not on engineering. The mining filter (mining/1getHFdatasets.py:20) admits a dataset only when its Hugging Face tags contain both modality:tabular and format:csv, so a parquet-native dataset never enters the corpus. A spot-check of 17 datasets drawn from data/message_refactoring_candidates.csv found zero .parquet paths in the mined (main) history. Ten of eleven checked do expose an auto-generated refs/convert/parquet ref, but that is produced by Hugging Face's conversion bot, sits outside the mined history, and contains no human refactoring commits.
So a parquet code path added today would have nothing in the corpus to run against, and no way to be tested end-to-end.
Unblocking this requires first deciding whether to re-mine with a widened format filter. That decision is not free: widening the filter changes the corpus, which changes the candidate set, which would invalidate the gold set (tests/gold_set.csv) and any classifier scored against it. It should not be made mid-stream, i.e. not before #11 and its evaluation land.
Two things this issue will need when it is unblocked:
inspect() filters to p[-1].lower().endswith(".csv") (analysis/inspect_commit.py:93), so no non-CSV path is ever reached.load_lfs_pointer opens the materialized file in text mode (open(full_path, "r")) and reads the first line, which is meaningless against parquet's binary container. Reading a parquet schema needs pyarrow.
Do not pick this up before the re-mining decision.
Split out of #10, which bundled two separable deliverables. The LFS half shipped (#23, #32); this is the parquet half.
Blocked, and not on engineering. The mining filter (
mining/1getHFdatasets.py:20) admits a dataset only when its Hugging Face tags contain bothmodality:tabularandformat:csv, so a parquet-native dataset never enters the corpus. A spot-check of 17 datasets drawn fromdata/message_refactoring_candidates.csvfound zero.parquetpaths in the mined (main) history. Ten of eleven checked do expose an auto-generatedrefs/convert/parquetref, but that is produced by Hugging Face's conversion bot, sits outside the mined history, and contains no human refactoring commits.So a parquet code path added today would have nothing in the corpus to run against, and no way to be tested end-to-end.
Unblocking this requires first deciding whether to re-mine with a widened format filter. That decision is not free: widening the filter changes the corpus, which changes the candidate set, which would invalidate the gold set (
tests/gold_set.csv) and any classifier scored against it. It should not be made mid-stream, i.e. not before #11 and its evaluation land.Two things this issue will need when it is unblocked:
inspect()filters top[-1].lower().endswith(".csv")(analysis/inspect_commit.py:93), so no non-CSV path is ever reached.load_lfs_pointeropens the materialized file in text mode (open(full_path, "r")) and reads the first line, which is meaningless against parquet's binary container. Reading a parquet schema needspyarrow.Do not pick this up before the re-mining decision.