During the PriDe reproduction (see examples/pride_reproduction.md, step 3), pride_from_artifacts.py's duplicate-question_id guard caught 26 duplicate questions in MMLU's official test split (identical question text, subject, and answer; two content-hash IDs colliding on genuinely identical content). Without that downstream guard, evaluate_run.py and every metric would have silently double-counted them.
The detection shouldn't depend on one script's defensive check. prepare_data.py / the benchmark loading path should warn (not error, not silently pass) when a prepared benchmark CSV contains duplicate question_ids, reporting the count and example IDs, and optionally offer a --dedupe flag that drops all copies of colliding IDs with a sidecar accounting file (matching the permutation-filter sidecar pattern).
During the PriDe reproduction (see examples/pride_reproduction.md, step 3), pride_from_artifacts.py's duplicate-question_id guard caught 26 duplicate questions in MMLU's official test split (identical question text, subject, and answer; two content-hash IDs colliding on genuinely identical content). Without that downstream guard, evaluate_run.py and every metric would have silently double-counted them.
The detection shouldn't depend on one script's defensive check. prepare_data.py / the benchmark loading path should warn (not error, not silently pass) when a prepared benchmark CSV contains duplicate question_ids, reporting the count and example IDs, and optionally offer a --dedupe flag that drops all copies of colliding IDs with a sidecar accounting file (matching the permutation-filter sidecar pattern).