[Visualization] Critical Difference

[bruAristimunha]

Include this type of plot:

[davisethan]

Commenting what I hope is helpful context for anyone interested in #1127 and related work. The critical difference diagram was proposed by Demšar ¹. His approach is the most commonly used today for comparing two or more classifiers over multiple datasets, though, more recently Benavoli et al. ² (coauthor Demšar) have made a widely cited case for Bayesian hypothesis testing.

Extension: Chapters 1-9 of Moritz ³ summarizes lessons from ML benchmarking on fixed shared datasets (the "ImageNet era"). Moritz notes what Duda et al. ⁴ call training on the testing data: when a classifier undergoes a long series of refinements guided by the results of repeated testing on the same data. As a countermeasure, ch. 5 presents a differentially private holdout method that keeps an evaluation statistically valid under far more adaptive reuse.

Footnotes

1. Demšar, Janez. "Statistical Comparisons of Classifiers over Multiple Data Sets." Journal of Machine Learning Research, vol. 7, no. 1, 2006, pp. 1–30. ↩

2. Benavoli, Alessio, et al. "Time for a Change: A Tutorial for Comparing Multiple Classifiers Through Bayesian Analysis." Journal of Machine Learning Research, vol. 18, no. 77, 2017, pp. 1–36. ↩

3. Hardt, Moritz. The Emerging Science of Machine Learning Benchmarks. Princeton University Press, 2026. ↩

4. Duda, Richard O., et al. Pattern Classification. 2nd ed., Wiley-Interscience, 2001. ↩