Subsetting the SPEC CPU2006 benchmark suite by Phansalkar et al. 2007 shows an example of benchmark subsetting. Table 1 shows features used or each benchmark. Figure 5 shows PCA. Figure 6 shows a dendrogram of hierarchical clsutering.
BenchSubset: A framework for selecting benchmark subsets based on consensus clustering by Zhan et al. 2021 synthesizes prior subsetting works to create a general best-practice.
Producing Wrong Data Without Doing Anything Obviously Wrong! by Mytkowicz e tal. 2009. Incredible paper.
Reproducible benchmarking in Linux-based environments from Julia developers
You may need to use Setuid demystified by Chen et al. 2002 to safely implement benchmark procedures at a user-level.
Stepping Towards Noiseless Linux Environment by Akkan, Lang, and Liebrock 2012
How to not lie with statistics; The correct way to summarize benchmark results by Flemming and Wallace 1986 explains why arithmetic mean of speedup ratios can be misleading and change based on what one is normalizing to (Tables III and IV). It shows how geometric mean is robust to rescaling (Tables V and VI).
Characterizing computer performance with a single number by Smith 1988 argues for harmonic mean. In my view, geometric mean is best for speedup ratios whereas harmonic mean is best for combining rates.
The Harmonic or Geometric Mean: Does it Really Matter? by Citron, Huraney, and Gnadrey 2006 documents the War of the Means.
SoK: Benchmarking Flaws in Systems Security by Kouwe et al. 2019 samples literature to find that most empirical performance claims do not include any measure of uncertainty (B4) and also count the number of claims that use the wrong mean (B5).
War of the Benchmark Means: Time for a Truce by Mashey 2004. While this discusses the war of the means, I brought it up to show that program runtimes are right-skewed and possibly log-normal. Errors can be more accurately estimated from this.
In probability theory, a probability distribution is infinitely divisible (Wikipedia) if it can be expressed as the probability distribution of the sum of an arbitrary number of independent and identically distributed (i.i.d.) random variables. The characteristic function of any infinitely divisible distribution is then called an infinitely divisible characteristic function.
What distribution do you think program runtimes are (on Sam's Bluesky)?
Uncertainty quantification (Wikipedia) is the science of quantitative characterization and estimation of uncertainties in both computational and real world applications.
In probability theory and statistics, the gamma distribution (Wikipdia) is a versatile two-parameter family of continuous probability distributions.
Statistical significance is logically independent of clinical relevance according to Figure 2 of Confidence Interval or P-Value? by Prel et al. 2009.