Summary
HealthBench reports bootstrap_std for every aggregated metric, but _compute_clipped_stats(..., "bootstrap_std") draws bootstrap samples from NumPy's process-global RNG without a fixed/local seed.
As a result, the same set of per-example scores can produce different reported uncertainty values across otherwise identical runs.
Current behavior
bootstrap_samples = [np.random.choice(values, len(values)) for _ in range(1000)]
The output therefore depends on whatever global NumPy RNG state happens to exist when aggregation runs.
Impact
HealthBench result JSON can change even when the sampled model outputs and grader results are identical. This makes reported uncertainty harder to reproduce, compare, and regression-test.
Proposed resolution
Use a local deterministic NumPy generator for bootstrap resampling so aggregation does not depend on or mutate process-global RNG state.
Add a regression that changes the global NumPy seed between calls and verifies the bootstrap standard deviation for the same values is unchanged.
Summary
HealthBench reports
bootstrap_stdfor every aggregated metric, but_compute_clipped_stats(..., "bootstrap_std")draws bootstrap samples from NumPy's process-global RNG without a fixed/local seed.As a result, the same set of per-example scores can produce different reported uncertainty values across otherwise identical runs.
Current behavior
The output therefore depends on whatever global NumPy RNG state happens to exist when aggregation runs.
Impact
HealthBench result JSON can change even when the sampled model outputs and grader results are identical. This makes reported uncertainty harder to reproduce, compare, and regression-test.
Proposed resolution
Use a local deterministic NumPy generator for bootstrap resampling so aggregation does not depend on or mutate process-global RNG state.
Add a regression that changes the global NumPy seed between calls and verifies the bootstrap standard deviation for the same values is unchanged.