Default Deno.bench warmups may not be high enough for some CI runners

[GJZwiers]

Recently I have been benchmarking with Deno.bench and having fun, although it seems that on some CI runners the default 1000 warmups is not sufficient. On the Ubuntu runner in Github Actions in particular there is a major difference in the measured time of whatever benchmark is run first, compared to other runners and my local machine.

For example, I am benching a JS and WASM implementation of a string distance algorithm. The results are very consistent on my local setup and the MacOS/Windows runners: JS is faster on relatively short strings while WASM is faster on long ones:

bench Myers 32bp string JavaScript ... 1000 iterations 1,035 ns/iter (700..59,100 ns/iter) ok (17ms)
bench Myers 32bp string WebAssembly ... 1000 iterations 2,726 ns/iter (2,400..8,900 ns/iter) ok (12ms)
bench Myers 1024bp string JavaScript ... 1000 iterations 425,071 ns/iter (401,300..565,900 ns/iter) ok (884ms)
bench Myers 1024bp string WebAssembly ... 1000 iterations 80,577 ns/iter (78,500..122,900 ns/iter) ok (166ms)

On the Ubuntu runner though, there is a time anomaly in the first bench, showing a much higher average ns/iter:

bench Myers 32bp string JavaScript ... 1000 iterations 10,783 ns/iter (9,300..88,499 ns/iter) ok (35ms)
bench Myers 32bp string WebAssembly ... 1000 iterations 6,350 ns/iter (3,400..60,500 ns/iter) ok (24ms)
bench Myers 1024bp string JavaScript ... 1000 iterations 673,224 ns/iter (556,596..3,331,975 ns/iter) ok (1s)
bench Myers 1024bp string WebAssembly ... 1000 iterations 152,748 ns/iter (118,299..4,364,268 ns/iter) ok (294ms)

I can actually shift the extra time in the CI to a different benchmark by having it run first:

bench test bench JavaScript ... 1000 iterations 9,218 ns/iter (1,400..96,701 ns/iter) ok (59ms)
bench Myers 32bp string JavaScript ... 1000 iterations 1,322 ns/iter (1,000..5,600 ns/iter) ok (16ms)
bench Myers 32bp string WebAssembly ... 1000 iterations 4,417 ns/iter (4,300..22,500 ns/iter) ok (33ms)
bench Myers 1024bp string JavaScript ... 1000 iterations 750,559 ns/iter (736,909..3,943,152 ns/iter) ok (1s)
bench Myers 1024bp string WebAssembly ... 1000 iterations 141,181 ns/iter (140,301..225,303 ns/iter) ok (290ms)

Using more warmups solves this problem, somewhere around 3000-5000 giving results that are more in line with my expectation:

bench Myers 32bp string JavaScript ... 1000 iterations 1,470 ns/iter (1,100..64,300 ns/iter) ok (47ms)
bench Myers 32bp string WebAssembly ... 1000 iterations 4,473 ns/iter (4,100..129,601 ns/iter) ok (17ms)
bench Myers 1024bp string JavaScript ... 1000 iterations 739,221 ns/iter (731,106..1,865,416 ns/iter) ok (1s)
bench Myers 1024bp string WebAssembly ... 1000 iterations 144,435 ns/iter (143,501..224,102 ns/iter) ok (298ms)

Don't know if anyone else has encountered something similar, but this made me wonder if the default warmups for Deno.bench should be higher, assuming more people will be using this in CI environments in the future.

[bartlomieju]

This was pointed out by @evanwashere when iterating on Deno.bench(); I went with conservative 1k iterations for both warmup and measured, but it should probably be an order of magnitude higher.