First column in benchmark is faster than the rest

[birkskyum]

When I ran the benchmark today, it seemed like our current main is performing worse than version 2.3, which was release yesterday. Only two tests affected are Layout and WorkerTrasnfer, and only change since 2.3 are made by dependabot.

Left column is 2.3 ( 91567e9 ), middle column is main right now at fbaebce

[HarelM]

Add another version, there's a problem in the code...

[wipfli]

Workertransfer has always a much faster first run. That is a bug I think..

[wipfli]

In case we get dedicated machines to run CI on then it would be great to have the benchmarks there @hy9be. They don't run in CI at the moment because GitHub runners are too random I think...

[birkskyum]

Okay, we definitely have to fix this, because now I even doubt if the columns represents the commit hash above. Dedicated machines would be a great step. @wipfli I also noticed you mentioned the xvfb in another context related to unstable github runners, and I thought that maybe we could swap that with X11+dummy or wayland in hopes it would be more stable window server,

[birkskyum]

I've checked out main and are running benchmarks against it so we can have a better idea about whats going on.

[birkskyum]

So when I'm running against main it fortunately shows the same, so it's consistent so far.

[birkskyum]

This is where I checked out 2.3 and ran the benchmarks, so the first and third column should be the same, and the one in the middle is main, which is a bit ahead.

[birkskyum]

No matter what versions I pick it's always the same - the first is faster in Layout and WorkerTransfer than the rest.

[birkskyum]

Basically this points towards #982

[HarelM]

While I think the original issue I opened is still relevant, the issue at hand seems to be related to the browser getting warmed-up and might be solved simply by running the first version twice and reporting only the second run...