Improve results of benchmark-web-vitals by not reusing browser process (and its cache) between requests
#179
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
westonruter
force-pushed
the
fix/benchmark-web-vitals-requests
branch
from
a274ecb to
919cbaf
Compare
swissspidy
approved these changes
Apr 1, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I discovered that
benchmark-web-vitalswas reusing the same browser instance across all requests. This meant that assets cached for the initial request would then be available for subsequent requests. When testing the performance impact of preloading an image, for example, this would mean that if you first tested the URL with the image preloaded and then tested the page without the preloading, you'd see barely any an improvement (if any). But if you switched the order around, then you'd see a big improvement if the URL with the image preloading happened after the request without the preloading. So this PR makes sure that a fresh browser instance is used for every request to prevent one request from impacting the performance of another request. As an additional safeguard, it also explicitly disables the cache for the launched browser.Test Setup
I used LocalWP for my test and I added an
.htaccessfile that enabled far-future expires for image files. Since typically local development environments are configured to serve assets withCache-Control: no-cache, the issue may not have apparent when compared with benchmarking a production environment. In #178 I'm benchmarking production URLs with a single request to a URL with Optimization Detective disabled followed by a single request to Optimization enabled, and as can be seen below, the effect of the browser cache is very apparent.I have configured a post to contain six images, five of which are
IMGelements in columns (where the first getsfetchpriority=highby WordPress core) and sixth one which is a CSSbackground-imageof a Group block which is the actual LCP element:Block Markup
I have the Twenty Twenty-Five theme active as well as the following plugins:
Note I've added plugins that inline all stylesheets so that the differences in image prioritization are more pronounced.
I have two files containing the URLs for this test post with Optimization Detective disabled and enabled, and then enabled and disabled:
od-disabled-then-od-enabled.txt:od-enabled-then-od-disabled.txt:Diff of HTML with Optimization Detective disabled and enabled
In the following tests I've added Fast 4G network emulation because without it, requests to load images from
localhosthappen extremely fast and any effect of image prioritization optimizations may be not apparent.Before
With 10 iterations
Output:
Notice how the LCP-TTFB is very small, even though it's on "Fast 4G". Notice also how in the first call to
benchmark-web-vitalswhen the OD disabled URL goes before the OD enabled URL, the LCP-TTFB is much greater that when the OD enabled URL is benchmarked before the OD disabled URL. Often, whichever URL is requested first will be the one that appears worse because it is the one that is hit with a cold browser cache. This is made more stark when reducing the iterations.With 1 iteration over 10 repetitions
Output, where the cells in the first column alternate between OD Disabled and OD Enabled:
Notice how the one request in the first column is always extremely slow (10x) compared to the second column, regardless of the optimizations being applied by Optimization Detective. This is because the same browser is being used between the two requests for OD disabled and OD disabled, meaning the second request will be able to reuse cached resources and always be faster.
After
With 10 iterations
Output:
Notice now how the OD enabled version is now significantly faster than the OD disabled version (and by the same margin), regardless of whether the OD enabled URL is queried first or the OD disabled URL is queried first.
With 1 iteration over 10 repetitions
Output, where the cells in the first column alternate between OD Disabled and OD Enabled:
Notice how the better LCP-TTFB value switches after each row, where the better value corresponds to when the URL had OD enabled. This is in contrast with the before test above where the first column (and the first request) is always much slower than the second column (the second request) regardless of whether the URL had OD enabled or disabled.
Additional Changes
This also will by default make a request to a URL prior to making the first request to collect metric. This is to ensure that the DNS lookups have been cached in the operating system so that the TTFB for the initial request won't be slower than the rest. The
--skip-network-primingoption can be used to disable this.Additionally, there is now a
--pause-duration <milliseconds>option which can be used to add a delay of the provided milliseconds between each request. This is to give the server a chance to catch its breath, preventing the CPU from getting increasingly taxed which would progressively reflect poorly on TTFB. It's also provided as an option to be a good netizen when benchmarking a site in the field since therndquery parameter will usually bust page caches.