Add commands to analyze URLs from HTTP Archive for Optimization Detective's effectiveness

[westonruter]

Query HTTP Archive for the URLs of pages which have Image Prioritizer active (there are currently 17k+) using this sample BigQuery query. Output these URLs to a file urls-with-image-prioritizer-on-2025-03-01.csv. A survey script in wpp-research will take each URL and load four versions in Puppeteer:

1. Mobile device with Optimization Detective disabled
2. Mobile device with Optimization Detective enabled
3. Desktop device with Optimization Detective disabled
4. Desktop device with Optimization Detective enabled

Accessing pages with Optimization Detective disabled is achieved by adding ?optimization_detective_disabled=1, which is a feature of the plugin used for comparing optimized with non-optimized.

Metrics for each page load are gathered and stored for later summarization.

To obtain the initial data, run the following command:

npm run research -- survey-optimization-detective-effectiveness \
   -f urls-with-image-prioritizer-on-2025-03-01.csv --parallel=10 -o output/image-prioritizer-2025-03-01

The results are output to the output/image-prioritizer-2025-03-01 subdirectory. There is a separate subdirectory created for each URL analyzed, with the directory name being an MD5 hash of the URL (with additional intermediary directories to avoid excessive directories). For example:

output/image-prioritizer-2025-03-01
├── 4a
│   └── d5
│       └── 4ad5d4dce334fc92c0e1f81e0a5278ec
│           ├── desktop
│           │   ├── optimized
│           │   │   ├── content.html
│           │   │   ├── headers.txt
│           │   │   └── results.json
│           │   ├── original
│           │   │   ├── content.html
│           │   │   ├── headers.txt
│           │   │   └── results.json
│           │   └── results-diff.json
│           ├── mobile
│           │   ├── optimized
│           │   │   ├── content.html
│           │   │   ├── headers.txt
│           │   │   └── results.json
│           │   ├── original
│           │   │   ├── content.html
│           │   │   ├── headers.txt
│           │   │   └── results.json
│           │   └── results-diff.json
│           ├── url.txt
│           └── version.txt
│...
├── result-manifest-20250321T025313913Z.tsv

The optimized directories contain the data for the page with Optimization Detective enabled, and the original directories contain the data for when it is disabled. The headers.txt files contain the HTTP response headers and the content.html contains the HTML response body. The metrics are located in results.json, which looks like the following:

{
  "device": {
    "userAgent": "Mozilla/5.0 (Linux; Android 11; moto g power (2022)) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Mobile Safari/537.36",
    "viewport": {
      "isMobile": true,
      "width": 412,
      "height": 823,
      "isLandscape": false,
      "deviceScaleFactor": 1.75,
      "hasTouch": true
    }
  },
  "metrics": {
    "TTFB": {
      "value": 136.39999997615814
    },
    "LCP": {
      "value": 240,
      "url": "http://localhost:8888/wp-content/uploads/2024/06/bison1-1024x668.jpg",
      "element": {
        "tagName": "DIV",
        "attributes": {
          "class": "wp-block-cover__image-background wp-image-13 size-large has-parallax",
          "style": "background-position:50% 50%;background-image:url(http://localhost:8888/wp-content/uploads/2024/06/bison1-1024x668.jpg)"
        },
        "metaAttributes": {
          "xpath": "/HTML/BODY/DIV[@class='wp-site-blocks']/*[2][self::MAIN]/*[1][self::DIV]/*[3][self::DIV]/*[1][self::DIV]/*[2][self::DIV]"
        }
      },
      "initiatorType": "link",
      "preloadedByOD": true
    },
    "LCP-TTFB": {
      "value": 103.60000002384186
    }
  },
  "pluginVersions": {
    "optimization-detective": "optimization-detective 1.0.0-beta3",
    "image-prioritizer": "image-prioritizer 1.0.0-beta2"
  },
  "odPreloadLinks": [
    {
      "data-od-added-tag": "",
      "rel": "preload",
      "fetchpriority": "high",
      "as": "image",
      "href": "http://localhost:8888/wp-content/uploads/2024/06/bison1-1024x668.jpg",
      "media": "screen and (width <= 480px)"
    },
    {
      "data-od-added-tag": "",
      "rel": "preload",
      "fetchpriority": "high",
      "as": "image",
      "href": "http://localhost:8888/wp-content/uploads/2024/06/bison1-1024x668.jpg",
      "media": "screen and (782px < width)"
    }
  ],
  "odTagsWithXpathAttrs": 1,
  "images": {
    "imgCount": 0,
    "lazyImgCount": 0,
    "lazyImgInsideViewportCount": 0,
    "jsLazyLoadedImgCount": 0,
    "fetchpriorityHighAttrImages": {
      "insideViewportCount": 0,
      "outsideViewportCount": 0
    }
  }
}

Then to summarize the results, run the following command:

npm run research -- summarize-optimization-detective-effectiveness -o output/image-prioritizer-2025-03-01

This will include pass/fail data for:

1. Whether the LCP image was prioritized:
   1. When Optimization Detective is disabled, this is whether the LCP element with an image is an IMG and it has fetchpriority=high. If the LCP element has an image but it is a background-image, then this is always considered a failure since WordPress Core does not currently prioritize loading background images.
   2. When Optimization Detective is enabled, this is whether or not there is a preload link added by OD for the LCP image (whether IMG or background-image).
2. Whether there is a lazy-loaded IMG in the initial viewport.
3. Whether there is an IMG with fetchpriority=high outside the viewport.

It also will also report the diffs for the TTFB, LCP, and LCP-TTFB metrics between the original and optimized versions of the pages as both the average and median of all values.

[westonruter]

• Add network throttling emulation for mobile and desktop which match what Lighthouse uses.

• https://developers.google.com/speed/docs/insights/v5/about#what-device-and-network-conditions-does-lighthouse-use-to-simulate-a-page-load
• https://github.com/GoogleChrome/lighthouse/blob/main/docs/throttling.md#the-mobile-network-throttling-preset
• https://github.com/GoogleChrome/lighthouse/blob/main/core/config/constants.js
• https://github.com/GoogleChrome/lighthouse/blob/7422df63662a0f926d3ac9e696d1db1a108f87ec/core/config/desktop-config.js#L15
• https://github.com/GoogleChrome/lighthouse/tree/7422df63662a0f926d3ac9e696d1db1a108f87ec/core/config
• https://github.com/GoogleChrome/lighthouse/blob/7422df63662a0f926d3ac9e696d1db1a108f87ec/core/config/lr-mobile-config.js
• https://github.com/GoogleChrome/lighthouse/blob/7422df63662a0f926d3ac9e696d1db1a108f87ec/core/config/lr-desktop-config.js

In Lantern.Simulation.Constants.throttling from @paulirish/trace_engine/models/trace/lantern/lantern.js (doc):

PSI: Network throttling: 40 ms TCP RTT, 10,240 kb/s throughput (Simulated)

WPT: