Inaccurate Browser Web Vitals Measurement

[CkSned]

Brief summary

The FCP and LCP browser web vitals differ by multiple seconds when test steps change only slightly, and they show inaccurate measurements when compared to Lighthouse scores, which are multiple seconds lower.

k6 version

1.6.1

OS

macOS

Docker version and image (if applicable)

No response

Steps to reproduce the problem

1. Download the attached example.js file. This is a very basic test that opens the https://quickpizza.grafana.com URL in a browser and checks thresholds for TTFB, FCP, and LCP.
2. Run the test as is using k6 run example.js and review the output for the above web vitals.
3. Comment out line 20 of the test, and uncomment line 21, then re-run. Again, review the web vitals output.
4. Comment out line 21 of the test, and uncomment both line 20 and line 22, then re-run. Reviewing the web vitals output one final time.

example.js

Expected behaviour

In each test run, the FCP and LCP vitals should accurately reflect both what is visible to the human eye in a browser and what is measured via Lighthouse when visiting https://quickpizza.grafana.com. The expected result is a quick FCP and LCP of roughly 1s.

Actual behaviour

There are two observable problems from performing the steps above:

• Visiting a page, waiting for a load state, and performing no actions either exits the test too early for vitals to be collected, or vitals under 1s aren't displayed correctly for FCP and LCP, resulting in the vitals showing 0s.
• Performing basic actions on the page, like waiting for or counting elements, is increasing the FCP/LCP vitals above what is observed in other tools (like Lighhouse), from 0.8s to 3/4s.

Example One
When simply visiting a page and waiting for the load event, the web vitals return 0s.

     execution: local
        script: example.js
        output: -

     scenarios: (100.00%) 1 scenario, 1 max VUs, 10m30s max duration (incl. graceful stop):
              * ui: 1 iterations shared among 1 VUs (maxDuration: 10m0s, gracefulStop: 30s)



  █ THRESHOLDS 

    browser_web_vital_fcp
    ✓ 'p(75) <= 1800' p(75)=0s

    browser_web_vital_lcp
    ✓ 'p(75) <= 2500' p(75)=0s

    browser_web_vital_ttfb
    ✓ 'p(75) <= 800' p(75)=386.59ms


  █ TOTAL RESULTS 

    EXECUTION
    iteration_duration..........: avg=2s       min=2s       med=2s       max=2s       p(90)=2s       p(95)=2s      
    iterations..................: 1      0.148565/s
    vus.........................: 1      min=1      max=1
    vus_max.....................: 1      min=1      max=1

    NETWORK
    data_received...............: 0 B    0 B/s
    data_sent...................: 0 B    0 B/s

    BROWSER
    browser_data_received.......: 20 kB  3.0 kB/s
    browser_data_sent...........: 5.6 kB 825 B/s
    browser_http_req_duration...: avg=312.23ms min=127.87ms med=343.99ms max=434.55ms p(90)=400.11ms p(95)=417.33ms
    browser_http_req_failed.....: 0.00%  0 out of 5

    WEB_VITALS
    browser_web_vital_fcp.......: avg=0s       min=0s       med=0s       max=0s       p(90)=0s       p(95)=0s      
    browser_web_vital_lcp.......: avg=0s       min=0s       med=0s       max=0s       p(90)=0s       p(95)=0s      
    browser_web_vital_ttfb......: avg=386.59ms min=386.59ms med=386.59ms max=386.59ms p(90)=386.59ms p(95)=386.59ms




running (00m06.7s), 0/1 VUs, 1 complete and 0 interrupted iterations
ui   ✓ [======================================] 1 VUs  00m06.7s/10m0s  1/1 shared iters

Example Two
When visiting a page and waiting for the networkidle event, the web vitals return roughly 3/4s. This is inconsistent with the Lighthouse score (which is multiple seconds lower) and the load event test.

     execution: local
        script: example.js
        output: -

     scenarios: (100.00%) 1 scenario, 1 max VUs, 10m30s max duration (incl. graceful stop):
              * ui: 1 iterations shared among 1 VUs (maxDuration: 10m0s, gracefulStop: 30s)



  █ THRESHOLDS 

    browser_web_vital_fcp
    ✗ 'p(75) <= 1800' p(75)=3.62s

    browser_web_vital_lcp
    ✗ 'p(75) <= 2500' p(75)=3.62s

    browser_web_vital_ttfb
    ✓ 'p(75) <= 800' p(75)=382.9ms


  █ TOTAL RESULTS 

    EXECUTION
    iteration_duration..........: avg=5.55s    min=5.55s    med=5.55s    max=5.55s    p(90)=5.55s    p(95)=5.55s   
    iterations..................: 1      0.097536/s
    vus.........................: 1      min=1       max=1
    vus_max.....................: 1      min=1       max=1

    NETWORK
    data_received...............: 0 B    0 B/s
    data_sent...................: 0 B    0 B/s

    BROWSER
    browser_data_received.......: 516 kB 50 kB/s
    browser_data_sent...........: 7.2 kB 700 B/s
    browser_http_req_duration...: avg=367.21ms min=125.66ms med=338.72ms max=629.23ms p(90)=538.47ms p(95)=606.54ms
    browser_http_req_failed.....: 0.00%  0 out of 26

    WEB_VITALS
    browser_web_vital_cls.......: avg=0        min=0        med=0        max=0        p(90)=0        p(95)=0       
    browser_web_vital_fcp.......: avg=3.62s    min=3.62s    med=3.62s    max=3.62s    p(90)=3.62s    p(95)=3.62s   
    browser_web_vital_lcp.......: avg=3.62s    min=3.62s    med=3.62s    max=3.62s    p(90)=3.62s    p(95)=3.62s   
    browser_web_vital_ttfb......: avg=382.9ms  min=382.9ms  med=382.9ms  max=382.9ms  p(90)=382.9ms  p(95)=382.9ms 




running (00m10.3s), 0/1 VUs, 1 complete and 0 interrupted iterations
ui   ✓ [======================================] 1 VUs  00m10.3s/10m0s  1/1 shared iters
ERRO[0010] thresholds on metrics 'browser_web_vital_fcp, browser_web_vital_lcp' have been crossed 

Example Three
When visiting a page and waiting for the load event and for an element to become visible, web vitals will return either 0s or roughly 3/4s, giving us inconsistent results.

     execution: local
        script: example.js
        output: -

     scenarios: (100.00%) 1 scenario, 1 max VUs, 10m30s max duration (incl. graceful stop):
              * ui: 1 iterations shared among 1 VUs (maxDuration: 10m0s, gracefulStop: 30s)



  █ THRESHOLDS 

    browser_web_vital_fcp
    ✗ 'p(75) <= 1800' p(75)=3.67s

    browser_web_vital_lcp
    ✓ 'p(75) <= 2500' p(75)=0s

    browser_web_vital_ttfb
    ✓ 'p(75) <= 800' p(75)=366.09ms


  █ TOTAL RESULTS 

    EXECUTION
    iteration_duration..........: avg=4.69s    min=4.69s    med=4.69s    max=4.69s    p(90)=4.69s    p(95)=4.69s   
    iterations..................: 1      0.11368/s
    vus.........................: 1      min=1       max=1
    vus_max.....................: 1      min=1       max=1

    NETWORK
    data_received...............: 0 B    0 B/s
    data_sent...................: 0 B    0 B/s

    BROWSER
    browser_data_received.......: 515 kB 59 kB/s
    browser_data_sent...........: 7.2 kB 815 B/s
    browser_http_req_duration...: avg=487.97ms min=125.28ms med=451.33ms max=668.88ms p(90)=658.23ms p(95)=658.51ms
    browser_http_req_failed.....: 0.00%  0 out of 25

    WEB_VITALS
    browser_web_vital_fcp.......: avg=3.67s    min=3.67s    med=3.67s    max=3.67s    p(90)=3.67s    p(95)=3.67s   
    browser_web_vital_lcp.......: avg=0s       min=0s       med=0s       max=0s       p(90)=0s       p(95)=0s      
    browser_web_vital_ttfb......: avg=366.09ms min=366.09ms med=366.09ms max=366.09ms p(90)=366.09ms p(95)=366.09ms




running (00m08.8s), 0/1 VUs, 1 complete and 0 interrupted iterations
ui   ✓ [======================================] 1 VUs  00m08.8s/10m0s  1/1 shared iters