[Browser plugin] assertScreenshotMatches() fails on byte differences, making the pixel tolerances unreachable

[efimdev]

Package: pestphp/pest-plugin-browser v5.0.1 (Pest 5.1.1, PHP 8.5.9, Playwright 1.62.1, Chromium, Linux/WSL2)

Summary

Page::expectScreenshot() compares the raw PNG bytes first, and throws whenever those bytes differ — regardless of what the pixel comparison says. The configured tolerances (maxDiffPixels: 300, threshold: 0.3, detectAntialiasing: true) can therefore never let a test pass. Since Chromium's PNG output is not byte-reproducible, every screenshot assertion is effectively a coin flip.

What we observed

A suite of 35 screenshot assertions, baselines committed and unchanged, run three times in a row on the same machine:

Run	Failures
1	10
2	11
3	11

Different tests failed each time. Extracting the expected/actual pair out of the generated ImageDiffView HTML and comparing them pixel by pixel (PHP GD):

Test	Size	Differing pixels	Max channel delta
range calendar	1728×1117	4 (0.0002%)	1
sold out occurrence	1728×1117	24 (0.0012%)	1
tablet / seat map	768×1024	6 (0.0008%)	1
phone / long day slot grid	375×812	5 (0.0016%)	21
phone / range calendar	375×812	2 (0.0007%)	21

Two to twenty-four pixels out of one to two million — font antialiasing, invisible to the eye and far below maxDiffPixels: 300.

The generated diff views confirm the tolerance was satisfied: they embed the 2 KB ImageDiffView::missingImage() placeholder instead of a real diff map, i.e. Playwright returned no diff — and the test still failed.

Root cause

src/Playwright/Page.php, expectScreenshot():

try {
    expect($actualImageBlob)->toMatchSnapshot();   // ← exact byte comparison
} catch (ExpectationFailedException) {
    // ... pixel comparison with maxDiffPixels / threshold / detectAntialiasing ...

    foreach ($response as $message) {
        if (isset($message['result']['diff'])) {
            // pixel comparison found a difference → throw
        }
    }

    // pixel comparison found nothing → throw anyway
    $this->createImageDiffView(..., ImageDiffView::missingImage(), $openDiff);

    throw new ExpectationFailedException('Screenshot does not match the last one. ...');
}

Both branches of the catch throw. The pixel comparison only decides which error message is produced and whether a diff image is attached; it cannot make the assertion pass. So the byte comparison is the real gate, and the documented tolerances are dead configuration.

The error message adds to the confusion: --update-snapshots is suggested for a screenshot that Playwright considers a match, which sends people re-baselining noise.

Suggested fix

Return instead of falling through when the pixel comparison reports no difference:

$matched = true;

foreach ($response as $message) {
    if (isset($message['result']['diff'])) {
        $matched = false;
        // ... createImageDiffView + throw ...
    }
}

if ($matched) {
    return; // within maxDiffPixels / threshold — the assertion holds
}

Optionally refresh the stored snapshot in that case, so the byte-level fast path keeps working on later runs.

Reproduction

1. Take any page with text and assert assertScreenshotMatches().
2. Run the test twice with an unchanged page and unchanged baseline.
3. Some runs pass, some fail with "Screenshot does not match the last one." The ImageDiffView HTML of a failing run contains the placeholder image, showing the pixel comparison found nothing.

Workaround we are using

A local assertion that captures the frame ($page->screenshot()), reads the baseline from Pest's snapshot repository, and counts differing pixels itself, allowing up to 150 changed pixels with a per-channel delta above 3. Same suite, three consecutive runs, 35/35 green — and it still catches real layout changes (it immediately surfaced two screens that had genuinely changed and whose baselines were stale).