What Happened
A toMatchSnapshot() assertion inside a test marked flaky() never reports a mismatch. The retry resolves a different snapshot file than the first attempt, so the baseline it was meant to compare against is quietly left behind.
Locally, with a baseline that does not match the current value:
- First run: attempt 1 compares against the baseline and fails. Attempt 2 looks for
..._2.snap, which does not exist, so it is created from the current value. The test ends as incomplete and the suite is green.
- Every run after that: attempt 1 still fails against the baseline, attempt 2 now matches the
..._2.snap it created earlier, and the test passes. The original baseline is never honored again.
On CI the failure is not silent, but it points at the wrong thing. Running the same test with CI=true:
Snapshot is missing at [tests/.pest/snapshots/Feature/FlakySnapshotTest/it_compares_a_snapshot_inside_a_flaky_test__3.snap].
Run Pest with --update-snapshots to create it.
The file it names is one the suite never created, and the value that actually differs is never mentioned.
The combination is not exotic: screenshots are both the assertion people mark flaky() and the assertion whose regression must not be swallowed. That is how we ran into it, on a visual regression suite where four baselines had been drifting for weeks behind a green run.
How to Reproduce
Fresh app, no application code involved:
laravel new pest-flaky-repro --react --pest --database=sqlite --no-authentication --no-node --no-boost
cd pest-flaky-repro
tests/Feature/FlakySnapshotTest.php:
<?php
declare(strict_types=1);
it('compares a snapshot inside a flaky test', function (): void {
expect('after')->toMatchSnapshot();
})->flaky(3);Write a baseline that does not match, so the very first attempt fails:
mkdir -p tests/.pest/snapshots/Feature/FlakySnapshotTest
printf 'before' > tests/.pest/snapshots/Feature/FlakySnapshotTest/it_compares_a_snapshot_inside_a_flaky_test.snap
Then:
vendor/bin/pest tests/Feature/FlakySnapshotTest.php # 1 incomplete, suite green
ls tests/.pest/snapshots/Feature/FlakySnapshotTest/ # a __2.snap now exists, holding "after"
vendor/bin/pest tests/Feature/FlakySnapshotTest.php # 1 passed
Expected: the test fails on all three attempts, since after never equals the before baseline.
Control: removing ->flaky(3) from the same test, with the same baseline, fails as it should.
For the CI path, delete the generated __2.snap first, then:
CI=true vendor/bin/pest tests/Feature/FlakySnapshotTest.php
Sample Repository
Reproduced on a clean install as described above, so no repository is needed. Happy to publish one if that helps.
It also reproduces outside Laravel entirely, on a bare composer.json whose only dependency is pestphp/pest, with the test in tests/ and the baseline in tests/.pest/snapshots/FlakySnapshotTest/. Same outcome: 1 incomplete, Snapshot created at [..._2.snap]. So nothing in the Laravel integration is involved.
Pest Version
5.1.1, and 5.0.4 behaves identically, so this is not a 5.1 regression.
PHP Version
8.5.8, and 8.4.23 behaves identically.
Operation System
macOS
Notes
The cause looks like a missing reset. SnapshotRepository::$expectationsCounter is static and keyed by test file plus test description, and it is incremented once per snapshot expectation so that several toMatchSnapshot() calls in one test get distinct files. Testable::__callClosure() replays the closure in the same process on a flaky retry without clearing that key, so the counter carries over and attempt 2 asks for __2.
The replay already restores object properties, mock objects and the output buffer before running again; the counter for the current key looks like it belongs to that same cleanup.
One thing that might look like it covers this case but does not: the guard that rethrows instead of retrying when __snapshotChanges is not empty. It protects the case where the first attempt creates a snapshot. Here the first attempt only compares, so the retry proceeds.
What Happened
A
toMatchSnapshot()assertion inside a test markedflaky()never reports a mismatch. The retry resolves a different snapshot file than the first attempt, so the baseline it was meant to compare against is quietly left behind.Locally, with a baseline that does not match the current value:
..._2.snap, which does not exist, so it is created from the current value. The test ends asincompleteand the suite is green...._2.snapit created earlier, and the test passes. The original baseline is never honored again.On CI the failure is not silent, but it points at the wrong thing. Running the same test with
CI=true:The file it names is one the suite never created, and the value that actually differs is never mentioned.
The combination is not exotic: screenshots are both the assertion people mark
flaky()and the assertion whose regression must not be swallowed. That is how we ran into it, on a visual regression suite where four baselines had been drifting for weeks behind a green run.How to Reproduce
Fresh app, no application code involved:
laravel new pest-flaky-repro --react --pest --database=sqlite --no-authentication --no-node --no-boost cd pest-flaky-reprotests/Feature/FlakySnapshotTest.php:Write a baseline that does not match, so the very first attempt fails:
Then:
Expected: the test fails on all three attempts, since
afternever equals thebeforebaseline.Control: removing
->flaky(3)from the same test, with the same baseline, fails as it should.For the CI path, delete the generated
__2.snapfirst, then:Sample Repository
Reproduced on a clean install as described above, so no repository is needed. Happy to publish one if that helps.
It also reproduces outside Laravel entirely, on a bare
composer.jsonwhose only dependency ispestphp/pest, with the test intests/and the baseline intests/.pest/snapshots/FlakySnapshotTest/. Same outcome:1 incomplete,Snapshot created at [..._2.snap]. So nothing in the Laravel integration is involved.Pest Version
5.1.1, and 5.0.4 behaves identically, so this is not a 5.1 regression.
PHP Version
8.5.8, and 8.4.23 behaves identically.
Operation System
macOS
Notes
The cause looks like a missing reset.
SnapshotRepository::$expectationsCounteris static and keyed by test file plus test description, and it is incremented once per snapshot expectation so that severaltoMatchSnapshot()calls in one test get distinct files.Testable::__callClosure()replays the closure in the same process on a flaky retry without clearing that key, so the counter carries over and attempt 2 asks for__2.The replay already restores object properties, mock objects and the output buffer before running again; the counter for the current key looks like it belongs to that same cleanup.
One thing that might look like it covers this case but does not: the guard that rethrows instead of retrying when
__snapshotChangesis not empty. It protects the case where the first attempt creates a snapshot. Here the first attempt only compares, so the retry proceeds.