Wait for Screenshot to Match

[htho]

Proposal

I'd like screenshot matchers to be re-executed until the screenshot matches.
Right now, one screenshot is taken and if it does not match, the test fails.

It may take a while until the browser/element is in the state where I want to take the screenshot.
It is not always useful/possible to wait for other factors that may indicate the page is in the desired state (i.e. existence of a css-class).

I'd like toMatchElementSnapshot and the others to be re-executed (within a configurable timeout) until the screenshot matches.

Prototype

I prototyped a solution where I locally changed toMatchElementSnapshot:

// @file ./node_modules/@wdio/visual-service/dist/matcher.js:97

export async function toMatchElementSnapshot(element, tag, expectedResultOrOptions, optionsOrUndefined) {
    const { expectedResult, options } = parseMatcherParams(tag, expectedResultOrOptions, optionsOrUndefined);
    const browser = getBrowserObject(await element);
    let compared;
    await browser.waitUntil(async () => {
        const result = await browser.checkElement(await element, tag, options);
        compared = compareResult(result, expectedResult || DEFAULT_EXPECTED_RESULT);
        return compared.pass;
    });
    return compared;
}

This has the desired results.

Custom-Matcher (not possible)

I tried to write a custom matcher:

// @file ./extensions/expect_toMatchElementSnapshotSoon.ts

import { WdioCheckElementMethodOptions } from '@wdio/visual-service/dist/types';
import { toMatchElementSnapshot } from '@wdio/visual-service/dist/matcher'; // not possible

export async function toMatchElementSnapshotSoon (
    element: WebdriverIO.Element,
    tag: string,
    expectedResultOrOptions?: number | ExpectWebdriverIO.PartialMatcher,
    optionsOrUndefined?: WdioCheckElementMethodOptions
) {
    let compared;
    await element.waitUntil(async () => {
        compared = await toMatchElementSnapshot(element, tag, expectedResultOrOptions, optionsOrUndefined);
        return compared.pass;
    });
    return compared;
}

But when I want to run the tests, @wdio/config:ConfigParser fails:

2024-12-17T08:52:30.261Z ERROR @wdio/config:ConfigParser: Failed loading configuration file: file:///C:/dev/wdio-visual/mocha-wdio.conf.ts: Package subpath './dist/matcher' is not defined by "exports" in C:\dev\wm-gti-uts\wdio\node_modules\@wdio\visual-service\package.json imported from C:\dev\wm-gti-uts\wdio\extensions\expect_toMatchElementSnapshotSoon.ts
Error [ERR_PACKAGE_PATH_NOT_EXPORTED]: Package subpath './dist/matcher' is not defined by "exports" in C:\dev\wm-gti-uts\wdio\node_modules\@wdio\visual-service\package.json imported from C:\dev\wm-gti-uts\wdio\extensions\expect_toMatchElementSnapshotSoon.ts
    at exportsNotFound (node:internal/modules/esm/resolve:304:10)
    at packageExportsResolve (node:internal/modules/esm/resolve:651:9)
    at packageResolve (node:internal/modules/esm/resolve:837:14)
    at moduleResolve (node:internal/modules/esm/resolve:927:18)
    at defaultResolve (node:internal/modules/esm/resolve:1169:11)
    at nextResolve (node:internal/modules/esm/hooks:866:28)
    at resolveBase (file:///C:/dev/wdio-visual/node_modules/tsx/dist/esm/index.mjs?1734425546202:2:3212)
    at resolveDirectory (file:///C:/dev/wdio-visual/node_modules/tsx/dist/esm/index.mjs?1734425546202:2:3584)
    at resolveTsPaths (file:///C:/dev/wdio-visual/node_modules/tsx/dist/esm/index.mjs?1734425546202:2:4073)
    at resolve (file:///C:/dev/wdio-visual/node_modules/tsx/dist/esm/index.mjs?1734425546202:2:4447) {
  code: 'ERR_PACKAGE_PATH_NOT_EXPORTED'
}

This is how I setup the custom matcher

import { toMatchElementSnapshotSoon } from "./extensions/expect_toMatchElementSnapshotSoon";

export const config: WebdriverIO.Config = {

// [...]

async before(): Promise<void> {
		// const {toMatchElementSnapshotSoon} = await import('./extensions/expect_toMatchElementSnapshotSoon');
		if (global.expect.expect !== undefined) { // Temporary workaround. See https://github.com/webdriverio/expect-webdriverio/issues/835
			global.expect = global.expect.expect;
		}
		expect.extend({
			toMatchElementSnapshotSoon,
		});
	},
 }

btw. before() seems not to be awaited. I initially tried the dynamic import, but neither the code in the module, nor the code
after the await is executed.

Implementation

I came to the conclusion that this feature needs to be added to @wdio/visual-service.

The wait/timeout and interval should be the same as for the matchers in expect-webdriverio.
It also should be possible to override these per call to toMatchElementSnapshot() etc.

Baseline screenshots should be taken when the test times out - if the page is not in the desired state at that point, the test will fail in the future anyway.

IMO toMatchElementSnapshot and the others can safely be changed to retry the match. I can not think of cases where tests would start to fail or to have false-positives.