[Bug]: Playwright Does Not Use "New Headless" Locally When Using chromium Channel

[tsherkness]

Version

1.49.1

Steps to reproduce

Set Playwright config to use Desktop Chrome with channel chromium, as shown in the example here: #33566

Set a baseline snapshot for visual comparison from the local machine.

Run the test in Azure/TFS.

Expected behavior

The visual comparison test should pass, expecting that both from the local and remote that the same "new" headless version was used to capture and test the screenshot.

Actual behavior

The test fails when run in Azure/TFS.

Local machine does not appear to use "new" headless version when using the chromium channel.

If I compare the local snapshot with how the headed browser looks, it is slightly different in that the element spacing is off.

If I compare the snapshot taken in Azure/TFS with how the headed browser looks, they are identical.

Additional context

I tried capturing screenshots using the headed browser locally and they still don't look correct compared to the screenshots captured in TFS, so I may not be 100% sure why the screenshots look different in TFS compared to local. This is only the case when I use the chromium channel. My visual comparison tests work fine if I do not use the chromium channel.

I always pull the latest from my repository and install playwright with its browser dependencies whenever my tests run in TFS.

Environment

System:
    OS: Windows 11 10.0.22631
    CPU: (22) x64 Intel(R) Core(TM) Ultra 7 155H
    Memory: 16.25 GB / 31.46 GB
  Binaries:
    Node: 22.12.0 - C:\Program Files\nodejs\node.EXE
    npm: 11.0.0 - C:\Program Files\nodejs\npm.CMD
  IDEs:
    VSCode: 1.96.2 - C:\Program Files\Microsoft VS Code\bin\code.CMD
  npmPackages:
    @playwright/test: ^1.49.1 => 1.49.1

[yury-s]

Do you take local screenshot on Windows and compare them with screenshot on Linux in Azure/TFS? Also can you share a repro where this happens?

[tsherkness]

Everything in my version of TFS is Windows based. The agent versions are on 3.239.1, if that helps at all.

I'm not sure how to share a repro of the issue since my version of TFS is on prem.

[yury-s]

Set a baseline snapshot for visual comparison from the local machine.

In general, for consistent screenshots, run tests in the same environment where the baseline screenshots were generated (see the warning). Browser rendering is quite sensitive to the host environment, so in your case we'd recommend you generate screenshots on the same CI machine where you run the tests.

I'm not sure how to share a repro of the issue since my version of TFS is on prem.

If you could share a minimal project with a test that produces different results and share two screenshots that you see locally and on CI, we could see if there is anything that can be done. Most likely it's just a difference in the browser rendering mentioned above, which is still an open issue.

[tsherkness]

Thank you for pointing out the warning in the documentation, I must have glanced over that before. I didn't have an issue until I switched the channel to chromium so I wasn't expecting differences to show up like this.

I'll try basing the screenshots from the same environment. I will also see about getting a working example that reproduces the issue.