Add Playwright tests

[bgrgicak]

Descriptions

This PR is a prototype of Playwright tests for the Playground tester and can be used as inspiration for adding E2E tests to the Playground tester in the future.

The code in this PR hasn't been cleaned up and it was used to generate one-time compatibility reports where we compared how Playground performed in December 2023 compared to December 2024.

Things to do

• install plugin dependencies
• check why large plugins like jetpack timeout after 30s instead of loading
• find a solution for plugins that don't work with a year old version of WordPress (example plugin) - these items are are excluded from the report
• run tests
• generate report

Testing instructions

• for this to work you need to run two versions of Playground locally using npm run dev.
• On port 5400 run a old version of Playground
• On port 5401 run the current version (manually change the port from 5400 to 5401 in the code)
• You can do this using git worktrees
• after running all tests use ./scripts/update-error-stats-data.sh to parse Playwright reports
• to generate a report page, run ./scripts/generate-reports.sh

[bgrgicak]

The reports are generated, so this PR can be closed.

Playwright tests were slower from Playground CLI, but they can test complex scenarios, so we should consider adding Playwright tests to the tester in the future.

[adamziel]

redirect to the correct URL

Wouldn't the plugin-issued redirect be the "correct" result? As in, the expected one for that plugin? Are we collecting any errors from that page?

[adamziel]

Also the debug.log errors may be significant. And whether we see the error reporting modal or any errors in devtools during this entire process. And whether the plugin remains active after the installation. All of those together tell us about compatibility. Also – what other factors do you think should we consider here before we declare the plugin as operational in Playground?

[bgrgicak]

Also – what other factors do you think should we consider here before we declare the plugin as operational in Playground?

This is a question I'm trying to answer right now in the new tester, and I'm not sure. Plugins can significantly change WordPress, so it's hard to determine what must work without it being plugin specific.
At a minimum, I think that we should be able to load wp-admin, open a post edit page, and save options.
All these operations shouldn't produce Playground crashes.
Ideally, there shouldn't be any PHP fatals, but some plugins have bugs, so these errors require manual review.

[bgrgicak]

Ultimately, I think it's important to have the same tests run in an old and a new version of Playgoround to demonstrate that we reduced the error rate.

[bgrgicak]

Wouldn't the plugin-issued redirect be the "correct" result? As in, the expected one for that plugin? Are we collecting any errors from that page?

This part of the test checks if the plugin is active by going to the /wp-admin/plugins.php page.

Sometimes a plugin like WooCommerce will redirect to their onboarding on the first /wp-admin/ requests after the plugin was installed.
This behavior breaks the test because it needs to be on the Plugins page for the test to work.

Ideally, this should be tested with unit tests by calling is_plugin_active from PHP, but we use e2e tests.

[JanJakes]

This part of the test checks if the plugin is active by going to the /wp-admin/plugins.php page.

This makes sense to me. We have no way of automatically going through the onboarding pages of different plugins. At the moment, we're focusing on testing the plugin activation, which doesn't necessarily mean that the plugin works, but it is a strong indication, because activation usually executes migrations, populates data, etc. Testing that the plugin successfully activates and doesn't break basic WordPress functionality seems like a reasonable approach to me.

[bgrgicak]

Thanks for your feedback! I will run the tests as-is today, just to get 2025 stats.

After that, I can experiment with adding more tests and manually review the results.
The question about what it means for a plugin and theme to work is on top of my mind now as I'm working on the Playground Tester, and this is a good way of exploring different approaches, as it doesn't require me to rerun all tests for all WP.org plugins and themes.