Protect: Add E2E Tests

[nateweller]

Proposed changes:

• Introduces projects/plugins/protect/tests/e2e, following the README with minor changes to the initial test style based on Explore usage of @wordpress/e2e-test-utils-playwright for E2E tests #40750.
• Adds the Protect E2E tests to the tests matrix.
• Minor fixes/updates to the E2E commons README.

Other information:

• Have you written new tests for your changes, if applicable?
• Have you checked the E2E test CI results, and verified that your changes do not break them?
• Have you tested your changes on WordPress.com, if applicable (if so, you'll see a generated comment below with a script to run)?

Jetpack product discussion

N/A

Does this pull request change what data or activity we track or use?

No

Testing instructions:

• See the E2E README to run tests locally. Alternatively, you can review the logs from the automated test run on this PR.

[github-actions]

Are you an Automattician? Please test your changes on all WordPress.com environments to help mitigate accidental explosions.

• To test on WoA, go to the Plugins menu on a WoA dev site. Click on the "Upload" button and follow the upgrade flow to be able to upload, install, and activate the Jetpack Beta plugin. Once the plugin is active, go to Jetpack > Jetpack Beta, select your plugin (Jetpack or WordPress.com Site Helper), and enable the add/protect/e2e-tests branch.
• To test on Simple, run the following command on your sandbox:

bin/jetpack-downloader test jetpack add/protect/e2e-tests

bin/jetpack-downloader test jetpack-mu-wpcom-plugin add/protect/e2e-tests

Interested in more tips and information?

• In your local development environment, use the jetpack rsync command to sync your changes to a WoA dev blog.
• Read more about our development workflow here: PCYsg-eg0-p2
• Figure out when your changes will be shipped to customers here: PCYsg-eg5-p2

[github-actions]

Thank you for your PR!

When contributing to Jetpack, we have a few suggestions that can help us test and review your patch:

• ✅ Include a description of your PR changes.
  
• ✅ Add a "[Status]" label (In Progress, Needs Review, ...).
  
• ✅ Add a "[Type]" label (Bug, Enhancement, Janitorial, Task).
  
• ✅ Add testing instructions.
  
• ✅ Specify whether this PR includes any changes to data or privacy.
  
• ✅ Add changelog entries to affected projects
  

This comment will be updated as you work on your PR and make changes. If you think that some of those checks are not needed for your PR, please explain why you think so. Thanks for cooperation 🤖

---

Follow this PR Review Process:

1. Ensure all required checks appearing at the bottom of this PR are passing.
2. Make sure to test your changes on all platforms that it applies to. You're responsible for the quality of the code you ship.
3. You can use GitHub's Reviewers functionality to request a review.
4. When it's reviewed and merged, you will be pinged in Slack to deploy the changes to WordPress.com simple once the build is done.

If you have questions about anything, reach out in #jetpack-developers for guidance!

---

Protect plugin:

No scheduled milestone found for this plugin.

If you have any questions about the release process, please ask in the #jetpack-releases channel on Slack.

[jp-launch-control]

Code Coverage Summary

This PR did not change code coverage!

That could be good or bad, depending on the situation. Everything covered before, and still is? Great! Nothing was covered before? Not so great. 🤷

Full summary · PHP report · JS report

[dkmyta]

I don't appear to be able to get the tests running locally and I don't see a line item in the PR checks for Jetpack Protect like for example Jetpack Boost has - does it make sense to separate them out early like that? If not, which of the E2E checks can I reference to verify that the tests are running/logging the results?

[nateweller]

Added the config needed to get the tests running in GitHub, but they aren't passing 🤔

Works fine locally, getting some 404 errors when running in the cloud:

Running 1 test using 1 worker

Warning: Plugin 'jetpack' is already active.
Failed to load resource: the server responded with a status of 404 (Not Found)
Failed to load resource: the server responded with a status of 404 (Not Found)
Failed to load resource: the server responded with a status of 404 (Not Found)
Failed to load resource: the server responded with a status of 404 (Not Found)
The above error occurred in the <v> component:

Still looking into it...

[nateweller]

@dkmyta Okay, all good for re-review here 👍

You should now see the Protect E2E tests running in GitHub along with the rest.

Re: Running locally, here are some random tips from getting it going recently myself:

// make sure the env is good
nvm use
rm -rf node_modules && pnpm i
jetpack install -r
cd projects/plugins/protect/tests/e2e

// if projects/plugins/protect/tests/e2e/config/local.cjs doesn't exist locally yet, make sure to decrypt the secret
CONFIG_KEY=YOUR_KEY pnpm config:decrypt

// continue making sure the env is good
pnpm env:reset        // or pnpm env:up
pnpm tunnel:reset     // or pnpm tunnel:up

// run the tests with live preview in browser
pnpm test:run --headed