From ec4ade018a8c45d5d7be6eb84e036f4f331a7ba9 Mon Sep 17 00:00:00 2001 From: Allie Mims Date: Tue, 11 Feb 2025 13:36:00 -0500 Subject: [PATCH 01/11] Initial commit From 0ff17d7f4e4b69e82ff2c7195b5c38fc287072e5 Mon Sep 17 00:00:00 2001 From: Allie Mims Date: Tue, 11 Feb 2025 16:09:54 -0500 Subject: [PATCH 02/11] WIP readme updates --- tests/e2e/README.md | 75 ++++++++++++++++++++++++--------------------- 1 file changed, 40 insertions(+), 35 deletions(-) diff --git a/tests/e2e/README.md b/tests/e2e/README.md index 38ca21dbdbd..abf5d1b8ba3 100644 --- a/tests/e2e/README.md +++ b/tests/e2e/README.md @@ -1,6 +1,8 @@ -# WooCommerce Payments End-to-end tests +# WooPayments end-to-end tests 🎭 -E2E tests can be run locally or in GitHub Actions. Github Actions are already configured and doesn't require any changes to run the tests. +WooPayments e2e tests can be found in the `./tests/e2e/specs` directory. These tests run with Playwright and replaced the Puppeteer e2e tests when we completed the migration to Playwright in early 2025. + +E2E tests can be run locally or in GitHub Actions. Github Actions are already configured and don't require any changes to run the tests. ## Setting up & running E2E tests @@ -11,8 +13,8 @@ For running E2E tests locally, create a new file named `local.env` under `tests/

``` -# WooCommerce Payments Dev Tools Repo -WCP_DEV_TOOLS_REPO='https://github.com/dev-tools-repo.git or git@github.com:org/dev-tools-repo.git' +# WooPayments Dev Tools Repo +WCP_DEV_TOOLS_REPO='git@github.com:org/dev-tools-repo.git' # Optional to see additional verbose output. Default false. DEBUG=false @@ -27,15 +29,15 @@ DEBUG=false

Choose Transact Platform Server instance

-It is possible to use the live server or a local docker instance of Transact Platform Server locally. On Github Actions, live server is used for tests. Add the following env variables to your `local.env` based on your preference (replace values as required). +It is possible to use the live server or a local docker instance of the Transact Platform Server when testing locally. On Github Actions, the live server is used for tests. Add the following env variables to your `local.env` based on your preference (replace values as required). **Using Local Server on Docker** -By default, the local E2E environment is configured to use Transact Platform local server instance. Add the following env variables to configure the local server instance. +By default, the local E2E environment is configured to use the Transact Platform local server instance. Add the following env variables to configure the local server instance. ``` # Transact Platform Server Repo -TRANSACT_PLATFORM_SERVER_REPO='https://github.com/server-repo.git or git@github.com:org/server-repo.git' +TRANSACT_PLATFORM_SERVER_REPO='git@github.com:org/server-repo.git' # Stripe account data. Need to support level 3 data to run tests successfully. # These values can be obtained from the Stripe Dashboard: https://dashboard.stripe.com/test/apikeys @@ -50,7 +52,7 @@ E2E_WOOPAY_BLOG_ID= **Using Live Server** -For using a live server, you'll need to add Jetpack blog token, user token & blog id from one of your test sites connected to WooCommerce Payments live account. On a connected test site, you can use the code below to extract the blog id & tokens. +For using the live server, you'll need to add a Jetpack blog token, user token, & blog id from one of your test sites connected to a WooPayments live account. On a connected test site, you can use the code below to extract the blog id & tokens. ``` Jetpack_Options::get_option( 'id' ); Jetpack_Options::get_option( 'blog_token' ); @@ -59,7 +61,7 @@ Jetpack_Options::get_option( 'user_tokens' ); Set the value of `E2E_USE_LOCAL_SERVER` to `false` to enable live server. -Once you have the blog id & tokens, add the following ev variables to your `local.env`. +Once you have the blog id & tokens, add the following env variables to your `local.env`. ``` # Set local server to false for using live server. Default: true. E2E_USE_LOCAL_SERVER=false @@ -78,7 +80,7 @@ E2E_BLOG_ID=''

Installing Plugins

-If you wish to run E2E test for WC Subscriptions, the following env variables needs to be added to your `local.env` (replace values as required). +If you wish to run E2E tests for WC Subscriptions, the following env variables need to be added to your `local.env` (replace values as required). For the `E2E_GH_TOKEN`, follow [these instructions to generate a GitHub Personal Access Token](https://docs.github.com/en/github/authenticating-to-github/creating-a-personal-access-token) and assign the `repo` scope to it. @@ -96,7 +98,7 @@ WC_SUBSCRIPTIONS_REPO='{owner}/{repo}'

Skipping Plugins

-If you wish to skip E2E tests for WC Subscriptions, Action Scheduler or WC Gutenberg Products Blocks, the following env variables needs to be added to your `local.env`. +If you wish to skip E2E tests for WC Subscriptions, Action Scheduler, or WC Gutenberg Products Blocks, the following env variables need to be added to your `local.env`. ``` SKIP_WC_SUBSCRIPTIONS_TESTS=1 SKIP_WC_ACTION_SCHEDULER_TESTS=1 @@ -109,10 +111,10 @@ SKIP_WC_BLOCKS_TESTS=1 ---

-Using specific version of WordPress / WooCommerce +Using a specific version of WordPress or WooCommerce

-To use a specific version of WordPress or WooCommerce for testing, the following env variables needs to be added to your `local.env`. +To use a specific version of WordPress or WooCommerce for testing, the following env variables need to be added to your `local.env`. ``` E2E_WP_VERSION='' E2E_WC_VERSION='' @@ -127,8 +129,8 @@ E2E_WC_VERSION=''

Initialize E2E docker environment

- 1. Make sure to run `npm install`, `composer install` and `npm run build:client` before running setup script. - 2. Run setup script `npm run test:e2e-setup` to spin up E2E environment in docker containers. + 1. Make sure to run `npm install`, `composer install` and `npm run build:client` before running the setup script. + 2. Run the setup script `npm run test:e2e-setup` to spin up E2E environment in docker containers. After the E2E environment is up, you can access the containers on: @@ -147,47 +149,50 @@ E2E_WC_VERSION=''

There are two modes for running tests: -1. **Headless mode**: `npm run test:e2e`. In headless mode test runner executes all or specified specs without launching Chromium interface. This mode is used in CI environment. -2. **Dev mode**: `npm run test:e2e-dev`. Dev mode is interactive and launches Chromium UI. It's useful for developing, debugging and troubleshooting failing tests. There is a custom config used for `jest-puppeteer` to run tests in dev mode. -**Running only a single test suite** +1. **Headless mode**: `npm run test:e2e`. In headless mode test runner executes all or specified specs without launching a Chromium user interface. +2. **UI mode**: `npm run test:e2e-ui`. UI mode is interactive and launches a Chromium user interface. It's useful for developing, debugging, and troubleshooting failing tests. For more information about Playwright UI mode, see [Playwright UI Mode docs](https://playwright.dev/docs/test-ui-mode#introduction). + +#### Running only a single test suite -If you would like to run only one test suite, you can pass the relative path to the test file along with any of the modes mentioned above. e.g. `npm run test:e2e-dev path/to/test`. +If you would like to run only one test suite, you can pass the relative path to the test file along with any of the modes mentioned above. e.g. `npm run test:e2e-ui path/to/test`. -**Running tests in group** +#### Running tests in group By adding additional env variables, it is possible to run a group of tests. e.g. -* `E2E_GROUP='wcpay' E2E_BRANCH='merchant' npm run test:e2e-dev` runs merchant tests for WCPay. -* `E2E_GROUP='wcpay' E2E_BRANCH='shopper' npm run test:e2e-dev` runs shopper tests for WCPay. -* `E2E_GROUP='wcpay' npm run test:e2e-dev` runs merchant & shopper tests for WCPay. +- `E2E_GROUP='wcpay' E2E_BRANCH='merchant' npm run test:e2e-ui` runs merchant tests for WCPay in UI mode. +- `E2E_GROUP='wcpay' E2E_BRANCH='shopper' npm run test:e2e-ui` runs shopper tests for WCPay in UI mode. +- `E2E_GROUP='wcpay' npm run test:e2e-ui` runs merchant & shopper tests for WCPay in UI mode. -Handy utility scripts for managing environment: +Handy utility scripts for managing your E2E environment: -* `npm run test:e2e-down` Stops E2E environment containers. -* `npm run test:e2e-cleanup` Removes fetched dependencies and docker volumes. -* `npm run test:e2e-reset` Stops containers and performs cleanup. -* `npm run test:e2e-up` Starts containers without setting up again. +- `npm run test:e2e-down` Stops E2E environment containers. +- `npm run test:e2e-cleanup` Removes fetched dependencies and docker volumes. +- `npm run test:e2e-reset` Stops containers and performs cleanup. +- `npm run test:e2e-up` Starts containers without setting up again.


-For running E2E tests on an Atomic site, follow the same guidelines mentioned above, and specify `NODE_ENV=atomic`, such as this: +#### Running on Atomic site -`npm run test:e2e-dev -- --NODE_ENV=atomic` - -Note that, at the moment, we are only able to run a subset of the regular E2E tests. +For running E2E tests on the Atomic site, follow the same guidelines mentioned above, and add `NODE_ENV=atomic` to your `local.env` file. Then run bring up your E2E environment. Lastly, run tests using `npm run test:e2e` or `npm run test:e2e-ui`. ## Writing tests -Package `@woocommerce/e2e-environment` overrides `it` method to attach custom reporter for failed tests. -It is important to write test cases within `it()` rather than `test()` function to make sure failed tests are reported to Slack channel. +Place new spec files in the appropriate directory under `tests/e2e/specs`. The directory structure is as follows: + +- **Subscriptions Merchant**: `tests/e2e/specs/subscriptions/merchant` - Subscription related tests for the merchant role. +- **Subscriptions Shopper**: `tests/e2e/specs/subscriptions/shopper` - Subscription related tests for the shopper role. +- **WooPayments Merchant**: `tests/e2e/specs/wcpay/merchant` - Tests for the merchant role in WooPayments. +- **WooPayments Shopper**: `tests/e2e/specs/wcpay/shopper` - Tests for the shopper role in WooPayments, including Blocks E2E tests. ## Debugging tests -Create file `local.env` inside `tests/e2e/config` folder and set `E2E_DEBUG=true` env variable to pause test runner when test fails. +Set `E2E_DEBUG=true` env variable in your `local.env` file inside the `tests/e2e/config` directory to pause the test runner when test fails. ## Slack integration From d96132037f69ce33f6b45e0df27026574565c65c Mon Sep 17 00:00:00 2001 From: Allie Mims Date: Wed, 12 Feb 2025 10:32:38 -0500 Subject: [PATCH 03/11] Update readme --- tests/e2e/README.md | 114 +++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 112 insertions(+), 2 deletions(-) diff --git a/tests/e2e/README.md b/tests/e2e/README.md index abf5d1b8ba3..73311be4440 100644 --- a/tests/e2e/README.md +++ b/tests/e2e/README.md @@ -153,6 +153,11 @@ There are two modes for running tests: 1. **Headless mode**: `npm run test:e2e`. In headless mode test runner executes all or specified specs without launching a Chromium user interface. 2. **UI mode**: `npm run test:e2e-ui`. UI mode is interactive and launches a Chromium user interface. It's useful for developing, debugging, and troubleshooting failing tests. For more information about Playwright UI mode, see [Playwright UI Mode docs](https://playwright.dev/docs/test-ui-mode#introduction). +**Additional options:** + +- `npm run test:e2e keyword` runs tests only with a specific keyword in the file name, e.g. `dispute` or `checkout`. +- `npm run test:e2e -- --update-snapshots` updates snapshots. This can be combined with a keyword to update a specific set of snapshots, e.g. `npm run test:e2e-pw -- --update-snapshots deposits`. + #### Running only a single test suite If you would like to run only one test suite, you can pass the relative path to the test file along with any of the modes mentioned above. e.g. `npm run test:e2e-ui path/to/test`. @@ -192,11 +197,13 @@ Place new spec files in the appropriate directory under `tests/e2e/specs`. The d ## Debugging tests -Set `E2E_DEBUG=true` env variable in your `local.env` file inside the `tests/e2e/config` directory to pause the test runner when test fails. +Currently, the best way to debug tests is to use the Playwright UI mode. This mode allows you to see the browser and interact with it after the test runs. +You can use the locator functionality to help correctly determine the locator syntax to correctly target the HTML element you need. Lastly, you can also use +`console.log()` to assist with debugging tests in UI mode. To run tests in UI mode, use the `npm run test:e2e-ui path/to/test.spec` command. ## Slack integration -Slack reporter requires custom jest config provided by `@woocommerce/e2e-environment` package. This config is only applied with `npm run test:e2e` command. +Slack reporter requires a custom jest config provided by `@woocommerce/e2e-environment` package. This integration is currently disabled. **Configuration steps:** @@ -214,3 +221,106 @@ E2E_SLACK_TOKEN='' E2E_CHANNEL_NAME='' E2E_SLACKBOT_USER='' ``` + +## FAQs + +**I'm getting errors that host.docker.internal is not found.** + +This is because the `host.docker.internal` alias is not available on Linux. You can use the `localhost` alias instead. To apply it, create a file called `docker-compose.override.yml` in the `tests/e2e` directory and add the following content: + +```yaml +services: + playwright: + environment: + - BASE_URL=http://localhost:8084 +``` + +**How do I wait for a page or element to load?** + +Since [Playwright automatically waits](https://playwright.dev/docs/actionability) for elements to be present in the page before interacting with them, you probably don't need to explicitly wait for elements to load. For example, all of the following locators will automatically wait for the element to be present and stable before asserting or interacting with it: + +```ts +await expect( page.getByRole( 'heading', { name: 'Sign up' } ) ).toBeVisible(); +await page.getByRole( 'checkbox', { name: 'Subscribe' } ).check(); +await page.getByRole( 'button', { name: /submit/i } ).click(); +``` + +In some cases, you may need to wait for the page to reach a certain load state before interacting with it. You can use `await page.waitForLoadState( 'domcontentloaded' );` to wait for the page to finish loading. + +**What is the best way to target elements in the page?** + +Prefer the use of [user-facing attribute or test-id locators](https://playwright.dev/docs/locators#locating-elements) to target elements in the page. This will make the tests more resilient to changes to implementation details, such as class names. + +```ts +// Prefer locating by role, label, text, or test id when possible. See https://playwright.dev/docs/locators +await page.getByRole( 'button', { name: 'All payouts' } ).click(); +await page.getByLabel( 'Select a deposit status' ).selectOption( 'Pending' ); +await expect( page.getByText( 'Order received' ) ).toBeVisible(); +await page.getByTestId( 'accept-dispute-button' ).click(); + +// Use CSS selectors as a last resort +await page.locator( 'button.components-button.is-secondary' ); +``` + +**How do I create a visual regression test?** + +Visual regression tests are captured by the [`toHaveScreenshot()` function](https://playwright.dev/docs/api/class-pageassertions#page-assertions-to-have-screenshot-2). This function takes a screenshot of a page or element and compares it to a reference image. If the images are different, the test will fail. + +```ts +await expect( page ).toHaveScreenshot(); + +await expect( + page.getByRole( 'button', { name: 'All payouts' } ) +).toHaveScreenshot(); +``` + +**How can I act as shopper or merchant in a test?** + +1. To switch between `shopper` and `merchant` role in a test, use the `getShopper` and `getMerchant` function: + +```ts +import { getShopper, getMerchant } from './utils/helpers'; + +test( 'should do things as shopper and merchant', async ( { browser } ) => { + const { shopperPage } = await getShopper( browser ); + const { merchantPage } = await getMerchant( browser ); + + // do things as shopper + await shopperPage.goto( '/cart/' ); + + // do things as merchant + await merchantPage.goto( '/wp-admin/admin.php?page=wc-settings' ); +} ); +``` + +1. To act as `shopper` **or** `merchant` for an entire test suite (`describe`), use the helper function `useShopper` or `useMerchant` from `tests/e2e-pw/utils/helpers.ts`: + +```ts +import { useShopper } from '../utils/helpers'; + +test.describe( 'Sign in as customer', () => { + useShopper(); + test( 'Load customer my account page', async ( { page } ) => { + // do things as shopper + await page.goto( '/my-account' ); + } ); +} ); +``` + +**How can I investigate and interact with a test failures?** + +- **Github Action test runs** + - View GitHub checks in the "Checks" tab of a PR + - There are currently four E2E test workflows: + - E2E Tests - Pull Request / WC - latest | wcpay - merchant (pull_request) + - E2E Tests - Pull Request / WC - latest | wcpay - shopper (pull_request) + - E2E Tests - Pull Request / WC - latest | subscriptions - merchant (pull_request) + - E2E Tests - Pull Request / WC - latest | subscriptions - shopper (pull_request) + - Click on the details link to the right of the failed job to see the summary + - In the job summary, click on the "Run tests, upload screenshots & logs" section. + - Click on the artifact download link at the end of the section, then extract and copy the `playwright-report` directory to the root of the WooPayments repository + - Run `npx playwright show-report` to open the report in a browser + - Alternatively, after extracting the artifact you can open the `playwright-report/index.html` file in a browser. +- **Local test runs**: + - Local test reports will output in the `playwright-report` directory + - Run `npx playwright show-report` to open the report in a browser From 91e41aff08a0d10891c3e7c73eb55117040b48b4 Mon Sep 17 00:00:00 2001 From: Allie Mims Date: Wed, 12 Feb 2025 10:50:22 -0500 Subject: [PATCH 04/11] Update readme - fixes and formatting --- tests/e2e/README.md | 38 +++++++++++++++++++++----------------- 1 file changed, 21 insertions(+), 17 deletions(-) diff --git a/tests/e2e/README.md b/tests/e2e/README.md index 73311be4440..fed2a7df15b 100644 --- a/tests/e2e/README.md +++ b/tests/e2e/README.md @@ -9,7 +9,7 @@ E2E tests can be run locally or in GitHub Actions. Github Actions are already co For running E2E tests locally, create a new file named `local.env` under `tests/e2e/config` folder with the following env variables (replace values as required).
-Required env variables +

Required env variables

``` @@ -26,7 +26,7 @@ DEBUG=false ---

-Choose Transact Platform Server instance +

Choose Transact Platform Server instance

It is possible to use the live server or a local docker instance of the Transact Platform Server when testing locally. On Github Actions, the live server is used for tests. Add the following env variables to your `local.env` based on your preference (replace values as required). @@ -52,7 +52,7 @@ E2E_WOOPAY_BLOG_ID= **Using Live Server** -For using the live server, you'll need to add a Jetpack blog token, user token, & blog id from one of your test sites connected to a WooPayments live account. On a connected test site, you can use the code below to extract the blog id & tokens. +For using the live server, you'll need to add a Jetpack blog token, user token, & blog id from one of your test sites connected to a WooPayments test account. On a connected test site, you can use the code below to extract the blog id & tokens. ``` Jetpack_Options::get_option( 'id' ); Jetpack_Options::get_option( 'blog_token' ); @@ -77,7 +77,7 @@ E2E_BLOG_ID='' ---

-Installing Plugins +

Installing Plugins

If you wish to run E2E tests for WC Subscriptions, the following env variables need to be added to your `local.env` (replace values as required). @@ -95,7 +95,7 @@ WC_SUBSCRIPTIONS_REPO='{owner}/{repo}' ---

-Skipping Plugins +

Skipping Plugins

If you wish to skip E2E tests for WC Subscriptions, Action Scheduler, or WC Gutenberg Products Blocks, the following env variables need to be added to your `local.env`. @@ -111,7 +111,7 @@ SKIP_WC_BLOCKS_TESTS=1 ---

-Using a specific version of WordPress or WooCommerce +

Using a specific version of WordPress or WooCommerce

To use a specific version of WordPress or WooCommerce for testing, the following env variables need to be added to your `local.env`. @@ -126,7 +126,7 @@ E2E_WC_VERSION='' ---

-Initialize E2E docker environment +

Initialize E2E docker environment

1. Make sure to run `npm install`, `composer install` and `npm run build:client` before running the setup script. @@ -145,22 +145,22 @@ E2E_WC_VERSION='' ---

-Running tests +

Running tests

There are two modes for running tests: -1. **Headless mode**: `npm run test:e2e`. In headless mode test runner executes all or specified specs without launching a Chromium user interface. -2. **UI mode**: `npm run test:e2e-ui`. UI mode is interactive and launches a Chromium user interface. It's useful for developing, debugging, and troubleshooting failing tests. For more information about Playwright UI mode, see [Playwright UI Mode docs](https://playwright.dev/docs/test-ui-mode#introduction). +1. **Headless mode**: `npm run test:e2e`. In headless mode the test runner executes all or specified specs without launching a Chromium user interface. +2. **UI mode**: `npm run test:e2e-ui`. UI mode is interactive and launches a Chromium user interface. It's useful for developing, debugging, and troubleshooting failing tests. For more information about Playwright UI mode, see the [Playwright UI Mode docs](https://playwright.dev/docs/test-ui-mode#introduction). -**Additional options:** +**Additional options** - `npm run test:e2e keyword` runs tests only with a specific keyword in the file name, e.g. `dispute` or `checkout`. -- `npm run test:e2e -- --update-snapshots` updates snapshots. This can be combined with a keyword to update a specific set of snapshots, e.g. `npm run test:e2e-pw -- --update-snapshots deposits`. +- `npm run test:e2e -- --update-snapshots` updates snapshots. This can be combined with a keyword to update a specific set of snapshots, e.g. `npm run test:e2e -- --update-snapshots deposits`. #### Running only a single test suite -If you would like to run only one test suite, you can pass the relative path to the test file along with any of the modes mentioned above. e.g. `npm run test:e2e-ui path/to/test`. +If you would like to run only one test suite, you can pass the relative path to the test file along with any of the modes mentioned above. e.g. `npm run test:e2e-ui path/to/test.spec.ts`. #### Running tests in group @@ -170,6 +170,12 @@ By adding additional env variables, it is possible to run a group of tests. e.g. - `E2E_GROUP='wcpay' E2E_BRANCH='shopper' npm run test:e2e-ui` runs shopper tests for WCPay in UI mode. - `E2E_GROUP='wcpay' npm run test:e2e-ui` runs merchant & shopper tests for WCPay in UI mode. +It is also possible to run the groups using the relative path to the tests. e.g. + +- `npm run test:e2e-ui tests/e2e/specs/wcpay/merchant` runs merchant tests for WCPay in UI mode. +- `npm run test:e2e-ui tests/e2e/specs/wcpay/shopper` runs shopper tests for WCPay in UI mode. +- `npm run test:e2e-ui tests/e2e/specs/wcpay` runs merchant & shopper tests for WCPay in UI mode. + Handy utility scripts for managing your E2E environment: - `npm run test:e2e-down` Stops E2E environment containers. @@ -180,11 +186,9 @@ Handy utility scripts for managing your E2E environment:

-
- -#### Running on Atomic site +### Running on Atomic site -For running E2E tests on the Atomic site, follow the same guidelines mentioned above, and add `NODE_ENV=atomic` to your `local.env` file. Then run bring up your E2E environment. Lastly, run tests using `npm run test:e2e` or `npm run test:e2e-ui`. +For running E2E tests on the Atomic site, follow the same guidelines mentioned above, and add `NODE_ENV=atomic` to your `local.env` file. Then bring up your E2E environment. Lastly, run tests using `npm run test:e2e` or `npm run test:e2e-ui`. ## Writing tests From ef87a53ea99d828050a7d677fc6e50609a4e8c2a Mon Sep 17 00:00:00 2001 From: Allie Mims Date: Wed, 12 Feb 2025 10:52:28 -0500 Subject: [PATCH 05/11] Fix summary headers --- tests/e2e/README.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/tests/e2e/README.md b/tests/e2e/README.md index fed2a7df15b..2ba48d55a30 100644 --- a/tests/e2e/README.md +++ b/tests/e2e/README.md @@ -9,7 +9,7 @@ E2E tests can be run locally or in GitHub Actions. Github Actions are already co For running E2E tests locally, create a new file named `local.env` under `tests/e2e/config` folder with the following env variables (replace values as required).
-

Required env variables

 +Required env variables

``` @@ -26,7 +26,7 @@ DEBUG=false ---

-

Choose Transact Platform Server instance

 +Choose Transact Platform Server instance

It is possible to use the live server or a local docker instance of the Transact Platform Server when testing locally. On Github Actions, the live server is used for tests. Add the following env variables to your `local.env` based on your preference (replace values as required). @@ -77,7 +77,7 @@ E2E_BLOG_ID='' ---

-

Installing Plugins

 +Installing Plugins

If you wish to run E2E tests for WC Subscriptions, the following env variables need to be added to your `local.env` (replace values as required). @@ -95,7 +95,7 @@ WC_SUBSCRIPTIONS_REPO='{owner}/{repo}' ---

-

Skipping Plugins

 +Skipping Plugins

If you wish to skip E2E tests for WC Subscriptions, Action Scheduler, or WC Gutenberg Products Blocks, the following env variables need to be added to your `local.env`. @@ -111,7 +111,7 @@ SKIP_WC_BLOCKS_TESTS=1 ---

-

Using a specific version of WordPress or WooCommerce

 +Using a specific version of WordPress or WooCommerce

To use a specific version of WordPress or WooCommerce for testing, the following env variables need to be added to your `local.env`. @@ -126,7 +126,7 @@ E2E_WC_VERSION='' ---

-

Initialize E2E docker environment

 +Initialize E2E docker environment

1. Make sure to run `npm install`, `composer install` and `npm run build:client` before running the setup script. @@ -145,7 +145,7 @@ E2E_WC_VERSION='' ---

-

Running tests

 +Running tests

There are two modes for running tests: From b5b6f2d11e7995d751cbb9550db64e06295b44a9 Mon Sep 17 00:00:00 2001 From: Allie Mims Date: Wed, 12 Feb 2025 10:54:20 -0500 Subject: [PATCH 06/11] Remove e2e-pw/readme --- tests/e2e-pw/README.md | 116 ----------------------------------------- 1 file changed, 116 deletions(-) delete mode 100644 tests/e2e-pw/README.md diff --git a/tests/e2e-pw/README.md b/tests/e2e-pw/README.md deleted file mode 100644 index e7f7f50e671..00000000000 --- a/tests/e2e-pw/README.md +++ /dev/null @@ -1,116 +0,0 @@ -# Playwright end-to-end tests 🎭 - -Playwright e2e tests can be found in the `./tests/e2e-pw/specs` directory. These will run in parallel with the existing Puppeteer e2e tests and are intended to replace them as they are migrated. - -## Setup local e2e environment - -See [tests/e2e/README.md](/tests/e2e/README.md) for detailed e2e environment setup instructions. - -1. `npm run test:e2e-setup` -1. `npm run test:e2e-up` - -> [!TIP] -> In case some tests fail due to the lack of `data-test-id` attributes, you'll need to run `npm start` or `NODE_ENV=test npm run build:client` to re-build the assets. - -## Running Playwright e2e tests - -- `npm run test:e2e-pw` headless run from within a linux docker container. -- `npm run test:e2e-pw-ui` runs tests in interactive UI mode from within a linux docker container – recommended for authoring tests and re-running failed tests. -- `npm run test:e2e-pw keyword` runs tests only with a specific keyword in the file name, e.g. `dispute` or `checkout`. -- `npm run test:e2e-pw -- --update-snapshots` updates snapshots. This can be combined with a keyword to update a specific set of snapshots, e.g. `npm run test:e2e-pw -- --update-snapshots deposits`. - -## FAQs - -**I'm getting errors that host.docker.internal is not found.** - -This is because the `host.docker.internal` alias is not available on Linux. You can use the `localhost` alias instead. To apply it, create a file called `docker-compose.override.yml` in the `tests/e2e-pw` directory and add the following content: - -```yaml -services: - playwright: - environment: - - BASE_URL=http://localhost:8084 -``` - -**How do I wait for a page or element to load?** - -Since [Playwright automatically waits](https://playwright.dev/docs/actionability) for elements to be present in the page before interacting with them, you probably don't need to explicitly wait for elements to load. For example, all of the following locators will automatically wait for the element to be present and stable before asserting or interacting with it: - -```ts -await expect( page.getByRole( 'heading', { name: 'Sign up' } ) ).toBeVisible(); -await page.getByRole( 'checkbox', { name: 'Subscribe' } ).check(); -await page.getByRole( 'button', { name: /submit/i } ).click(); -``` - -In some cases, you may need to wait for the page to reach a certain load state before interacting with it. You can use `await page.waitForLoadState( 'domcontentloaded' );` to wait for the page to finish loading. - -**What is the best way to target elements in the page?** - -Prefer the use of [user-facing attribute or test-id locators](https://playwright.dev/docs/locators#locating-elements) to target elements in the page. This will make the tests more resilient to changes to implementation details, such as class names. - -```ts -// Prefer locating by role, label, text, or test id when possible. See https://playwright.dev/docs/locators -await page.getByRole( 'button', { name: 'All payouts' } ).click(); -await page.getByLabel( 'Select a deposit status' ).selectOption( 'Pending' ); -await expect( page.getByText( 'Order received' ) ).toBeVisible(); -await page.getByTestId( 'accept-dispute-button' ).click(); - -// Use CSS selectors as a last resort -await page.locator( 'button.components-button.is-secondary' ); -``` - -**How do I create a visual regression test?** - -Visual regression tests are captured by the [`toHaveScreenshot()` function](https://playwright.dev/docs/api/class-pageassertions#page-assertions-to-have-screenshot-2). This function takes a screenshot of a page or element and compares it to a reference image. If the images are different, the test will fail. - -```ts -await expect( page ).toHaveScreenshot(); - -await expect( - page.getByRole( 'button', { name: 'All payouts' } ) -).toHaveScreenshot(); -``` - -**How can I act as shopper or merchant in a test?** - -1. To switch between `shopper` and `merchant` role in a test, use the `getShopper` and `getMerchant` function: - -```ts -import { getShopper, getMerchant } from './utils/helpers'; - -test( 'should do things as shopper and merchant', async ( { browser } ) => { - const { shopperPage } = await getShopper( browser ); - const { merchantPage } = await getMerchant( browser ); - - // do things as shopper - await shopperPage.goto( '/cart/' ); - - // do things as merchant - await merchantPage.goto( '/wp-admin/admin.php?page=wc-settings' ); -} ); -``` - -2. To act as `shopper` or `merchant` for an entire test suite (`describe`), use the helper function `useShopper` or `useMerchant` from `tests/e2e-pw/utils/helpers.ts`: - -```ts -import { useShopper } from '../utils/helpers'; - -test.describe( 'Sign in as customer', () => { - useShopper(); - test( 'Load customer my account page', async ( { page } ) => { - // do things as shopper - await page.goto( '/my-account' ); - } ); -} ); -``` - -**How can I investigate and interact with a test failures?** - -- **Github Action test runs** - - View GitHub checks in the "Checks" tab of a PR - - Click on the "E2E Playwright Tests" job to see the job summary - - Download the `playwright-report.zip` artifact, extract and copy the `playwright-report` directory to the root of the WooPayments repository - - Run `npx playwright show-report` to open the report in a browser -- **Local test runs**: - - Local test reports will output in the `playwright-report` directory - - Run `npx playwright show-report` to open the report in a browser From 789222b56086c5017aa83b16be48f9e95f5a90a1 Mon Sep 17 00:00:00 2001 From: Allie Mims Date: Wed, 12 Feb 2025 10:54:30 -0500 Subject: [PATCH 07/11] Add changelog --- changelog/dev-10334-update-playwright-e2e-docs | 4 ++++ 1 file changed, 4 insertions(+) create mode 100644 changelog/dev-10334-update-playwright-e2e-docs diff --git a/changelog/dev-10334-update-playwright-e2e-docs b/changelog/dev-10334-update-playwright-e2e-docs new file mode 100644 index 00000000000..462528e2345 --- /dev/null +++ b/changelog/dev-10334-update-playwright-e2e-docs @@ -0,0 +1,4 @@ +Significance: patch +Type: dev + +Update E2E readme doc. From 1f75b1f375a93fc2a99e5d1defb9cba393338243 Mon Sep 17 00:00:00 2001 From: Allie Mims Date: Wed, 12 Feb 2025 12:19:05 -0500 Subject: [PATCH 08/11] Update repo var examples --- tests/e2e/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/e2e/README.md b/tests/e2e/README.md index 2ba48d55a30..6e2e6c3fa3e 100644 --- a/tests/e2e/README.md +++ b/tests/e2e/README.md @@ -14,7 +14,7 @@ For running E2E tests locally, create a new file named `local.env` under `tests/ ``` # WooPayments Dev Tools Repo -WCP_DEV_TOOLS_REPO='git@github.com:org/dev-tools-repo.git' +WCP_DEV_TOOLS_REPO='https://github.com/dev-tools-repo.git or git@github.com:org/dev-tools-repo.git' # Optional to see additional verbose output. Default false. DEBUG=false @@ -37,7 +37,7 @@ By default, the local E2E environment is configured to use the Transact Platform ``` # Transact Platform Server Repo -TRANSACT_PLATFORM_SERVER_REPO='git@github.com:org/server-repo.git' +TRANSACT_PLATFORM_SERVER_REPO='https://github.com/server-repo.git or git@github.com:org/server-repo.git' # Stripe account data. Need to support level 3 data to run tests successfully. # These values can be obtained from the Stripe Dashboard: https://dashboard.stripe.com/test/apikeys From cceee6211ae63d6a8de95f890d067d94710b1c11 Mon Sep 17 00:00:00 2001 From: Allie Mims Date: Wed, 12 Feb 2025 15:38:31 -0500 Subject: [PATCH 09/11] Update running tests in group --- tests/e2e/README.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/tests/e2e/README.md b/tests/e2e/README.md index 6e2e6c3fa3e..6d88d5839db 100644 --- a/tests/e2e/README.md +++ b/tests/e2e/README.md @@ -164,11 +164,13 @@ If you would like to run only one test suite, you can pass the relative path to #### Running tests in group -By adding additional env variables, it is possible to run a group of tests. e.g. +By adding additional env variables to your `local.env` file, it is possible to run a group of tests. e.g. -- `E2E_GROUP='wcpay' E2E_BRANCH='merchant' npm run test:e2e-ui` runs merchant tests for WCPay in UI mode. -- `E2E_GROUP='wcpay' E2E_BRANCH='shopper' npm run test:e2e-ui` runs shopper tests for WCPay in UI mode. -- `E2E_GROUP='wcpay' npm run test:e2e-ui` runs merchant & shopper tests for WCPay in UI mode. +- Adding `E2E_GROUP='wcpay'` and `E2E_BRANCH='merchant'` to your `local.env` file, then running `npm run test:e2e-ui` runs the WooPayments merchant tests for WCPay in UI mode. +- Adding `E2E_GROUP='wcpay'` and `E2E_BRANCH='shopper'` to your `local.env` file, then running `npm run test:e2e-ui` runs WooPayments shopper tests for WCPay in UI mode. +- Adding just `E2E_GROUP='wcpay'` to your `local.env` file, then running `npm run test:e2e-ui` runs WooPayments merchant & shopper tests for WCPay in UI mode. +- Available groups are `wcpay` and `subscriptions`. +- Available branches are `merchant` and `shopper`. It is also possible to run the groups using the relative path to the tests. e.g. From da8bc67a876d2ca1181c467a7b65b4f592f4df35 Mon Sep 17 00:00:00 2001 From: Allie Mims Date: Thu, 13 Feb 2025 10:53:57 -0500 Subject: [PATCH 10/11] Update Slack integration section --- tests/e2e/README.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/tests/e2e/README.md b/tests/e2e/README.md index 6d88d5839db..7c5d621886f 100644 --- a/tests/e2e/README.md +++ b/tests/e2e/README.md @@ -209,19 +209,20 @@ You can use the locator functionality to help correctly determine the locator sy ## Slack integration -Slack reporter requires a custom jest config provided by `@woocommerce/e2e-environment` package. This integration is currently disabled. +The Slack reporter is a custom reporter that sends e2e test failures to a Slack channel. The reporter is configured to only send the first failure of a test to Slack. If the retry also fails it will not be sent to prevent spamming the channel. **Configuration steps:** 1. Create public Slack channel for reporting. 2. [Create Slack app.](https://api.slack.com/apps/) 3. Add OAuth permissions to the app: - * `chat:write` - * `files:write` + - `chat:write` + - `files:write` 4. Install app into channel. `Settings > Install App` page. 5. Go to slack channel and manually invite created slack app by mentioning app bot username. User name can be found and configured on app config page `Features > App Home` page. 6. Set following env variables either locally or in CI: -``` + +```bash CI=true E2E_SLACK_TOKEN='' E2E_CHANNEL_NAME='' From 6fe56ea8c86d576394646b38ecce9673af46f4dc Mon Sep 17 00:00:00 2001 From: Allie Mims Date: Thu, 13 Feb 2025 14:13:52 -0500 Subject: [PATCH 11/11] Update Slack integration section --- tests/e2e/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/e2e/README.md b/tests/e2e/README.md index 7c5d621886f..442a633240f 100644 --- a/tests/e2e/README.md +++ b/tests/e2e/README.md @@ -209,7 +209,7 @@ You can use the locator functionality to help correctly determine the locator sy ## Slack integration -The Slack reporter is a custom reporter that sends e2e test failures to a Slack channel. The reporter is configured to only send the first failure of a test to Slack. If the retry also fails it will not be sent to prevent spamming the channel. +The Slack reporter is a custom reporter that sends e2e test failures to a public Slack channel (search Slack channel ID `CQ0Q6N62D`). The reporter is configured to only send the first failure of a test to Slack. If the retry also fails it will not be sent to prevent spamming the channel. **Configuration steps:**