From f1e7f065db8dcd7543815b2ee0523064fe76c767 Mon Sep 17 00:00:00 2001 From: jos <17252150+jostnes@users.noreply.github.com> Date: Mon, 18 Sep 2023 11:22:43 +0800 Subject: [PATCH 1/4] update ui test readme --- WordPress/UITests/README.md | 41 ++++++++++++++++++++++++++++++------- 1 file changed, 34 insertions(+), 7 deletions(-) diff --git a/WordPress/UITests/README.md b/WordPress/UITests/README.md index 7fd6be6203e8..82496fa92e1b 100644 --- a/WordPress/UITests/README.md +++ b/WordPress/UITests/README.md @@ -85,17 +85,44 @@ Note that due to the mock server setup, tests cannot be run on physical devices When adding a new UI test, consider: -* Whether you need to test a user flow (to accomplish a task or goal) or a specific feature (e.g. boundary testing). -* What screens are being tested (defined as page objects in `Screens/`). -* Whether there are repeated flows across tests (defined in `Flows/`). -* What network requests are made during the test (defined in `API-Mocks/`). +* Whether to test a user flow (to accomplish a task or goal) or a specific feature (e.g. boundary testing). +* What screens are being tested (defined as screen objects in [.../Screens](https://github.com/wordpress-mobile/WordPress-iOS/tree/trunk/WordPress/UITestsFoundation/Screens)). +* Whether there are repeated flows across tests (defined in [.../Flows](https://github.com/wordpress-mobile/WordPress-iOS/tree/trunk/WordPress/UITests/Flows)). +* What network requests are made during the test (defined in [.../API-Mocks](https://github.com/wordpress-mobile/WordPress-iOS/tree/trunk/API-Mocks)). -It's preferred to focus UI tests on entire user flows, and group tests with related flows or goals in the same test suite. +Tests classes are grouped together in [.../Tests](https://github.com/wordpress-mobile/WordPress-iOS/tree/trunk/WordPress/UITests/Tests) -When you add a new test, you may need to add new screens, methods, and flows. We use page objects and method chaining for clarity in our tests. Wherever possible, use an existing `accessibilityIdentifier` (or add one to the app) instead of a string to select a UI element on the screen. This ensures tests can be run regardless of the device language. +When you add a new test, you may need to add new screens, methods, and flows. We use page objects and method chaining for clarity in our tests. Wherever possible, use an existing `accessibilityIdentifier` (or add one to the app) instead of a string to select a UI element on the screen. This ensures tests can be run regardless of the device language.j + +## Naming convention + +* When creating new tests, use this format for the name to make it easier to see what the test is doing: `testActionFeature` e.g. `testCreateScheduledPost()` +* When creating new methods, use this format: `actionObject` e.g. `closePostSettings()` +* For assert methods, use this format `verifyWhatToVerify` e.g. `verifyPostExists()` +* Note that there’s a common global method `assertScreenIsLoaded()` that can be used to assert all screens + +## Passing hard-coded `Strings` | `Numbers` in tests + +There are some cases where we would need to pass hard-coded values in the test, this should happen on the Test level and not on the screen level (unless there’s a really good reason). + +This is so methods are not limited to being used with a fixed value and remain flexible. In the case where those values change we would also be able to update only the test file(s) without making changes elsewhere. ## Adding or updating network mocks When you add a test (or when the app changes), the request definitions for WireMock need to be updated in `API-Mocks/`. You can read WireMock’s documentation [here](http://wiremock.org/docs/). -If you are unsure what network requests need to be mocked for a test, an easy way to find out is to run the app through [Charles Proxy](https://www.charlesproxy.com/) and observe the required requests. +If you are unsure what network requests need to be mocked for a test, an easy way to find out is to run the app through [Proxyman](https://proxyman.io/) or [Charles Proxy](https://www.charlesproxy.com/) and observe the required requests. + +Currently, the project does not apply strict mock matching criteria, this means that if there are unmatched requests that are not being used by the test itself, the test should still work although errors like this can be seen in the logs: + +## Using stateful behavior for mocks +1. To add scenarios that use stateful behavior, do the following: +Add the new scenario in [scenarios.json](https://github.com//wordpress-mobile/WordPress-iOS/tree/trunk/API-Mocks/WordPressMocks/src/main/assets/mocks/__files/__admin/scenarios.json) +2. Fetch and reset scenario during `SetUp()` in the test class containing the test, e.g. seen on [Notification Test](https://github.com/wordpress-mobile/WordPress-iOS/blob/5730cee6568fe43fb3a5108e396e12244c62b3e5/WordPress/UITests/Tests/NotificationTests.swift#L18-L23) +3. Update JSON mappings to contain the following 3 new attributes, `scenarioName`, `requiredScenarioState` and `newScenarioState`, and the response matching the state of the scenario, e.g. seen on [Notification Test](https://github.com/wordpress-mobile/WordPress-iOS/blob/5730cee6568fe43fb3a5108e396e12244c62b3e5/API-Mocks/WordPressMocks/src/main/assets/mocks/mappings/wpcom/notifications/notifications_comment_reply_before.json#L2-L4) + +## Tips and tricks on using mocks +* When getting the same request with the same header but a different request body to return different responses, experiment with using [different matchers](https://docs.wiremock.io/request-matching/matcher-types/). From some experimenting, would recommend using `matchesJsonPath` which is used to differentiate [Create Page](https://github.com/wordpress-mobile/WordPress-iOS/blob/5a00e849d8877e8ae2a6ec6bc9c762e68e6e0620/API-Mocks/WordPressMocks/src/main/assets/mocks/mappings/wpcom/pages/sites_106707880_pages_new.json#L18) and [Create Post](https://github.com/wordpress-mobile/WordPress-iOS/blob/5a00e849d8877e8ae2a6ec6bc9c762e68e6e0620/API-Mocks/WordPressMocks/src/main/assets/mocks/mappings/wpcom/posts/posts_new.json#L18) +* Use `verbose` to debug errors, this can be updated adding the verbose parameter when [starting the WireMock server](https://github.com/wordpress-mobile/WordPress-iOS/blob/5a00e849d8877e8ae2a6ec6bc9c762e68e6e0620/API-Mocks/scripts/start.sh#L20-L23) (don’t forget the slash to not break the command) +* If there are no errors on the console, but the mocks don’t work as expected, check out the app’s logs for errors. Sometimes it could be that the JSON mapping is not parsed correctly. + From f9f7f130c29e0148fb469425d79569ca7c176e65 Mon Sep 17 00:00:00 2001 From: jos <17252150+jostnes@users.noreply.github.com> Date: Mon, 18 Sep 2023 15:51:03 +0800 Subject: [PATCH 2/4] remove ... from folder links --- WordPress/UITests/README.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/WordPress/UITests/README.md b/WordPress/UITests/README.md index 82496fa92e1b..21c499e4f258 100644 --- a/WordPress/UITests/README.md +++ b/WordPress/UITests/README.md @@ -86,13 +86,13 @@ Note that due to the mock server setup, tests cannot be run on physical devices When adding a new UI test, consider: * Whether to test a user flow (to accomplish a task or goal) or a specific feature (e.g. boundary testing). -* What screens are being tested (defined as screen objects in [.../Screens](https://github.com/wordpress-mobile/WordPress-iOS/tree/trunk/WordPress/UITestsFoundation/Screens)). -* Whether there are repeated flows across tests (defined in [.../Flows](https://github.com/wordpress-mobile/WordPress-iOS/tree/trunk/WordPress/UITests/Flows)). -* What network requests are made during the test (defined in [.../API-Mocks](https://github.com/wordpress-mobile/WordPress-iOS/tree/trunk/API-Mocks)). +* What screens are being tested (defined as screen objects in [Screens](https://github.com/wordpress-mobile/WordPress-iOS/tree/trunk/WordPress/UITestsFoundation/Screens)). +* Whether there are repeated flows across tests (defined in [Flows](https://github.com/wordpress-mobile/WordPress-iOS/tree/trunk/WordPress/UITests/Flows)). +* What network requests are made during the test (defined in [API-Mocks](https://github.com/wordpress-mobile/WordPress-iOS/tree/trunk/API-Mocks)). -Tests classes are grouped together in [.../Tests](https://github.com/wordpress-mobile/WordPress-iOS/tree/trunk/WordPress/UITests/Tests) +Tests classes are grouped together in [Tests](https://github.com/wordpress-mobile/WordPress-iOS/tree/trunk/WordPress/UITests/Tests) -When you add a new test, you may need to add new screens, methods, and flows. We use page objects and method chaining for clarity in our tests. Wherever possible, use an existing `accessibilityIdentifier` (or add one to the app) instead of a string to select a UI element on the screen. This ensures tests can be run regardless of the device language.j +When you add a new test, you may need to add new screens, methods, and flows. We use page objects and method chaining for clarity in our tests. Wherever possible, use an existing `accessibilityIdentifier` (or add one to the app) instead of a string to select a UI element on the screen. This ensures tests can be run regardless of the device language. ## Naming convention From 1c3cbadd9694413e88a9083e716ac361eb4e201f Mon Sep 17 00:00:00 2001 From: jos <17252150+jostnes@users.noreply.github.com> Date: Mon, 18 Sep 2023 16:29:46 +0800 Subject: [PATCH 3/4] add missing screenshot, add clarifying statement --- WordPress/UITests/README.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/WordPress/UITests/README.md b/WordPress/UITests/README.md index 21c499e4f258..3c1ba2db06da 100644 --- a/WordPress/UITests/README.md +++ b/WordPress/UITests/README.md @@ -113,7 +113,11 @@ When you add a test (or when the app changes), the request definitions for WireM If you are unsure what network requests need to be mocked for a test, an easy way to find out is to run the app through [Proxyman](https://proxyman.io/) or [Charles Proxy](https://www.charlesproxy.com/) and observe the required requests. -Currently, the project does not apply strict mock matching criteria, this means that if there are unmatched requests that are not being used by the test itself, the test should still work although errors like this can be seen in the logs: +Currently, the project does not apply strict mock matching criteria, this means that if there are unmatched requests that are not being used by the test itself. The test should still work although errors like this can be seen in the logs: + +unmatched request + +When adding a new test however, it is recommended that all requests are matched so that the logs are not further cluttered with `Request was not matched` like in the above screenshot. ## Using stateful behavior for mocks 1. To add scenarios that use stateful behavior, do the following: From 3eae490316d07040c80a197cc4bac290adad4a7d Mon Sep 17 00:00:00 2001 From: jos <17252150+jostnes@users.noreply.github.com> Date: Tue, 19 Sep 2023 11:19:38 +0800 Subject: [PATCH 4/4] review updates --- WordPress/UITests/README.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/WordPress/UITests/README.md b/WordPress/UITests/README.md index 3c1ba2db06da..f5e90aa54f01 100644 --- a/WordPress/UITests/README.md +++ b/WordPress/UITests/README.md @@ -87,7 +87,7 @@ When adding a new UI test, consider: * Whether to test a user flow (to accomplish a task or goal) or a specific feature (e.g. boundary testing). * What screens are being tested (defined as screen objects in [Screens](https://github.com/wordpress-mobile/WordPress-iOS/tree/trunk/WordPress/UITestsFoundation/Screens)). -* Whether there are repeated flows across tests (defined in [Flows](https://github.com/wordpress-mobile/WordPress-iOS/tree/trunk/WordPress/UITests/Flows)). +* Are there repeated flows across tests (defined in [Flows](https://github.com/wordpress-mobile/WordPress-iOS/tree/trunk/WordPress/UITests/Flows)). * What network requests are made during the test (defined in [API-Mocks](https://github.com/wordpress-mobile/WordPress-iOS/tree/trunk/API-Mocks)). Tests classes are grouped together in [Tests](https://github.com/wordpress-mobile/WordPress-iOS/tree/trunk/WordPress/UITests/Tests) @@ -96,16 +96,16 @@ When you add a new test, you may need to add new screens, methods, and flows. We ## Naming convention -* When creating new tests, use this format for the name to make it easier to see what the test is doing: `testActionFeature` e.g. `testCreateScheduledPost()` -* When creating new methods, use this format: `actionObject` e.g. `closePostSettings()` -* For assert methods, use this format `verifyWhatToVerify` e.g. `verifyPostExists()` +* When creating new tests, use the `testActionFeature` format for the test name to make it easier to see what the test is for: e.g. `testCreateScheduledPost()` +* When creating new methods, use the `actionObject` format: e.g. `closePostSettings()` +* For assert methods, use the `verifyWhatToVerify` format: e.g. `verifyPostExists()` * Note that there’s a common global method `assertScreenIsLoaded()` that can be used to assert all screens ## Passing hard-coded `Strings` | `Numbers` in tests -There are some cases where we would need to pass hard-coded values in the test, this should happen on the Test level and not on the screen level (unless there’s a really good reason). +There are some cases where we would need to pass hard-coded values in the test, this should happen on the Test level and not on the Screen level. -This is so methods are not limited to being used with a fixed value and remain flexible. In the case where those values change we would also be able to update only the test file(s) without making changes elsewhere. +This is so that test methods are not limited to being used with a fixed value and remain flexible. In the case where those values change we would also be able to update only the test file(s) without making changes elsewhere. ## Adding or updating network mocks @@ -120,13 +120,13 @@ Currently, the project does not apply strict mock matching criteria, this means When adding a new test however, it is recommended that all requests are matched so that the logs are not further cluttered with `Request was not matched` like in the above screenshot. ## Using stateful behavior for mocks -1. To add scenarios that use stateful behavior, do the following: -Add the new scenario in [scenarios.json](https://github.com//wordpress-mobile/WordPress-iOS/tree/trunk/API-Mocks/WordPressMocks/src/main/assets/mocks/__files/__admin/scenarios.json) + +1. Add the new scenario in [scenarios.json](https://github.com//wordpress-mobile/WordPress-iOS/tree/trunk/API-Mocks/WordPressMocks/src/main/assets/mocks/__files/__admin/scenarios.json) 2. Fetch and reset scenario during `SetUp()` in the test class containing the test, e.g. seen on [Notification Test](https://github.com/wordpress-mobile/WordPress-iOS/blob/5730cee6568fe43fb3a5108e396e12244c62b3e5/WordPress/UITests/Tests/NotificationTests.swift#L18-L23) 3. Update JSON mappings to contain the following 3 new attributes, `scenarioName`, `requiredScenarioState` and `newScenarioState`, and the response matching the state of the scenario, e.g. seen on [Notification Test](https://github.com/wordpress-mobile/WordPress-iOS/blob/5730cee6568fe43fb3a5108e396e12244c62b3e5/API-Mocks/WordPressMocks/src/main/assets/mocks/mappings/wpcom/notifications/notifications_comment_reply_before.json#L2-L4) ## Tips and tricks on using mocks -* When getting the same request with the same header but a different request body to return different responses, experiment with using [different matchers](https://docs.wiremock.io/request-matching/matcher-types/). From some experimenting, would recommend using `matchesJsonPath` which is used to differentiate [Create Page](https://github.com/wordpress-mobile/WordPress-iOS/blob/5a00e849d8877e8ae2a6ec6bc9c762e68e6e0620/API-Mocks/WordPressMocks/src/main/assets/mocks/mappings/wpcom/pages/sites_106707880_pages_new.json#L18) and [Create Post](https://github.com/wordpress-mobile/WordPress-iOS/blob/5a00e849d8877e8ae2a6ec6bc9c762e68e6e0620/API-Mocks/WordPressMocks/src/main/assets/mocks/mappings/wpcom/posts/posts_new.json#L18) -* Use `verbose` to debug errors, this can be updated adding the verbose parameter when [starting the WireMock server](https://github.com/wordpress-mobile/WordPress-iOS/blob/5a00e849d8877e8ae2a6ec6bc9c762e68e6e0620/API-Mocks/scripts/start.sh#L20-L23) (don’t forget the slash to not break the command) +* When getting the same request with the same header but a different request body to return different responses, experiment with using [different matchers](https://docs.wiremock.io/request-matching/matcher-types/). From some experimenting, I would recommend using `matchesJsonPath` which is used to differentiate [Create Page](https://github.com/wordpress-mobile/WordPress-iOS/blob/5a00e849d8877e8ae2a6ec6bc9c762e68e6e0620/API-Mocks/WordPressMocks/src/main/assets/mocks/mappings/wpcom/pages/sites_106707880_pages_new.json#L18) and [Create Post](https://github.com/wordpress-mobile/WordPress-iOS/blob/5a00e849d8877e8ae2a6ec6bc9c762e68e6e0620/API-Mocks/WordPressMocks/src/main/assets/mocks/mappings/wpcom/posts/posts_new.json#L18). +* Use `verbose` to debug errors, this can be updated by adding the `verbose` parameter when [starting the WireMock server](https://github.com/wordpress-mobile/WordPress-iOS/blob/5a00e849d8877e8ae2a6ec6bc9c762e68e6e0620/API-Mocks/scripts/start.sh#L20-L23) (don’t forget the slash to not break the command). * If there are no errors on the console, but the mocks don’t work as expected, check out the app’s logs for errors. Sometimes it could be that the JSON mapping is not parsed correctly.