chore(tests): end to end testing again #11428

brainbicycle · 2025-01-22T14:41:34Z

This PR resolves PHIRE-1827

Description

Builds on the work @araujobarret did to get maestro tests running.

Adds a few basic end to end tests using maestro on iOS simulators, running nightly for now.
Adds react native launch args and some init code to enable quick log in and sign out

Here is a successful test run:
https://app.circleci.com/pipelines/github/artsy/eigen/70100/workflows/6de254f2-5ab2-4417-a12c-b071d1bead5b/jobs/244401

Follow-ups

we should allow running on demand, via label, branch name, some other means?
Fix the checkbox fragile point based tap
Add a linkTo launch arg to quickly jump to screens for faster testing
Slack notifications on failures
Android testing

PR Checklist

I have tested my changes on iOS and Android.
I hid my changes behind a feature flag, or they don't need one.
I have included screenshots or videos, or I have not changed the UI.
I have added tests, or my changes don't require any.
I added an app state migration, or my changes do not require one.
I have documented any follow-up work that this PR will require, or it does not require any.
I have added a changelog entry below, or my changes do not require one.

To the reviewers 👀

I would like at least one of the reviewers to run this PR on the simulator or device.

Changelog updates

Cross-platform user-facing changes

iOS user-facing changes

Android user-facing changes

Dev changes

add first end to end tests - brian, george

Need help with something? Have a look at our docs, or get in touch with us.

Co-authored-by: George <[email protected]>

src/app/store/AuthModel.ts

brainbicycle · 2025-05-30T14:56:07Z

run_maestro_shard.sh

+  if [ $? -ne 0 ]; then
+    EXIT_CODE=1
+  fi
+done


split tests across circle nodes, tests need to be run and exit code tracked to avoid swallowing and passing even when tests fail

brainbicycle · 2025-05-30T14:57:10Z

src/app/system/devTools/useMaestroInitialization.ts

+    } else if (shouldSignOut && isLoggedIn) {
+      GlobalStore.actions.auth.signOut()
+    }
+  }, [])


allow quick log in for faster tests, log out to make sure app is in logged out state, next we can add a linkTo arg to allow quicker feature testing

brainbicycle · 2025-05-30T14:57:40Z

yarn.lock

+  version "4.0.4"
+  resolved "https://registry.yarnpkg.com/react-native-launch-arguments/-/react-native-launch-arguments-4.0.4.tgz#5cdf12265b50bf98ad05c87f849da99e07ef5500"
+  integrity sha512-cB7Sy9m9MX5MvNJliSHEMFWRScSbr95gFuGWnaINBoIK9sP8hloKqhn0vKUHrMQGmNsC1PcpyeiniBqTiU9d5g==
+


since we are reusing the beta build this needs to be a proper included dep, but it is small

ArtsyOpenSource · 2025-06-03T22:31:38Z

This PR contains the following changes:

Dev changes (add first end to end tests - brian, george - brainbicycle)

Generated by 🚫 dangerJS against e8051af

MounirDhahri

LGTM! I am happy we are finally doing this.

The more use this gets in the future, the more we would have to think to run these tests only on merge to main, but that's a problem for another day.

ios/Artsy/Networking/API_Modules/ArtsyAPI+DeviceTokens.m

MounirDhahri · 2025-06-04T07:37:37Z

src/app/system/devTools/useMaestroInitialization.ts

+    const args = LaunchArguments.value<MaestroLaunchArguments>()
+    const email = args.email
+    const password = args.password
+    const shouldSignOut = args.shouldSignOut
+
+    if (email && password) {
+      GlobalStore.actions.auth.signIn({


nice one 😄
I think I will add a helper for this on my CLI for quick login

src/app/system/devTools/useMaestroInitialization.ts

araujobarret

Thanks for keeping it rolling, so cool to see it running on the CI, and it's green consistently from the past runs!

Started a discussion about reusing the account for some flows, we can move on with the current approach but wanted to raise awareness of that 🦺

e2e/config.yml

e2e/set_env_vars.sh

fastlane/utility_fastlane.rb

install_app_in_booted_sims.sh

araujobarret · 2025-06-04T07:42:05Z

src/app/system/devTools/useMaestroInitialization.ts

+      GlobalStore.actions.auth.signOut()
+    }
+  }, [isHydrated])
+}


I understand the reasoning of this to speed up the tests 100%, timewise it's super expensive, but I'm more defensive of reusing accounts for some tasks because of shared state between test scenarios.
For example, using account X, we enter test scenario 1, which fails and leaves the account in a bad state. Consequently, test scenario 2 also fails due to the current state left by test scenario 1.
Reusing state in e2e tests usually introduces flakiness, so I wonder if we really should do it 🤔
It's a tradeoff of speed vs flakiness, I don't think it's a blocker but my fear is the pipeline gets flaky at some point and we just start ignoring it because the noise 🦺

i hear the concern and definitely don't want flaky tests either but I am not sure reusing accounts is reasonably avoidable, we are gonna add a huge time cost to every scenario and it will make writing tests more costly because you will need to wait for manual sign up every time or at least account for both signed out and signed in state in your flow writing.

We could possibly:

have multiple accounts for different flows

have some guidelines about relying on account state for tests

have nightly jobs to reset account states or even create new accounts in staging for tests

I am in agreement we want to avoid flakiness as much as possible though.

Yeah, it takes a lot of time just from one account to get to a point where you can go somewhere and test the flow.

I like the idea of having multiple accounts for different flows, about the guidelines is very broad imo, all dev, QA engineers creating scenarios should be aware to avoid any reusage of other scenarios' state, any overflow should be mitigated, but I don't know how to recommend that.
For example, let's say a test where we create a convo, and we have another scenario that opens a convo and reply to something there, the other scenario shouldn't rely on the top conversation to reply, so these kind of things are flags to us reviewers to observe when reviewing any e2e PR, not only Maestro's, Ingetrity too 👍

We can keep the idea of helpers to reset the account state(follows, etc), maybe more stuff to the dev menu so we can have some cleanup when a scenario finishes or breaks, we can evolve this idea as we go, I believe.

Thanks for the response, as I said I liked the ideas you suggested, I think it's def gonna reduce any flakiness level here 💯

araujobarret · 2025-06-04T07:44:09Z

e2e/flows/signup.yml

+- runFlow:
+    commands:
+      - tapOn:
+          point: 15%,36%


been there, done that 😢
I can help poking around our checkboxes to make them fully accessible to use with maestro if you want, I totally forgot about it 🫠

I actually haven't tried updating this since way back, I think George made some changes here recently, will check if there is already a better way

I did make the checkboxes accessible already

i checked in maestro studio and still have the problem that all accessible touchables lead to link opening rather than checkbox getting checked, will take a look after this pr

brainbicycle added 6 commits January 22, 2025 12:57

update sign up flow for updated design

9929a99

accesibility labels for easier testing

c92bc80

assert on welcome screen to make wait reliably

57d459a

update login flow

b1eb0f4

get rid of broken license check for now

4196325

attempt to install maestro in ci

a47bea1

brainbicycle self-assigned this Jan 22, 2025

brainbicycle added hackathon hack15 labels Jan 22, 2025

brainbicycle marked this pull request as draft January 22, 2025 14:41

brainbicycle and others added 20 commits January 22, 2025 15:46

attempt to run a test in ci

16f3df8

attempt with s3 download for now

655e178

need to check out project

4edfc54

try using m2 pro

d1d1b2b

ignore maestro folder for now

d637806

Co-authored-by: George <[email protected]>

convenience script for installing app in booted sims

1712a5d

Co-authored-by: George <[email protected]>

move flows to top level for now

92c37c8

Co-authored-by: George <[email protected]>

attempt to run tests in parallel in ci

312d109

Co-authored-by: George <[email protected]>

increase timeout for maestro tests

7bca89b

Co-authored-by: George <[email protected]>

basic deeplink testing

70bee59

try using xcode 16.2

9f03980

add missing idb install

8c8af6e

don't clear state for deeplink tests to make more reliable

b958392

try just one flow again

c003aa8

try just booting one simulator

2a9d3ed

try parallelizing using circle parallelism

a700b69

Co-authored-by: George <[email protected]>

duplicate assertion in deeplink test

41df5b6

fix yml matching

b307641

add launch args dep and convenience methods for sign in and sign out

9fe0b4f

Co-authored-by: George <[email protected]>

use signOut method instead of buggy clearState

cf2fdb9

Co-authored-by: George <[email protected]>

brainbicycle commented May 30, 2025

View reviewed changes

src/app/store/AuthModel.ts Outdated Show resolved Hide resolved

brainbicycle commented May 30, 2025

View reviewed changes

brainbicycle added 10 commits May 30, 2025 12:47

check to avoid crash when native module called early

acfde67

bring back native module code, wait on hydration

4d8533b

tie e2e to beta ios builds for now

981f23c

fix zip file creation

7051f2d

update version number

d35f9cc

add active arch flag to speed up sim build

2ec1c41

upgrade to xcode 16.2 and 18.2 sim

afee5ec

upload sim builds to s3

3ba4a50

make e2e dependent on test app rather than betas

8a849a6

clean up

5cb2371

brainbicycle marked this pull request as ready for review June 3, 2025 22:29

brainbicycle requested review from MounirDhahri, araujobarret and gkartalis June 3, 2025 22:29

MounirDhahri previously approved these changes Jun 4, 2025

View reviewed changes

araujobarret previously approved these changes Jun 4, 2025

View reviewed changes

typo in script

4b8960c

brainbicycle dismissed stale reviews from araujobarret and MounirDhahri via 4b8960c June 4, 2025 14:00

brainbicycle added 3 commits June 4, 2025 13:54

pr review: remove hardcoded appId

06115c8

pr review: move script to utils

5a489b1

pr review: move env vars to file

e8051af

brainbicycle merged commit f95b8f1 into main Jun 4, 2025
7 checks passed

brainbicycle deleted the brian/e2e-again branch June 4, 2025 20:06

gkartalis mentioned this pull request Jul 11, 2025

fix(attempt): attempt to fix flashy splashscreen #12460

Closed

10 tasks

chore(tests): end to end testing again #11428

chore(tests): end to end testing again #11428

Uh oh!

Conversation

brainbicycle commented Jan 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Follow-ups

PR Checklist

To the reviewers 👀

Changelog updates

Cross-platform user-facing changes

iOS user-facing changes

Android user-facing changes

Dev changes

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ArtsyOpenSource commented Jun 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

This PR contains the following changes:

Uh oh!

MounirDhahri left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

araujobarret left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

brainbicycle Jun 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

brainbicycle commented Jan 22, 2025 •

edited

Loading

ArtsyOpenSource commented Jun 3, 2025 •

edited

Loading

brainbicycle Jun 4, 2025 •

edited

Loading