Skip to content

Conversation

@rachelmcr
Copy link
Contributor

Closes: #8002

Description

This PR adds the A/B test for the products onboarding banner, and removes it from behind the feature flag.

The banner will only be visible to eligible users in the A/B test treatment group, and only after the A/B test is started on the backend. (The banner is also visible to eligible a8c users in the treatment group while the test is in staging.)

Changes

  • Adds the products onboarding banner experiment.
  • Replaces the feature flag check with a check for the A/B test treatment group.
  • Adds MockABTesting for unit testing A/B test behavior:
    • setVariation(_:for:) lets us mock an experiment assignment in unit tests by manually setting the provided variation (control or treatment) for the given experiment.
    • This new method is used in our unit tests so we can still check that the products onboarding banner takes precedence when the user is eligible to see it.
  • Removes the unit test that was used for checking the banner behavior behind a feature flag.

Testing

Prerequisite (before launching the app)

  1. Open the experiment details in Abacus (linked at the top of pbxNRc-26F-p2 or ping me for the direct link).
  2. In the Audience section of the experiment setup, go to Variations > Manual Assignment and use the bookmarklet to assign yourself to the treatment group.

Test the treatment behavior

  1. Build and run the app. Note: You must be logged in to the app with an a8c account to test the experiment assignment/behavior while the experiment is in staging. If you are logged in to a different account, log out and log in again with your a8c account.
  2. Select a store with no products.
  3. Confirm you see the products onboarding banner on the My Store dashboard.

Test the control behavior

  1. Build and run the app.
  2. Log out and log in with a non-a8c test account. (Alternately, use the prerequisite steps to assign yourself to the control group to use your a8c account. While the test is in staging, manually assigning yourself to the control group or logging in with a non-a8c account will show you the control behavior.)
  3. Select a store with no products.
  4. Confirm you don't see any banner on the My Store dashboard.

Submitter Checklist

Update release notes:

  • I have considered if this change warrants user-facing release notes and have added them to RELEASE-NOTES.txt if necessary.

@rachelmcr rachelmcr added the feature: products onboarding Related to onboarding new users to manage products label Nov 1, 2022
@rachelmcr rachelmcr added this to the 11.1 milestone Nov 1, 2022
@wpmobilebot
Copy link
Collaborator

wpmobilebot commented Nov 1, 2022

You can test the changes from this Pull Request by:
  • Clicking here or scanning the QR code below to access App Center
  • Then installing the build number pr8003-5440aad on your iPhone

If you need access to App Center, please ask a maintainer to add you.

@rachelmcr rachelmcr marked this pull request as ready for review November 1, 2022 18:41
@Ecarrion Ecarrion self-assigned this Nov 2, 2022
Copy link
Contributor

@Ecarrion Ecarrion left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

One thing that happened to me is that I had to log out and log in in order for the variation status to reflect on the app.

I'd assume that only happens because of the staging environment, right?

@rachelmcr
Copy link
Contributor Author

I'd assume that only happens because of the staging environment, right?

Yeah, it's because of how we change the assigned variation while testing the staged experiment. Normally users will be get a single assignment for the experiment and it won't change, so they won't experience that.

@peril-woocommerce
Copy link

Warnings
⚠️ This PR is assigned to a milestone which is closing in less than 2 days Please, make sure to get it merged by then or assign it to a later expiring milestone

Generated by 🚫 dangerJS

@rachelmcr rachelmcr enabled auto-merge November 2, 2022 14:42
@rachelmcr rachelmcr merged commit e2e4a4b into trunk Nov 2, 2022
@rachelmcr rachelmcr deleted the issue/8002-onboarding-banner-experiment branch November 2, 2022 15:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature: products onboarding Related to onboarding new users to manage products

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Products Onboarding: Add the banner experiment to the app

4 participants