Update logged-out A/A test name and re-add logged-in A/A test #8250
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes: #8235
Description
Since we had an attempted fix for the previous A/A tests where there is a bias toward the control variant, I created another pair of A/A tests (logged-in/logged-out) so that we can see if the bias issue is fixed.
A note on the experiment name:
woocommerceios_explat_aa_test_logged_out_202212andwoocommerceios_explat_aa_test_logged_in_202212experiment names were already taken by @selanthiraiyan's previous PRs that were closed. I re-cloned the experiments with*_v2names since we can't reuse the same experiment name.Testing instructions
Prerequisite: a sandbox is required for testing manual assignments on A/B experiments, you can request one if you don't have one - it'd be useful for testing IAP later as well.
There are no visual differences for the A/A tests, you can test it by setting a breakpoint in the logged-out and logged-in state.
Logged-out state
Prerequisites:
ifconfig) in the local matchine's/etc/hostsExPlatServiceon the lineurlComponents.queryItems?.append(URLQueryItem.init(name: "anon_id", value: anonId))and launching the appPATCH /wpcom/v2/experiments/0.1.0/assignmentswith the followingbodyand make sure it's successful:StoreCreationCoordinator.startGet Startedor any other actions that trigger the breakpoint set in the first step. When the breakpoint is hit, check the A/A test variant bypo ABTest.aaTestLoggedOut.variationin the debug console --> it should saytreatment: nilLogged-in state
Prerequisites:
PATCH /wpcom/v2/experiments/0.1.0/assignmentswith the followingbodyand the bearer token for the wpcom account (I'd use an A8C account in this case), make sure it's successful:HubMenuwhen an element is tappedpo ABTest.aaTestLoggedIn.variationin the debug console --> it should saytreatment: nilRELEASE-NOTES.txtif necessary.