Skip to content

Experiments: Update documentation and remove logged-in A/A test #8041

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Nov 7, 2022
Merged

Experiments: Update documentation and remove logged-in A/A test #8041

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
839dd72
Document the advice about experiment exposure events
rachelmcr Nov 4, 2022
24f51ff
Remove logged-in A/A test
rachelmcr Nov 4, 2022
b438fd2
Merge branch 'trunk' into issue/8004-logged-in-aa-test-followup
rachelmcr Nov 7, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 1 addition & 6 deletions Experiments/Experiments/ABTest.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,6 @@ public enum ABTest: String, CaseIterable {
/// `An enum with no cases cannot declare a raw type`
case null

/// A/A test for ExPlat integration in the logged in state.
/// Experiment ref: pbxNRc-1QS-p2
///
case aaTestLoggedIn202210 = "woocommerceios_explat_aa_test_logged_in_202210"

/// A/A test to make sure there is no bias in the logged out state.
/// Experiment ref: pbxNRc-1S0-p2
case aaTestLoggedOut = "woocommerceios_explat_aa_test_logged_out_202211"
Expand All @@ -35,7 +30,7 @@ public enum ABTest: String, CaseIterable {
/// When adding a new experiment, add it to the appropriate case depending on its context (logged-in or logged-out experience).
public var context: ExperimentContext {
switch self {
case .aaTestLoggedIn202210, .productsOnboardingBanner:
case .productsOnboardingBanner:
return .loggedIn
case .aaTestLoggedOut, .abTestLoginWithWPComOnly:
return .loggedOut
Expand Down
10 changes: 10 additions & 0 deletions docs/EXPERIMENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,3 +38,13 @@ if ABTest.experimentName.variation == .control {
// Treatment logic
}
```
### Experiment Design Considerations

If your experiment uses an exposure event, watch out for race conditions between the exposure event and the test assignment or experiment metrics. The exposure event must be triggered _after_ the app receives the test assignment from the experiments endpoint, and _before_ any of the experiment metrics are triggered.

Due to potential race conditions, do not use these as exposure events:

- `woocommerceios_application_opened`
- `woocommerceios_application_installed`

When designing a logged-out experiment, consider that you may not need an exposure event at all.