-
Notifications
You must be signed in to change notification settings - Fork 121
[ABTesting] Cache variations of logged out context experiments #8806
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
selanthiraiyan
merged 25 commits into
release/12.1
from
bugfix/8782-cache-logged-out-ab-tests
Feb 3, 2023
Merged
Changes from 22 commits
Commits
Show all changes
25 commits
Select commit
Hold shift + click to select a range
5a9608e
Introduce `ABTestVariationProvider` for reading `ABTest` from Woo layer.
selanthiraiyan b8868f1
`AuthenticationManager` - Inject `ABTestVariationProvider` and use it…
selanthiraiyan 3813dc4
`AppCoordinator` - Inject `ABTestVariationProvider` and use it for re…
selanthiraiyan 1c5fdc8
`AppCoordinator` - Inject `ABTestVariationProvider` and use it for re…
selanthiraiyan c3414ad
Inject ABTest variation from unit tests.
selanthiraiyan 66a0321
Stop having `variation` as public to force usage of `ABTestVariationP…
selanthiraiyan 0033332
Unit test the REST API AB test tracks event value.
selanthiraiyan 4e7535e
Move `MockABTestVariationProvider` to a separate file for reusability.
selanthiraiyan 5ae2cda
Update unit test name to reflect the actual test.
selanthiraiyan b4b1582
Add unit tests to check the scenarios when application password authe…
selanthiraiyan 0c32dd1
Make ABTest confirm to Codable to store it to disk
selanthiraiyan 7c5a5cf
Introduce `VariationCache` to store the ABTest variation information …
selanthiraiyan ada4aa6
Add cached implementation of `ABTestVariationProvider`
selanthiraiyan f9107f3
Point Automattic tracks pod to beta version.
selanthiraiyan 99b0953
ABTest - Return nil instead of assuming control as variation.
selanthiraiyan dcca964
Add ExperimentsTests to unit test plan.
selanthiraiyan 9803a25
Start using `CachedABTestVariationProvider` instead of `DefaultABTest…
selanthiraiyan 8908527
Point tracks pod to latest beta commit.
selanthiraiyan e0d9ee5
Use Mock `ABTest` values for unit testing.
selanthiraiyan c4ce4ca
Explain why we cache only logged out experiments.
selanthiraiyan 737ee32
Avoid sending mock ABTest cases to server.
selanthiraiyan c0708bf
Give preference to remote value over cached value.
selanthiraiyan 40c08c3
Point `Automattic-Tracks-iOS` to latest beta version.
selanthiraiyan 527373f
Update podfile.lock.
selanthiraiyan d6df1c6
Point tracks pod to 2.0.0-beta.1
selanthiraiyan File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,16 @@ | ||
| import AutomatticTracks | ||
|
|
||
| /// For getting the variation of a `ABTest` | ||
| public protocol ABTestVariationProvider { | ||
| /// Returns the `Variation` for the provided `ABTest` | ||
| func variation(for abTest: ABTest) -> Variation | ||
| } | ||
|
|
||
| /// Default implementation of `ABTestVariationProvider` | ||
| public struct DefaultABTestVariationProvider: ABTestVariationProvider { | ||
| public init() { } | ||
|
|
||
| public func variation(for abTest: ABTest) -> Variation { | ||
| abTest.variation ?? .control | ||
| } | ||
| } |
30 changes: 30 additions & 0 deletions
30
Experiments/Experiments/CachedABTestVariationProvider.swift
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| import AutomatticTracks | ||
|
|
||
| /// Cache based implementation of `ABTestVariationProvider` | ||
| /// | ||
| public struct CachedABTestVariationProvider: ABTestVariationProvider { | ||
|
|
||
| private let cache: VariationCache | ||
|
|
||
| public init(cache: VariationCache = VariationCache(userDefaults: .standard)) { | ||
| self.cache = cache | ||
| } | ||
|
|
||
| public func variation(for abTest: ABTest) -> Variation { | ||
| // We cache only logged out ABTests as they are assigned based on `anonId` by `ExPlat`. | ||
| // There will be one value assigned to one device and it won't change. | ||
| // | ||
| guard abTest.context == .loggedOut else { | ||
| return abTest.variation ?? .control | ||
| } | ||
|
|
||
| if let variation = abTest.variation { | ||
| try? cache.assign(variation: variation, for: abTest) | ||
| return variation | ||
| } else if let cachedVariation = cache.variation(for: abTest) { | ||
| return cachedVariation | ||
| } else { | ||
| return .control | ||
| } | ||
| } | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,47 @@ | ||
| import AutomatticTracks | ||
|
|
||
| enum VariationCacheError: Error { | ||
| case onlyLoggedOutExperimentsShouldBeCached | ||
| } | ||
|
|
||
| public struct VariationCache { | ||
| private let variationKey = "VariationCacheKey" | ||
|
|
||
| private let userDefaults: UserDefaults | ||
|
|
||
| public init(userDefaults: UserDefaults) { | ||
| self.userDefaults = userDefaults | ||
| } | ||
|
|
||
| func variation(for abTest: ABTest) -> Variation? { | ||
| guard abTest.context == .loggedOut else { | ||
| return nil | ||
| } | ||
|
|
||
| guard let data = userDefaults.object(forKey: variationKey) as? Data, | ||
| let cachedVariations = try? JSONDecoder().decode([CachedVariation].self, from: data), | ||
| case let variation = cachedVariations.first(where: { $0.abTest == abTest })?.variation | ||
| else { | ||
| return nil | ||
| } | ||
|
|
||
| return variation | ||
| } | ||
|
|
||
| func assign(variation: Variation, for abTest: ABTest) throws { | ||
| guard abTest.context == .loggedOut else { | ||
| throw VariationCacheError.onlyLoggedOutExperimentsShouldBeCached | ||
| } | ||
|
|
||
| var variations = userDefaults.object(forKey: variationKey) as? [CachedVariation] ?? [] | ||
| variations.append(CachedVariation(abTest: abTest, variation: variation)) | ||
|
|
||
| let encodedVariation = try JSONEncoder().encode(variations) | ||
| userDefaults.set(encodedVariation, forKey: variationKey) | ||
| } | ||
| } | ||
|
|
||
| public struct CachedVariation: Codable { | ||
| let abTest: ABTest | ||
| let variation: Variation | ||
| } |
29 changes: 29 additions & 0 deletions
29
Experiments/ExperimentsTests/CachedABTestVariationProviderTests.swift
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,29 @@ | ||
| import XCTest | ||
| @testable import Experiments | ||
|
|
||
| final class CachedABTestVariationProviderTests: XCTestCase { | ||
| func test_variation_is_control_when_the_value_does_not_exist() throws { | ||
| // Given | ||
| let userDefaults = try XCTUnwrap(UserDefaults(suiteName: UUID().uuidString)) | ||
|
|
||
| // When | ||
| let cache = VariationCache(userDefaults: userDefaults) | ||
| let provider = CachedABTestVariationProvider(cache: cache) | ||
|
|
||
| // Then | ||
| XCTAssertEqual(provider.variation(for: .mockLoggedOut), .control) | ||
| } | ||
|
|
||
| func test_correct_variation_is_returned_after_caching_it() throws { | ||
| // Given | ||
| let userDefaults = try XCTUnwrap(UserDefaults(suiteName: UUID().uuidString)) | ||
| let cache = VariationCache(userDefaults: userDefaults) | ||
| let provider = CachedABTestVariationProvider(cache: cache) | ||
|
|
||
| // When | ||
| try cache.assign(variation: .treatment, for: .mockLoggedOut) | ||
|
|
||
| // Then | ||
| XCTAssertEqual(provider.variation(for: .mockLoggedOut), .treatment) | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,36 @@ | ||
| import XCTest | ||
| @testable import Experiments | ||
|
|
||
| final class VariationCacheTests: XCTestCase { | ||
| func test_variation_is_nil_when_the_value_does_not_exist() throws { | ||
| // Given | ||
| let userDefaults = try XCTUnwrap(UserDefaults(suiteName: UUID().uuidString)) | ||
|
|
||
| // When | ||
| let cache = VariationCache(userDefaults: userDefaults) | ||
|
|
||
| // Then | ||
| XCTAssertNil(cache.variation(for: .mockLoggedOut)) | ||
| } | ||
|
|
||
| func test_correct_variation_is_returned_after_setting_it() throws { | ||
| // Given | ||
| let userDefaults = try XCTUnwrap(UserDefaults(suiteName: UUID().uuidString)) | ||
| let cache = VariationCache(userDefaults: userDefaults) | ||
|
|
||
| // When | ||
| try cache.assign(variation: .treatment, for: .mockLoggedOut) | ||
|
|
||
| // Then | ||
| XCTAssertEqual(cache.variation(for: .mockLoggedOut), .treatment) | ||
| } | ||
|
|
||
| func test_it_throws_when_trying_to_cache_logged_in_experiment() throws { | ||
| // Given | ||
| let userDefaults = try XCTUnwrap(UserDefaults(suiteName: UUID().uuidString)) | ||
| let cache = VariationCache(userDefaults: userDefaults) | ||
|
|
||
| // When | ||
| XCTAssertThrowsError(try cache.assign(variation: .treatment, for: .mockLoggedIn)) | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit: should we leave a comment on why we are caching only for the logged-out context.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure if we need to restrict to logged-out experiments only 🤔 We are requesting only experiments for specific contexts so it should be safe to apply this for all contexts right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I hope you mean that we only request experiments for current contexts from the server. Yes, I agree.
But, I would still prefer only to cache logged-out experiments to minimize the impact of this change. Also, we don't face any crossover issues with logged-in experiments.
WDYT?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Makes sense, please go ahead with this solution 👍