Repository navigation
Conversation
Proposes one mechanism for putting a share of visitors on a toggle, for A/B tests and for percentage rollouts of feature flags and phases. A CloudFront Function on viewer-request assigns visitors before the cache lookup, with running allocations held in a CloudFront KeyValueStore. The current Lambda@Edge assigns at origin-request, which only runs on a cache miss, so cohorts follow the cache and not the visitor. The RFC records a spike against the staging site that confirmed the replacement works with our cache key. For wellcomecollection/wellcomecollection.org#13836, refs wellcomecollection/wellcomecollection.org#13785.
The first version described a test and a rollout as one object with a percentage dial. They answer different questions: a test measures whether a change is better, and a rollout releases something already decided. The RFC now proposes one shared assignment mechanism with tests and rollouts as separate things on it. A test needs a plan before it can start (hypothesis, primary metric, planned sample or duration, success criteria), its percentages are fixed while it measures, a change needs a recorded reason and starts a new analysis period, and it ends with a recorded decision. A rollout ramps freely and is not a measurement. For wellcomecollection/wellcomecollection.org#13836.
Removes the percentage rollout from the proposal. What the beta testing ticket wants from a share of visitors is data and a comparison, which is an A/B test, and a test's percentages are fixed while it runs. The RFC now says so and no longer specifies a second thing with a percentage that rises. A test's variant can be a value of a feature flag or a phase of a phased flag, which is how a new phase is shown to a share of visitors with a control group. A gradual release mechanism is recorded under alternatives as not needed now. The folder is renamed to 094-ab-testing to match the title. For wellcomecollection/wellcomecollection.org#13836.
Readers may not know the two terms. Adds a background section that defines each, shows worked numbers, and says why the control group is not the same as visitors outside the test. For wellcomecollection/wellcomecollection.org#13836.
The section on fixed percentages named the paradox without a source. Links a worked example of a mid-test split change reversing the pooled result.
| Exposure to a test is reported with a new event that follows [Google's convention for experiment integrations](https://developers.google.com/analytics/devguides/collection/ga4/integration): an `experience_impression` event with an `exp_variant_string` parameter. | ||
|
|
||
| - The event is sent when a component that depends on a test renders, through a small hook that wraps `useABTest`. A visitor who is assigned to a test but never reaches the page it changes is not counted as exposed. | ||
| - The value has the form `wc-TESTID.vVERSION-COHORT`, for example `wc-newTags.v2-variant`, which fits GA's 100-character limit on parameter values. The version identifies the analysis period. |
There was a problem hiding this comment.
worth nothing that the limit is for all the test names appended together. Tbf we never have any, but wanted to clarify it wasn't a 100 char limit PER test.
|
|
||
| This is the cautious reading. The ICO's [guidance on storage and access technologies](https://ico.org.uk/for-organisations/direct-marketing-and-privacy-and-electronic-communications/guidance-on-the-use-of-storage-and-access-technologies/what-are-the-exceptions/) lists A/B testing as an example likely to meet the statistical purposes exception, which would allow enrolling visitors without consent provided they are told and can object. That exception has conditions, and the guidance does not discuss a cookie that keeps a visitor's cohort. Enrolling only consenting visitors costs a test nothing in measurement, since they are the only visitors GA records. It does mean that a test's percentages are shares of consenting visitors, and that non-consenting visitors never see a variant until it is released. | ||
|
|
||
| Kiosk devices are treated as consenting by the webapp. I propose excluding them from tests, because they show a different version of the site to a different audience. |
There was a problem hiding this comment.
We agreed Modes were not part of roll outs
Phased and Feature flags only.
Modes are not experiments to release, but permanent ways to use the website differently.
|
|
||
| [#13836](https://github.com/wellcomecollection/wellcomecollection.org/issues/13836) asks whether we want an alarm on the cache hit rate. I think we do, alongside an alarm on function errors, since both are ways this mechanism could quietly cost us. | ||
|
|
||
| ### Dashboard |
There was a problem hiding this comment.
wellcomecollection/wellcomecollection.org#13837 will require closing or a full rewrite depending on the outcome of the RFC - just flagging.
There was a problem hiding this comment.
I like it having its own page, it's getting busy in there.
|
|
||
| The function works from the list of running tests in the KeyValueStore. Each entry names a test, its exposure and its split. | ||
|
|
||
| The function keeps two cookies of its own on the visitor's browser. Neither is part of the cache key and neither is sent to the webapp. Both are set for `.wellcomecollection.org`, so the dashboard can read them, and both last six months from the visitor's most recent visit. |
There was a problem hiding this comment.
Why two cookies rather than one? I can guess — wc_ab_bucket has to stay stable while wc_ab is rewritten constantly, so a corrupt wc_ab can be dropped and rebuilt from the hash without moving anyone between versions — but the RFC doesn't say, and open question 5 asks about the lifetime of "the two cookies" as though the split is settled. A line here would help, especially since the cookie policy gains two entries instead of one.
Drafted with Claude Code.
That said I'm not 100% on what this looks like with two tests running in parallel. What is the actual format of wc_ab, could I see it with a two-test example/would we need parsing if it's the same name?
For wellcomecollection/wellcomecollection.org#13836, refs wellcomecollection/wellcomecollection.org#13785.
Preview
What does this change?
Adds RFC 094, which proposes rebuilding the A/B testing mechanism on wellcomecollection.org.
The beta testing ticket describes a percentage that rises over time alongside a comparison between visitors who have a feature and those who do not. The RFC treats the comparison as the need and meets it with a test whose percentages do not move. The section "How this relates to the beta testing tickets" lists each difference from those tickets and the reason.
rfcs/README.mdhas the listing row regenerated by.scripts/create_table_summary.py.How to test
python3 .scripts/validate_rfc.pypython3 .scripts/create_table_summary.py --check-readmeHow can we measure success?
No runtime criteria; this is a documentation RFC. Success is agreement from the Digital Experience team and the analyst on the mechanism and on fixed percentages, and an answer to the consent question, which is the next step that needs a decision from outside the two teams.
Have we considered potential risks?
Nothing changes at runtime. The RFC's own risks are in its Impact section: a fault in the function affects every page, and the beta testing tickets need revising because the RFC does not deliver a percentage that rises over time.
Note
I'd like views in review on whether a test with fixed percentages meets what the beta testing work needs.