Skip to content

RFC 094: A/B testing on wellcomecollection.org - #183

Open
kenoir wants to merge 5 commits into
mainfrom
rk-rfc-094-ab-testing
Open

kenoir wants to merge 5 commits into
mainfrom
rk-rfc-094-ab-testing

Conversation

@kenoir

@kenoir kenoir commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

For wellcomecollection/wellcomecollection.org#13836, refs wellcomecollection/wellcomecollection.org#13785.

Preview


What does this change?

Adds RFC 094, which proposes rebuilding the A/B testing mechanism on wellcomecollection.org.

  • Today's assignment runs in a Lambda@Edge at origin-request, which only fires on a cache miss. Visitors who get a cached page are given the cohort of whoever caused the miss, so past test results were not from a randomised sample.
  • The proposal moves assignment to a CloudFront Function on viewer-request, before the cache lookup, with running tests held in a CloudFront KeyValueStore. A spike on a temporary distribution in front of the staging site confirmed this works with our cache key.
  • A test needs a written plan before it starts, its percentages are fixed while it measures, and it ends with a recorded decision. Its variant can be a value of a feature flag or a phase of a phased flag.

The beta testing ticket describes a percentage that rises over time alongside a comparison between visitors who have a feature and those who do not. The RFC treats the comparison as the need and meets it with a test whose percentages do not move. The section "How this relates to the beta testing tickets" lists each difference from those tickets and the reason.

rfcs/README.md has the listing row regenerated by .scripts/create_table_summary.py.

How to test

  • Read the rendered RFC.
  • python3 .scripts/validate_rfc.py
  • python3 .scripts/create_table_summary.py --check-readme

How can we measure success?

No runtime criteria; this is a documentation RFC. Success is agreement from the Digital Experience team and the analyst on the mechanism and on fixed percentages, and an answer to the consent question, which is the next step that needs a decision from outside the two teams.

Have we considered potential risks?

Nothing changes at runtime. The RFC's own risks are in its Impact section: a fault in the function affects every page, and the beta testing tickets need revising because the RFC does not deliver a percentage that rises over time.

Note

I'd like views in review on whether a test with fixed percentages meets what the beta testing work needs.

Proposes one mechanism for putting a share of visitors on a toggle, for
A/B tests and for percentage rollouts of feature flags and phases. A
CloudFront Function on viewer-request assigns visitors before the cache
lookup, with running allocations held in a CloudFront KeyValueStore.

The current Lambda@Edge assigns at origin-request, which only runs on a
cache miss, so cohorts follow the cache and not the visitor. The RFC
records a spike against the staging site that confirmed the replacement
works with our cache key.

For wellcomecollection/wellcomecollection.org#13836, refs
wellcomecollection/wellcomecollection.org#13785.
The first version described a test and a rollout as one object with a
percentage dial. They answer different questions: a test measures
whether a change is better, and a rollout releases something already
decided. The RFC now proposes one shared assignment mechanism with
tests and rollouts as separate things on it.

A test needs a plan before it can start (hypothesis, primary metric,
planned sample or duration, success criteria), its percentages are
fixed while it measures, a change needs a recorded reason and starts a
new analysis period, and it ends with a recorded decision. A rollout
ramps freely and is not a measurement.

For wellcomecollection/wellcomecollection.org#13836.
Removes the percentage rollout from the proposal. What the beta testing
ticket wants from a share of visitors is data and a comparison, which
is an A/B test, and a test's percentages are fixed while it runs. The
RFC now says so and no longer specifies a second thing with a
percentage that rises.

A test's variant can be a value of a feature flag or a phase of a
phased flag, which is how a new phase is shown to a share of visitors
with a control group. A gradual release mechanism is recorded under
alternatives as not needed now.

The folder is renamed to 094-ab-testing to match the title.

For wellcomecollection/wellcomecollection.org#13836.
@kenoir kenoir changed the title RFC 094: A/B tests and percentage rollouts on wellcomecollection.org RFC 094: A/B testing on wellcomecollection.org Nov 16, 2026
Readers may not know the two terms. Adds a background section that
defines each, shows worked numbers, and says why the control group is
not the same as visitors outside the test.

For wellcomecollection/wellcomecollection.org#13836.
The section on fixed percentages named the paradox without a source.
Links a worked example of a mid-test split change reversing the pooled
result.
@kenoir
kenoir marked this pull request as ready for review November 16, 2026 16:00
@kenoir
kenoir requested review from a team as code owners November 16, 2026 16:00
Exposure to a test is reported with a new event that follows [Google's convention for experiment integrations](https://developers.google.com/analytics/devguides/collection/ga4/integration): an `experience_impression` event with an `exp_variant_string` parameter.

- The event is sent when a component that depends on a test renders, through a small hook that wraps `useABTest`. A visitor who is assigned to a test but never reaches the page it changes is not counted as exposed.
- The value has the form `wc-TESTID.vVERSION-COHORT`, for example `wc-newTags.v2-variant`, which fits GA's 100-character limit on parameter values. The version identifies the analysis period.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

worth nothing that the limit is for all the test names appended together. Tbf we never have any, but wanted to clarify it wasn't a 100 char limit PER test.


This is the cautious reading. The ICO's [guidance on storage and access technologies](https://ico.org.uk/for-organisations/direct-marketing-and-privacy-and-electronic-communications/guidance-on-the-use-of-storage-and-access-technologies/what-are-the-exceptions/) lists A/B testing as an example likely to meet the statistical purposes exception, which would allow enrolling visitors without consent provided they are told and can object. That exception has conditions, and the guidance does not discuss a cookie that keeps a visitor's cohort. Enrolling only consenting visitors costs a test nothing in measurement, since they are the only visitors GA records. It does mean that a test's percentages are shares of consenting visitors, and that non-consenting visitors never see a variant until it is released.

Kiosk devices are treated as consenting by the webapp. I propose excluding them from tests, because they show a different version of the site to a different audience.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We agreed Modes were not part of roll outs

Phased and Feature flags only.
Modes are not experiments to release, but permanent ways to use the website differently.


[#13836](https://github.com/wellcomecollection/wellcomecollection.org/issues/13836) asks whether we want an alarm on the cache hit rate. I think we do, alongside an alarm on function errors, since both are ways this mechanism could quietly cost us.

### Dashboard

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wellcomecollection/wellcomecollection.org#13837 will require closing or a full rewrite depending on the outcome of the RFC - just flagging.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like it having its own page, it's getting busy in there.


The function works from the list of running tests in the KeyValueStore. Each entry names a test, its exposure and its split.

The function keeps two cookies of its own on the visitor's browser. Neither is part of the cache key and neither is sent to the webapp. Both are set for `.wellcomecollection.org`, so the dashboard can read them, and both last six months from the visitor's most recent visit.

@rcantin-w rcantin-w Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why two cookies rather than one? I can guess — wc_ab_bucket has to stay stable while wc_ab is rewritten constantly, so a corrupt wc_ab can be dropped and rebuilt from the hash without moving anyone between versions — but the RFC doesn't say, and open question 5 asks about the lifetime of "the two cookies" as though the split is settled. A line here would help, especially since the cookie policy gains two entries instead of one.
Drafted with Claude Code.

That said I'm not 100% on what this looks like with two tests running in parallel. What is the actual format of wc_ab, could I see it with a two-test example/would we need parsing if it's the same name?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants