[RFC] Criteria for removing code from `trl.experimental`

[albertvillanova]

Context

#4223 moved most trainers into trl.experimental with an explicit two-part plan: move first, and "discuss later which one should be promoted, which one should stay in experimental, and which one should be removed."

The first part happened. The second never did. #4223 is closed, and since then removals have been decided one PR at a time: #6235, #6309, #7020, and #7136 currently in flight. Each thread re-argues the same policy question in front of a different, largely self-selected audience, and each one produces a precedent that nobody agreed to in advance.

This RFC is to settle the policy once, so that a maintainer can apply it without reopening the debate, and so that anyone relying on something in trl.experimental knows in advance what would put it at risk.

Two populations arrived under different terms

trl.experimental currently holds 27 modules, and lumping them together is what makes these threads difficult.

Demoted in the #4223 wave (Oct to Dec 2025): bco, xpo, cpo, gkd, nash_md, orpo, prm, online_dpo. These had users on the stable API who were moved under "may change or be removed in any release" without being asked.

Incubated directly into experimental: bema_for_ref_model, grpo_with_replay_buffer, gspo_token, gold, minillm, async_grpo, sdft, sdpo, ssd, tpo, openreward, a2po, gmpo, harbor, iw_opd, server_distillation, async_distillation. These accepted that contract when they landed. Some are trainers, some are callbacks or integrations, and the criteria below may need to differ for the non-trainers.

kto and distillation are import shims left from completed promotions, not candidates.

The starting proposal is that the second group can be removed with no notice, which is exactly the contract they arrived under, and that the first group is where the question is genuinely open.

Axes to agree on

Rather than argue module by module, the proposal is to agree the bar and let the list follow from it.

1. Time without feature work. At least N releases in trl.experimental with only upkeep commits.
2. Usage, and how we read it. Both signals we have under-count in the same direction, so they are not independent confirmations of each other. Telemetry only covers instantiations since Add telemetry to trainers #5758, is disabled under CI and HF_HUB_OFFLINE, and for demoted trainers only ever observes the period after we delisted them from the docs. Hub tag counts miss everything unpublished. Filters matter too: for BCO, dropping device=cpu rows takes ~70 to ~10.
3. External investment. Open issues or PRs from outside the maintainer team that represent real use rather than bug-hunting.
4. Downstream consumers. Checked rather than assumed.
5. Triage surface. Unmaintained code attracts machine-generated findings, and the cost is maintainer attention per report regardless of user impact. BCOTrainer fails before the first step on any encoder-decoder model #6996 and Fix the BCO encoder-decoder path #6999 are the current example: an open issue and an open +145/-16 fix on a BCO code path the reporter says nobody is waiting on.
6. Notice. Whether a removal cycle applies, to which population, and in what form.

Questions

1. Are these the right axes, and what should N be?

2. Do demoted trainers get notice before removal, and if so what kind? A release-notes line is cheap. A FutureWarning naming a target version is stronger but commits us to running a cycle: picking the version, landing the warning, remembering it next release, chasing it when a release slips. Worth deciding once here rather than per PR, since the per-PR cost is most of the objection to doing it at all.

3. Paper implementations. We accept these into trl.experimental and require a paper_index.md entry for them. If a paper implementation can be removed when the paper does not gain traction, contributors should be told that when their PR is merged, not two years later. What do we want to promise them?

4. Machine-generated findings. This is now a significant share of the load, and it is not specific to any one trainer: it lands wherever code is unmaintained, which is most of trl.experimental. Removing modules one at a time removes one surface at a time. Followed to its conclusion, if report volume is what decides whether code stays, then whoever runs the most agents is choosing what TRL keeps. Do we want a triage policy for reports against experimental code alongside the removal bar?

5. If you rely on anything listed above, please say so here, including workloads that are never published and so appear in neither signal. This thread is the measurement our metrics cannot provide, and it is the most useful thing anyone outside the maintainer team can add.

Scope and process

This RFC decides the policy and the candidate list. It does not decide individual removals, and cleanup already in flight is not blocked on it. Whatever we agree gets written into experimental_overview.md, which currently documents only the promotion path and says nothing about how anything leaves, so that the next removal does not start this discussion again.

CC: @huggingface/trl