Retire three tiny model generators with no test consumer - #7186
Open
albertvillanova wants to merge 3 commits into
Open
Retire three tiny model generators with no test consumer#7186albertvillanova wants to merge 3 commits into
albertvillanova wants to merge 3 commits into
Conversation
`trl-internal-testing/tiny-Qwen3MoeForSequenceClassification` was only ever a tokenizer source in the chat template tests. #5469 replaced those uses with `tiny-Qwen3MoeForCausalLM`, which is the natural source for a Qwen3-MoE tokenizer, and left the generator behind with no consumer.
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three tiny models are generated and published but loaded by no test. This PR drops their generators.
Found by cross-referencing every
push_to_hubcall in the 64 generator scripts against the model ids appearing undertests/.tiny-Qwen2ForCausalLM-2.5-Codertests/, in any committiny-Qwen3MoeForSequenceClassificationtiny-Qwen3MoeForCausalLMtiny-PaliGemmaForConditionalGenerationWhy this is worth doing
An unused tiny model is not free. It is regenerated on every alignment pass, and each regeneration costs a Hub PR to open, review and merge, with no test that would notice if the result were broken.
It also makes the pre-merge check for a tiny model PR silently vacuous.
apply_model_revisionsintests/conftest.pyinjects a revision only when a test loads that exact id, so aMODEL_REVISIONSentry for an unused model never fires: the suite goes green whether or not the regenerated artefact is sound. That is what happened on #7134, where the entry would have had no effect and the run was skipped.Questions
CC: @huggingface/trl
Delete or revive? Deleting is the claim that these three models are not worth covering. The opposite reading is defensible, particularly for PaliGemma: it is a prefix-LM VLM, a shape no other tiny model in the suite has, and it was disabled from the moment it was added with no reason recorded, unlike the Idefics2 line above it which cites a transformers issue. If you would rather revive one, say which and I will drop that commit and open the coverage work separately. There is one commit per model, so any of the three can be dropped on its own.
What about the Hub repos? This PR only removes the generators. The repositories under
trl-internal-testingstay as they are. Options are to leave them, archive them, or delete them. Leaving them means an id that still resolves, with no script left to regenerate it, which is the state that produced this cleanup in the first place. I would rather not touch the org without a decision from you.Changes
for_causal_lm/qwen2_for_causal_lm_2_5_coder.pyfor_sequence_classification/qwen3_moe_for_sequence_classification.pyfor_conditional_generation/paligemma_for_conditional_generation.pyNote
Low Risk
Deletes maintenance-only scripts with no test consumers; no runtime or library behavior changes.
Overview
Removes three unused tiny-model generator scripts under
scripts/generate_tiny_models/: Qwen2.5-Coder causal LM, Qwen3 MoE sequence classification, and PaliGemma conditional generation. Each script built a shrunk config, ran smoke checks, and pushed to Hub viapush_to_hub, but no test in the suite loads those published ids anymore.This stops regenerating and opening Hub alignment PRs for models that nothing in CI would exercise. Hub repos are unchanged—only the local generators are deleted.
Reviewed by Cursor Bugbot for commit 0ed090d. Bugbot is set up for automated code reviews on this repo. Configure here.