Move the latest release tests from T4 to L40S - #7184
Open
albertvillanova wants to merge 1 commit into
Open
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR runs the daily latest-release tests on the same GPU runner as the rest of the GPU CI.
Motivation
tests_latest.ymlstayed onaws-g4dn-2xlarge(T4) when the other GPU workflows moved toaws-g6e-4xlarge(L40S) in #6741: the release branch in use at the time (v1.10-release) predated the flash-attn2 xfail markers and would have failed on Ada hardware. It is now the only GPU workflow left on T4.Both halves of that reason are gone. The workflow runs against
v1.13-release, and the xfail markers were removed in #7057 once the padding-free tests passed.Solution
On T4,
is_ampere_or_newer()skips the two padding-free tests, so the move makes them run:tests/test_dpo_trainer.py::TestDPOTrainer::test_train_padding_freeandtests/test_sft_trainer.py::TestSFTTrainer::test_train_padding_free. Both are identical onv1.13-releaseandmain, and both pass on L40S in theTests with dev dependenciesjob oftests.yml, which uses the same container, runner and dependency set as this workflow.The other tests gated on
is_ampere_or_newer()are unaffected: the GRPO continuous batching test is marked slow, and the experimental ones are excluded from collection bynorecursedirs.Changes
aws-g6e-4xlargeinstead ofaws-g4dn-2xlargeNote
Low Risk
CI-only runner change with no application code touched; expanded test execution matches the existing dev-dependencies GPU job that already passes on the same hardware.
Overview
Aligns the daily Tests latest TRL release with dev dependencies workflow with the rest of GPU CI by switching the runner from
aws-g4dn-2xlarge(T4) toaws-g6e-4xlarge(L40S) and dropping the outdated TODO that blocked the move.On L40S,
is_ampere_or_newer()is true, so the job will execute the padding-free DPO and SFT trainer tests that were previously skipped on T4; other Ampere-gated tests stay unchanged (slow GRPO batching, experimental paths excluded from collection).Reviewed by Cursor Bugbot for commit b547b23. Bugbot is set up for automated code reviews on this repo. Configure here.