Fix async trainers resuming past the epoch target - #7036
Open
DaoyuanLi2816 wants to merge 1 commit into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Fixes #7035.
When an async trainer resumes from a checkpoint whose saved prompt position already reaches the configured epoch target, the epoch-stop callback currently runs only after the first optimizer step. This PR applies the follow-up recommended in #6823 (comment) to both async trainers:
Regression coverage verifies that both
AsyncGRPOTrainerandAsyncDistillationTrainerstop before delegating to the base epoch loop when checkpoint progress already meets the target.Validation:
python -m pytest tests/experimental/test_async_grpo_trainer.py::TestEpochStopOnResume tests/experimental/test_async_grpo_trainer.py::TestRolloutStateCheckpoint tests/experimental/test_async_distillation_trainer.py::TestEpochStop tests/experimental/test_async_distillation_trainer.py::TestRolloutStateCheckpoint -q --no-cov— 16 passedpre-commit run ruff-checkon the four changed files — passedpre-commit run ruff-formaton the four changed files — passedgit diff --check— passedThe existing GPU end-to-end GRPO epoch test was not runnable in this local environment because Transformers requires the optional compatible
kernelspackage for its Flash Attention implementation.Before submitting
AI writing disclosure
Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag members/contributors who may be interested in your PR.
Note
Medium Risk
Touches training-loop control flow and rollout-worker startup on resume for both async trainers; behavior change is narrow (already-at-target checkpoints) but affects when vLLM sync and rollouts run.
Overview
Fixes async GRPO and distillation trainers continuing (and starting the rollout worker) after resume when checkpoint progress already satisfies
num_train_epochs._EpochStopCallbacknow runs the same target check aton_train_beginas at step end (via shared_check_target), using resumed_groups_before_resume/_prompts_before_resume. The callback is registered before_TrainBeginCallbackso the stop flag is set before cold weight sync and worker startup;_TrainBeginCallbackno-ops whenshould_training_stopis already true._run_epochreturns immediately without entering the first batch, because the base trainer only reads the stop flag after the epoch loop returns.Regression tests assert both trainers set
should_training_stopat train begin and never delegate to the base_run_epochwhen resume count already equals the epoch target.Reviewed by Cursor Bugbot for commit 6be8d40. Bugbot is set up for automated code reviews on this repo. Configure here.