Attempt to add rank max learning rate logic into in-loop-evals PR#586
Draft
piperwolters wants to merge 6 commits into
Draft
Attempt to add rank max learning rate logic into in-loop-evals PR#586piperwolters wants to merge 6 commits into
piperwolters wants to merge 6 commits into
Conversation
This was a one-off harness used to launch the 400k v1.2 + GeoBench-2 in-loop eval validation run, not part of the feature itself. The in-loop eval functionality lives in loop_eval_launch.py, evaluator_callback.py, and wandb.py. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
checkpoint_sweep_evals builds a MockOlmoEarthDataLoader to satisfy the trainer interface, but it hardcoded dp_world_size=1. The trainer validates the loader's DP world size against the process-group world size, so a rank-max LR eval job launched with >1 GPU failed with: OLMoConfigurationError: data loader's DP world size ... got 1, expected 8 Read the world size/rank from the distributed context instead (no-op for single-process jobs, where get_world_size() returns 1). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…/olmoearth_pretrain into piperw/in-loop-evals-test
Collaborator
|
comment from @pjreddie - since we run the evals in a separate job now, can we sweep learning rates but do it sequentially, so this only requires a single GPU ? This might make the evals very slow to run though |
Contributor
Author
Hmm good point. Is there usually a "rush" to get results when sweeping learning rates, or is it more of an after-thought once we've found a good model? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Based on tests, this is seeming to run fine with
run_as_beaker_jobTrue or False. When True, each eval job is auto launched with rank_max_lr_num_gpus (default is 8). If False, the GPU amount just uses the training world size.Logic taken from: #505
Testing in progress here: