Skip to content

Fix LR schedulers when warmup/decay rounds to zero#655

Open
xsinxx wants to merge 1 commit into
allenai:mainfrom
xsinxx:feat/warmup_adjust
Open

Fix LR schedulers when warmup/decay rounds to zero#655
xsinxx wants to merge 1 commit into
allenai:mainfrom
xsinxx:feat/warmup_adjust

Conversation

@xsinxx

@xsinxx xsinxx commented Apr 9, 2026

Copy link
Copy Markdown

Summary This PR makes LR schedulers robust to warmup=0 / decay=0 (and fraction-based warmup/decay that rounds to 0). Previously, these configurations could lead to division-by-zero in the linear warmup/decay helpers.

Changes

  • Treat warmup=0 as “no warmup” and avoid entering linear warmup when warmup == 0 :

    • ConstantWithWarmup
    • WSD
  • Treat decay=0 as “no decay” and avoid entering linear decay when decay == 0 :

    • WSD
    • CosWithWarmupAndLinearDecay
  • Handle WSDS period decay steps D == 0 by returning decay_min_lr directly to avoid division-by-zero.
    Tests

  • Added unit tests covering:

    • ConstantWithWarmup(warmup=0)
    • ConstantWithWarmup(warmup_fraction=...) that rounds to zero warmup
    • WSD(warmup=0, decay=0)
  • Ran: python -m pytest -q src/test/optim/scheduler_test.py (all passed)
    Behavioral Impact

  • No behavior change for configurations with warmup > 0 and decay > 0 .

  • Configurations with warmup=0 / decay=0 now behave as “disabled phase” instead of raising due to division-by-zero.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant