[BUG] fp16 optimizers: log overflow/scale messages on rank 0 only

[udsy19]

Fixes #1533

Problem

FP16_Optimizer (fused_optimizer.py) and FP16_UnfusedOptimizer
(unfused_optimizer.py) call logger.info() directly for grad-overflow and
dynamic-loss-scale-change messages inside step(), step_fused_adam()/
step_fused_lamb(), and _update_scale(). Every rank prints the same line on
every occurrence instead of only rank 0, so on a many-node run these flood the
logs with hundreds of duplicate lines and drown out real signal, exactly as
reported.

Impact: contract-violation. Who reaches it / triggered by: any multi-GPU or
multi-node training run that enables fp16 with dynamic loss scaling (the
default fp16 config) and hits a grad overflow or a scale change — which
happens routinely during normal fp16 training, not just on error. What is
observed: every rank prints the identical "Grad overflow on iteration",
"Reducing/Increasing dynamic loss scale", and "fp16 dynamic loss scale
overflow! Skipping step" lines, instead of only rank 0. This repo already
established rank-0-only logging as the correct behavior for this exact
message class — the sibling "Overflow detected" message in
FP16_Optimizer.step() was fixed this way in #416, and the ZeRO path's
DynamicLossScaler.update_scale() gates every one of its own overflow
messages on dist.get_rank() == 0 — so this PR just closes the same gap in
the two files the issue names.

Fix

Route the remaining messages through the existing log_dist(..., ranks=[0])
helper instead of logger.info(), matching the precedent above.

Testing

Added test_overflow_logs_only_on_rank_zero (parametrized over both
optimizer classes) in
tests/unit/runtime/half_precision/test_dynamic_loss_scale.py. It builds a
minimal CPU optimizer, forces an overflow with a NaN gradient, and fakes
dist.get_rank() to simulate rank 0 vs. rank 1 (no GPU or real multi-process
job needed to observe this logging behavior).

Negative control (revert only the two production files to upstream/master,
keep the new test): fails without the fix —

FAILED ...test_overflow_logs_only_on_rank_zero[FP16_Optimizer] - AssertionError: fp16 overflow/loss-scale messages must not be logged on non-zero ranks
FAILED ...test_overflow_logs_only_on_rank_zero[FP16_UnfusedOptimizer] - AssertionError: fp16 overflow/loss-scale messages must not be logged on non-zero ranks
2 failed

Restored: 2 passed. Full test_dynamic_loss_scale.py: unchanged pre-existing
results before/after this diff (2 passed here plus 3 pre-existing
environment failures needing a build toolchain this sandbox lacks —
identical on unmodified upstream/master).

yapf/flake8/check-torchdist.py/check-license.py clean on all changed
files.

Signed-off-by: Udaya Tejas udayatejas2004@gmail.com

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-16T22:10:07.186188Z	7792d57	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.