Commit 71335f7
authored
fix(examples): unblock Megatron TP notebook on GPU E2E (#3434)
* fix(e2e): bump Megatron notebook Complete-wait timeout to 10m
The Megatron TP notebook waits for the TrainJob to reach Complete with
timeout=120. On the oracle-vm-gpu-a10-1 CI runner the happy-path time
from TrainJob creation to Complete is ~96s (measured on the last passing
GPU E2E run, 2026-04-15). Any runner slowdown, image-pull delay, or
GPU-advertisement latency on top of that pushes the test past the 120s
budget and papermill raises TimeoutError, even though the TrainJob is
still on track to finish.
Every GPU E2E run has been failing here since 2026-04-16 ~16:00 UTC on
every branch, with no functional repo change between the last pass and
first fail. Bumping to 600s gives enough headroom for cold image pulls
and scheduling variance without masking real failures (papermill's
outer --execution-timeout is 1800s).
Signed-off-by: XploY04 <2004agarwalyash@gmail.com>
* fix(examples): update Megatron tokenizer library to null-text for 0.17.0
megatron-core 0.17.0 (published 2026-04-16 20:22 UTC) tightened the set
of accepted tokenizer library names in MegatronTokenizer.from_pretrained.
The bare "null" value is no longer accepted; the null-tokenizer keys are
now "null-text" and "null-multimodal":
0.16.1: if library not in ['byte-level', 'null']: assert tokenizer_path
0.17.0: if library not in ['byte-level', 'null-text', 'null-multimodal']:
assert tokenizer_path
The notebook's call with metadata_path={"library": "null"} therefore
triggers `AssertionError: Tokenizer path must be specified.` on the GPU
E2E runner, which now installs 0.17.0 by default. Renaming to
"null-text" routes through the same NullTokenizer(vocab_size) library
class the old "null" key used, so behavior is unchanged on 0.17.0.
This explains why GPU E2E ran green on 2026-04-15 (installed 0.16.1)
and started failing with the tokenizer assertion once 0.17.0 landed on
PyPI. The previous commit's 120s -> 600s wait timeout bump is what made
this assertion visible; with the old 120s budget the notebook timed out
before the TrainJob reached Failed and the real error was hidden.
Signed-off-by: XploY04 <2004agarwalyash@gmail.com>
---------
Signed-off-by: XploY04 <2004agarwalyash@gmail.com>1 parent 9c598fb commit 71335f7
1 file changed
Lines changed: 2 additions & 2 deletions
Lines changed: 2 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
180 | 180 | | |
181 | 181 | | |
182 | 182 | | |
183 | | - | |
| 183 | + | |
184 | 184 | | |
185 | 185 | | |
186 | 186 | | |
| |||
397 | 397 | | |
398 | 398 | | |
399 | 399 | | |
400 | | - | |
| 400 | + | |
401 | 401 | | |
402 | 402 | | |
403 | 403 | | |
| |||
0 commit comments