llama 3.1 405b compliance checker issue

[yanghong5]

Dear MLPerf Training WG,

We’re seeing a consistent off-by-one failure from the compliance checker on opt_learning_rate_decay_steps for the Llama3.1-405B benchmark under ruleset 5.1.0. Details below.

What we ran: Benchmark: LLAMA31_405B (closed)

Ruleset: 5.1.0

Effective global batch size (GBS): 1008

Reported warmup steps: 9143

Reported decay steps (logged): 1362287

Checker expectation for decay steps: 1362286

Result: checker FAIL (by exactly 1)

Error: INFO - Running compliance on file: /results/251006072002009929661_1.log
INFO - Compliance checks: training_5.1.0/common.yaml
INFO - Compliance checks: training_5.1.0/closed_common.yaml
INFO - Compliance checks: training_5.1.0/closed_llama31_405b.yaml
WARNING - Required AT_LEAST_ONE occurrence of 'cache_clear' but found 0

CHECK for 'opt_learning_rate_decay_steps' failed in line 22:
:::MLLOG {"namespace": "", "time_ms": 1759749837494, "event_type": "POINT_IN_TIME", "key": "opt_learning_rate_decay_steps", "value": 1362287, "metadata": {"file": "/workspace/llm/pretrain.py", "lineno": 601}}
failed test: v['value'] == math.ceil(1_200_000 * 1152 / s['global_batch_size'] ) - s['opt_learning_rate_warmup_steps']
current context[s]={
"global_batch_size": 1008,
"opt_learning_rate_warmup_steps": 9143
}
current line[v]={
"metadata": {
"file": "/workspace/llm/pretrain.py",
"lineno": 601
},
"value": 1362287
}
ERROR - FAILED
INFO - Running compliance on file: /results/251006072002009929661_1.log
INFO - Compliance checks: training_5.1.0/common.yaml
INFO - Compliance checks: training_5.1.0/closed_common.yaml
INFO - Compliance checks: training_5.1.0/closed_llama31_405b.yaml
WARNING - Required AT_LEAST_ONE occurrence of 'cache_clear' but found 0

The model training itself is fine; we are wondering if it is a logging math issue?

Thank you for your help!

Best,
Yang Hong

AI Support | Research Computing
UFIT | University of Florida

[ShriyaRishab]

Can you try changing the computation in your code to math.ceil(1_200_000 / s['global_batch_size'] * 1152 ) - s['opt_learning_rate_warmup_steps'] instead and see if it ends up logging 1362286. Basically first divide 1.2M by the GBS before multiplying by 1152.

When I do the math manually, the checker expectation is right so it does seem like a math issue in your submission codebase.

[yanghong5]

Thank you, @ShriyaRishab! We just submitted a new job and will see if we can pass the checker this time.

[yanghong5]

Hi @ShriyaRishab, thank you so much for your help!

Indeed, we updated the code to opt_lr_decay_steps = ceil(os.environ.get(“OPT_LR_DECAY_STEPS”, 0) * 1152.0/config.model.global_batch_size - config.model.optim.sched.warmup_steps) and it passed the checker:

INFO - Running compliance on file: /results/251006215410269743445_1.log
INFO - Running compliance on file: /results/251006215410269743445_1.log
INFO -  Compliance checks: training_5.1.0/common.yaml
INFO -  Compliance checks: training_5.1.0/common.yaml
INFO -  Compliance checks: training_5.1.0/closed_common.yaml
INFO -  Compliance checks: training_5.1.0/closed_llama31_405b.yaml
INFO - SUCCESS”

However, we no longer have the resources to rerun the large-node jobs three more times. Our cluster is a production system that serves the university. In order to run the benchmarks, we had to make a special reservation for the weekend, plus Monday, to minimize the service disruption to our users. The reservation has expired this morning.

All of our less-node runs for the Llama-405B model used the updated code and passed the compliance check, but the earlier more-node runs still fail the checker due to the opt_lr_decay_steps +1 rounding difference. Because this discrepancy is purely a mathematical issue that does not affect training behavior or results, we would like to confirm whether it is acceptable to either (1) ignore the opt_learning_rate_decay_steps compliance error for those large-node runs, or (2) manually adjust the recorded opt_learning_rate_decay_steps value in the logs so that they pass the checker.

Best,
Yang Hong

AI Support | Research Computing
UFIT | University of Florida

[ShriyaRishab]

Hi Yang, that's understandable. Also, you are not required to mention the system or scale that you are submitting before the submission deadline so feel free to modify your message if you want to :)

Also, you can do (2) by manually adjusting the logs and just create an issue against yourself in the submission repo linking to this conversation thread so the review committee is aware of what was manually updated and why. I don't see this being an issue.

Adding review chairs as well so they are aware - @hiwotadese @pbaumstarck

[yanghong5]

Hi Shriya,

Thank you so much for your help!

We will adjust the logs and create an issue in the submission repo once we submit our results.

Also, I have removed node details from our previous chat history. :)

Best,
Yang