[Test] E2E Nemotron-3-Super tests#36803
[Test] E2E Nemotron-3-Super tests#36803roikoren755 wants to merge 3 commits intovllm-project:mainfrom
Conversation
Signed-off-by: Roi Koren <roik@nvidia.com>
Signed-off-by: Roi Koren <roik@nvidia.com>
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request adds end-to-end tests for Nemotron-3-Super models in BF16, FP8, and NVFP4 formats. The changes include new Buildkite pipeline steps and corresponding test configuration files. My review found a potential issue with inconsistencies between filenames and the model_name specified within the new YAML configuration files. This could lead to confusion or incorrect model testing and should be addressed.
| @@ -0,0 +1,11 @@ | |||
| model_name: "nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16" | |||
There was a problem hiding this comment.
There's an inconsistency between the filename, which contains A20B, and the model_name value, which contains A12B. To maintain consistency and prevent potential confusion, they should match. If A20B is correct, please update the model name as suggested. However, if A12B is the correct model identifier, please rename this file accordingly and update its reference in tests/evals/gsm8k/configs/models-nemotron-3-super.txt.
model_name: "nvidia/NVIDIA-Nemotron-3-Super-120B-A20B-BF16"| @@ -0,0 +1,11 @@ | |||
| model_name: "nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-FP8" | |||
There was a problem hiding this comment.
There's an inconsistency between the filename, which contains A20B, and the model_name value, which contains A12B. To maintain consistency and prevent potential confusion, they should match. If A20B is correct, please update the model name as suggested. However, if A12B is the correct model identifier, please rename this file accordingly and update its reference in tests/evals/gsm8k/configs/models-nemotron-3-super.txt.
model_name: "nvidia/NVIDIA-Nemotron-3-Super-120B-A20B-FP8"| @@ -0,0 +1,11 @@ | |||
| model_name: "nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4" | |||
There was a problem hiding this comment.
There's an inconsistency between the filename, which contains A20B, and the model_name value, which contains A12B. To maintain consistency and prevent potential confusion, they should match. If A20B is correct, please update the model name as suggested. However, if A12B is the correct model identifier, please rename this file accordingly and update its reference in tests/evals/gsm8k/configs/models-nemotron-3-super-b200.txt.
model_name: "nvidia/NVIDIA-Nemotron-3-Super-120B-A20B-NVFP4"
Signed-off-by: Roi Koren <roik@nvidia.com>
| - label: Nemotron-3 Super 120B GSM8K (B200) | ||
| timeout_in_minutes: 120 | ||
| device: b200 | ||
| optional: true | ||
| num_devices: 4 |
There was a problem hiding this comment.
We only have B200 runners with 2 devices, so please update to that config
| - label: Nemotron-3 Super 120B GSM8K (H200) | ||
| timeout_in_minutes: 120 | ||
| device: h200 | ||
| optional: true | ||
| num_devices: 4 |
There was a problem hiding this comment.
I think our h200 resource is still disabled, only does 8 devices. I think you should be able to run on 4 H100 though
2 participants
Purpose
Adding 3 E2E tests for Nemotron-3-Super, in BF16, FP8 and NVFP4, with speculative decoding.
Test Plan
Three new tests pass.
Test Result
They do 🎉
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.