ci(vllm): add nemotron-nano-12b-v2 smoke test + split sagemaker endpoint config (#6349)

* ci(vllm): add nemotron-nano-12b-v2 to smoke tests

Adds NVIDIA-Nemotron-Nano-12B-v2 (Mamba2-Transformer hybrid, 12B params)
to the vLLM smoke-test matrix on the g6e.xl (L40S 48GB) fleet.

Requires vllm>=0.10.1 for hybrid Mamba2 support; the DLC vllm:server-cuda-v2
image ships vllm 0.24.0+amzn2023 which satisfies this. Uses
--mamba_ssm_cache_dtype float32 (required by the model card to preserve
accuracy) and --trust-remote-code for the custom configuration_nemotron_h.py.

Verified end-to-end on g6e.2xlarge: model loads as NemotronHForCausalLM,
Mamba2 SSD Triton kernels warm up, KV cache pool 369K tokens, both
/think reasoning-on and /no_think reasoning-off modes return correct
output. Steady-state GPU memory ~42 GiB out of 46 GiB.

Signed-off-by: Yadan Wei <yadanwei@amazon.com>

* ci(vllm): split sagemaker endpoint tests into vllm-sagemaker-tests.yml

Extracts the sagemaker: section from vllm-model-tests.yml into a
dedicated vllm-sagemaker-tests.yml so smoke-test and endpoint-test
configs evolve independently. Each file now owns its own s3_prefix and
test_fixtures_prefix.

- test_sm_model_serving.py now reads from vllm-sagemaker-tests.yml
- vllm.pr-amzn2023.yml and vllm.pr-ubuntu.yml add the new file to the
  top-level paths trigger and to sagemaker-test-change, so edits to
  endpoint entries fire the sagemaker suite (not the smoke suite)
- parse_model_config.py and vllm.tests-model.yml need no changes; they
  only read smoke-test:, and its file path is unchanged

Signed-off-by: Yadan Wei <yadanwei@amazon.com>

* ci(vllm): rename vllm-sagemaker-tests to vllm-sagemaker-endpoint-tests

Everything under .github/config/model-tests/ is a model test config.
"sagemaker-tests" was ambiguous — the smoke-test file also runs
SageMaker-image smoke tests. Rename to make the distinction explicit:
this file drives real SageMaker endpoint tests
(test_sm_model_serving.py), the sibling drives smoke tests.

Follow-up references updated in the sibling YAML comment, both PR
workflow triggers, and the sagemaker test loader.

Signed-off-by: Yadan Wei <yadanwei@amazon.com>

---------

Signed-off-by: Yadan Wei <yadanwei@amazon.com>
Co-authored-by: Yadan Wei <yadanwei@amazon.com>

Author: Yadan-Wei

.github/config/model-tests/vllm-model-tests.yml

@@ -1,14 +1,12 @@
-# vLLM Model Test Configuration
-# Smoke-test and SageMaker endpoint test suites.
-#
-# smoke-test:  model serving + inference validation
-# sagemaker:   config-driven real SageMaker endpoint tests
-#
-# Benchmark (throughput/latency) configs were split out into
-# vllm-text-benchmark-tests.yml and vllm-multimodal-benchmark-tests.yml.
+# vLLM Model Smoke Test Configuration
 #
+# Smoke tests: model serving + inference validation.
 # smoke-test has codebuild-fleet and runner-scale-sets sub-keys.
 # Workflow parsers construct s3_path from s3_prefix + s3_model.
+#
+# SageMaker endpoint tests live in the sibling vllm-sagemaker-endpoint-tests.yml.
+# Benchmark (throughput/latency) configs live in
+# vllm-text-benchmark-tests.yml and vllm-multimodal-benchmark-tests.yml.
 
 s3_prefix: "s3://dlc-cicd-models/llm-models"
 test_fixtures_prefix: "s3://dlc-cicd-models/test-fixtures"
@@ -39,6 +37,17 @@ smoke-test:
       fleet: "x86-g6exl-runner"
       extra_args: "--tensor-parallel-size 1 --max-model-len 4096 --dtype bfloat16 --reasoning-parser qwen3"
 
+    # NVIDIA Nemotron-Nano-12B-v2: Mamba2-Transformer hybrid (NemotronHForCausalLM),
+    # 12B params bf16 (~24GB weights). Requires vllm>=0.10.1 for hybrid support.
+    # Fits a single L40S (g6e.xl, 48GB) at tp=1 with ~42GB steady-state (23% headroom).
+    # --mamba_ssm_cache_dtype float32 is required by the model card — without it,
+    # SSM cache defaults to bf16 and accuracy degrades. --trust-remote-code is needed
+    # to load the custom configuration_nemotron_h.py from the HF repo.
+    - name: "nemotron-nano-12b-v2"
+      s3_model: "nemotron-nano-12b-v2.tar.gz"
+      fleet: "x86-g6exl-runner"
+      extra_args: "--tensor-parallel-size 1 --max-model-len 4096 --dtype bfloat16 --trust-remote-code --mamba_ssm_cache_dtype float32"
+
     - name: "qwen3-embedding-0.6b"
       s3_model: "qwen3-embedding-0.6b.tar.gz"
       fleet: "x86-g6xl-runner"
@@ -84,169 +93,3 @@ smoke-test:
           validate: "json_field:text"
 
   runner-scale-sets: []
-
-# --- SageMaker Endpoint Tests ---
-# Config-driven real endpoint tests. Each entry deploys a model from S3,
-# sends requests to the configured route, and validates the response.
-# Test script: test/vllm/sagemaker/test_sm_model_serving.py
-sagemaker:
-  # MiniCPM5-1B — dense 1B LlamaForCausalLM, bf16. Standard chat model served via
-  # the OpenAI /v1/chat/completions route. Validated offline on
-  # server-sagemaker-cuda-v2 with SM_VLLM_MAX_MODEL_LEN=8192. Hybrid reasoning
-  # emits a <think> block, so message.content is non-empty; json_field validation
-  # (non-empty check) mirrors the other entries' validate rule.
-  - name: "minicpm5-1b"
-    s3_model: "minicpm5-1b.tar.gz"
-    instance_type: "ml.g6.xlarge"
-    required_image_pattern: "amzn2023"
-    env:
-      SM_VLLM_MODEL: "/opt/ml/model"
-      SM_VLLM_MAX_MODEL_LEN: "8192"
-      SM_VLLM_DTYPE: "bfloat16"
-    test_cases:
-      - name: "chat-completion"
-        route: "/v1/chat/completions"
-        content_type: "application/json"
-        request:
-          body:
-            model: "/opt/ml/model"
-            messages:
-              - role: "user"
-                content: "What is the capital of France?"
-            max_tokens: 256
-            temperature: 0
-        validate: "json_field:choices.0.message.content"
-
-  - name: "voxtral-mini-4b"
-    s3_model: "voxtral-mini-4b.tar.gz"
-    instance_type: "ml.g6.xlarge"
-    required_image_pattern: "amzn2023"
-    env:
-      SM_VLLM_MODEL: "/opt/ml/model"
-      SM_VLLM_TOKENIZER_MODE: "mistral"
-      SM_VLLM_CONFIG_FORMAT: "mistral"
-      SM_VLLM_LOAD_FORMAT: "mistral"
-      SM_VLLM_ENFORCE_EAGER: "true"
-      SM_VLLM_MAX_MODEL_LEN: "8192"
-    test_cases:
-      - name: "transcription-en"
-        route: "/v1/audio/transcriptions"
-        content_type: "multipart/form-data"
-        request:
-          file_s3: "s3://dlc-cicd-models/test-fixtures/audio/asr_en.wav"
-          file_field: "file"
-          file_name: "test.wav"
-          file_content_type: "audio/wav"
-          fields:
-            model: "/opt/ml/model"
-        validate: "json_field:text"
-
-  - name: "granite-speech-4-1-2b"
-    s3_model: "granite-speech-4.1-2b.tar.gz"
-    instance_type: "ml.g6.xlarge"
-    required_image_pattern: "amzn2023"
-    env:
-      SM_VLLM_MODEL: "/opt/ml/model"
-      SM_VLLM_MAX_MODEL_LEN: "2048"
-    test_cases:
-      - name: "transcription-en"
-        route: "/v1/audio/transcriptions"
-        content_type: "multipart/form-data"
-        request:
-          file_s3: "s3://dlc-cicd-models/test-fixtures/audio/asr_en.wav"
-          file_field: "file"
-          file_name: "test.wav"
-          file_content_type: "audio/wav"
-          fields:
-            model: "/opt/ml/model"
-        validate: "json_field:text"
-
-  - name: "voxtral-mini-4b-network-isolated"
-    s3_model: "voxtral-mini-4b-isolated.tar.gz"
-    instance_type: "ml.g6.xlarge"
-    required_image_pattern: "amzn2023"
-    network_isolation: true
-    env:
-      SM_VLLM_MODEL: "/opt/ml/model"
-      SM_VLLM_TOKENIZER_MODE: "mistral"
-      SM_VLLM_CONFIG_FORMAT: "mistral"
-      SM_VLLM_LOAD_FORMAT: "mistral"
-      SM_VLLM_ENFORCE_EAGER: "true"
-      SM_VLLM_MAX_MODEL_LEN: "8192"
-      STANDARD_PIP_ARGS: "--no-index --find-links /opt/ml/model/requirements -r /opt/ml/model/requirements.txt"
-    test_cases:
-      - name: "transcription-en-isolated"
-        route: "/v1/audio/transcriptions"
-        content_type: "multipart/form-data"
-        request:
-          file_s3: "s3://dlc-cicd-models/test-fixtures/audio/asr_en.wav"
-          file_field: "file"
-          file_name: "test.wav"
-          file_content_type: "audio/wav"
-          fields:
-            model: "/opt/ml/model"
-        validate: "json_field:text"
-
-  - name: "granite-speech-4-1-2b-network-isolated"
-    s3_model: "granite-speech-4.1-2b-isolated.tar.gz"
-    instance_type: "ml.g6.xlarge"
-    required_image_pattern: "amzn2023"
-    network_isolation: true
-    env:
-      SM_VLLM_MODEL: "/opt/ml/model"
-      SM_VLLM_MAX_MODEL_LEN: "2048"
-      STANDARD_PIP_ARGS: "--no-index --find-links /opt/ml/model/requirements -r /opt/ml/model/requirements.txt"
-    test_cases:
-      - name: "transcription-en-isolated"
-        route: "/v1/audio/transcriptions"
-        content_type: "multipart/form-data"
-        request:
-          file_s3: "s3://dlc-cicd-models/test-fixtures/audio/asr_en.wav"
-          file_field: "file"
-          file_name: "test.wav"
-          file_content_type: "audio/wav"
-          fields:
-            model: "/opt/ml/model"
-        validate: "json_field:text"
-
-  # Qwen3-Reranker-4B — Path C: load as Qwen3ForSequenceClassification so the
-  # Cohere-compatible /v1/rerank endpoint registers. SM_VLLM_HF_OVERRIDES is
-  # passed verbatim to vLLM as --hf-overrides. The chat template is read from
-  # test/vllm/scripts/amzn2023/qwen3_reranker_chat_template.jinja (shared with
-  # the EC2 reranker scripts), flattened to single-line with {{ '\n' }} escapes
-  # at test runtime, and injected as SM_VLLM_CHAT_TEMPLATE. vLLM's
-  # --chat-template falls back to inline jinja for non-path values containing
-  # '{', '}', or newline; the {{ '\n' }} expressions render to real newlines
-  # at request time, reproducing the original template byte-for-byte.
-  - name: "qwen3-reranker-4b"
-    s3_model: "qwen3-reranker-4b.tar.gz"
-    instance_type: "ml.g6.xlarge"
-    required_image_pattern: "amzn2023"
-    chat_template_file: "test/vllm/scripts/amzn2023/qwen3_reranker_chat_template.jinja"
-    env:
-      SM_VLLM_MODEL: "/opt/ml/model"
-      SM_VLLM_RUNNER: "pooling"
-      # Plain JSON, no literal-single-quote wrapping. standard-supervisor (>=0.1.15,
-      # pinned in docker/vllm/Dockerfile.amzn2023) shlex.quote()s each argv element
-      # before the supervisord round-trip, so the inner double quotes survive intact
-      # and json.loads succeeds. Wrapping in single quotes here would pass them
-      # through verbatim and break json.loads on the leading quote.
-      SM_VLLM_HF_OVERRIDES: '{"architectures":["Qwen3ForSequenceClassification"],"classifier_from_token":["no","yes"],"is_original_qwen3_reranker":true}'
-      SM_VLLM_DTYPE: "bfloat16"
-      SM_VLLM_MAX_MODEL_LEN: "10000"
-      SM_VLLM_GPU_MEMORY_UTILIZATION: "0.85"
-    test_cases:
-      - name: "rerank-basic"
-        route: "/v1/rerank"
-        content_type: "application/json"
-        request:
-          body:
-            model: "/opt/ml/model"
-            query: "What is the capital of France?"
-            documents:
-              - "Paris is the capital of France."
-              - "Berlin is the capital of Germany."
-              - "The Eiffel Tower is in Paris."
-              - "Tokyo is the capital of Japan."
-        validate: "json_field:results.0.relevance_score"
-

.github/config/model-tests/vllm-sagemaker-endpoint-tests.yml

@@ -0,0 +1,178 @@
+# vLLM SageMaker Endpoint Test Configuration
+#
+# Config-driven real SageMaker endpoint tests. Each entry deploys a model
+# from S3, sends requests to the configured route, and validates the response.
+#
+# Test script: test/vllm/sagemaker/amzn2023/test_sm_model_serving.py
+#
+# Smoke-test entries live in the sibling vllm-model-tests.yml.
+
+s3_prefix: "s3://dlc-cicd-models/llm-models"
+test_fixtures_prefix: "s3://dlc-cicd-models/test-fixtures"
+
+sagemaker:
+  # MiniCPM5-1B — dense 1B LlamaForCausalLM, bf16. Standard chat model served via
+  # the OpenAI /v1/chat/completions route. Validated offline on
+  # server-sagemaker-cuda-v2 with SM_VLLM_MAX_MODEL_LEN=8192. Hybrid reasoning
+  # emits a <think> block, so message.content is non-empty; json_field validation
+  # (non-empty check) mirrors the other entries' validate rule.
+  - name: "minicpm5-1b"
+    s3_model: "minicpm5-1b.tar.gz"
+    instance_type: "ml.g6.xlarge"
+    required_image_pattern:
+      os_version: "amzn2023"
+    env:
+      SM_VLLM_MODEL: "/opt/ml/model"
+      SM_VLLM_MAX_MODEL_LEN: "8192"
+      SM_VLLM_DTYPE: "bfloat16"
+    test_cases:
+      - name: "chat-completion"
+        route: "/v1/chat/completions"
+        content_type: "application/json"
+        request:
+          body:
+            model: "/opt/ml/model"
+            messages:
+              - role: "user"
+                content: "What is the capital of France?"
+            max_tokens: 256
+            temperature: 0
+        validate: "json_field:choices.0.message.content"
+
+  - name: "voxtral-mini-4b"
+    s3_model: "voxtral-mini-4b.tar.gz"
+    instance_type: "ml.g6.xlarge"
+    required_image_pattern:
+      os_version: "amzn2023"
+    env:
+      SM_VLLM_MODEL: "/opt/ml/model"
+      SM_VLLM_TOKENIZER_MODE: "mistral"
+      SM_VLLM_CONFIG_FORMAT: "mistral"
+      SM_VLLM_LOAD_FORMAT: "mistral"
+      SM_VLLM_ENFORCE_EAGER: "true"
+      SM_VLLM_MAX_MODEL_LEN: "8192"
+    test_cases:
+      - name: "transcription-en"
+        route: "/v1/audio/transcriptions"
+        content_type: "multipart/form-data"
+        request:
+          file_s3: "s3://dlc-cicd-models/test-fixtures/audio/asr_en.wav"
+          file_field: "file"
+          file_name: "test.wav"
+          file_content_type: "audio/wav"
+          fields:
+            model: "/opt/ml/model"
+        validate: "json_field:text"
+
+  - name: "granite-speech-4-1-2b"
+    s3_model: "granite-speech-4.1-2b.tar.gz"
+    instance_type: "ml.g6.xlarge"
+    required_image_pattern:
+      os_version: "amzn2023"
+    env:
+      SM_VLLM_MODEL: "/opt/ml/model"
+      SM_VLLM_MAX_MODEL_LEN: "2048"
+    test_cases:
+      - name: "transcription-en"
+        route: "/v1/audio/transcriptions"
+        content_type: "multipart/form-data"
+        request:
+          file_s3: "s3://dlc-cicd-models/test-fixtures/audio/asr_en.wav"
+          file_field: "file"
+          file_name: "test.wav"
+          file_content_type: "audio/wav"
+          fields:
+            model: "/opt/ml/model"
+        validate: "json_field:text"
+
+  - name: "voxtral-mini-4b-network-isolated"
+    s3_model: "voxtral-mini-4b-isolated.tar.gz"
+    instance_type: "ml.g6.xlarge"
+    required_image_pattern:
+      os_version: "amzn2023"
+    network_isolation: true
+    env:
+      SM_VLLM_MODEL: "/opt/ml/model"
+      SM_VLLM_TOKENIZER_MODE: "mistral"
+      SM_VLLM_CONFIG_FORMAT: "mistral"
+      SM_VLLM_LOAD_FORMAT: "mistral"
+      SM_VLLM_ENFORCE_EAGER: "true"
+      SM_VLLM_MAX_MODEL_LEN: "8192"
+      STANDARD_PIP_ARGS: "--no-index --find-links /opt/ml/model/requirements -r /opt/ml/model/requirements.txt"
+    test_cases:
+      - name: "transcription-en-isolated"
+        route: "/v1/audio/transcriptions"
+        content_type: "multipart/form-data"
+        request:
+          file_s3: "s3://dlc-cicd-models/test-fixtures/audio/asr_en.wav"
+          file_field: "file"
+          file_name: "test.wav"
+          file_content_type: "audio/wav"
+          fields:
+            model: "/opt/ml/model"
+        validate: "json_field:text"
+
+  - name: "granite-speech-4-1-2b-network-isolated"
+    s3_model: "granite-speech-4.1-2b-isolated.tar.gz"
+    instance_type: "ml.g6.xlarge"
+    required_image_pattern:
+      os_version: "amzn2023"
+    network_isolation: true
+    env:
+      SM_VLLM_MODEL: "/opt/ml/model"
+      SM_VLLM_MAX_MODEL_LEN: "2048"
+      STANDARD_PIP_ARGS: "--no-index --find-links /opt/ml/model/requirements -r /opt/ml/model/requirements.txt"
+    test_cases:
+      - name: "transcription-en-isolated"
+        route: "/v1/audio/transcriptions"
+        content_type: "multipart/form-data"
+        request:
+          file_s3: "s3://dlc-cicd-models/test-fixtures/audio/asr_en.wav"
+          file_field: "file"
+          file_name: "test.wav"
+          file_content_type: "audio/wav"
+          fields:
+            model: "/opt/ml/model"
+        validate: "json_field:text"
+
+  # Qwen3-Reranker-4B — Path C: load as Qwen3ForSequenceClassification so the
+  # Cohere-compatible /v1/rerank endpoint registers. SM_VLLM_HF_OVERRIDES is
+  # passed verbatim to vLLM as --hf-overrides. The chat template is read from
+  # test/vllm/scripts/amzn2023/qwen3_reranker_chat_template.jinja (shared with
+  # the EC2 reranker scripts), flattened to single-line with {{ '\n' }} escapes
+  # at test runtime, and injected as SM_VLLM_CHAT_TEMPLATE. vLLM's
+  # --chat-template falls back to inline jinja for non-path values containing
+  # '{', '}', or newline; the {{ '\n' }} expressions render to real newlines
+  # at request time, reproducing the original template byte-for-byte.
+  - name: "qwen3-reranker-4b"
+    s3_model: "qwen3-reranker-4b.tar.gz"
+    instance_type: "ml.g6.xlarge"
+    required_image_pattern:
+      os_version: "amzn2023"
+    chat_template_file: "test/vllm/scripts/amzn2023/qwen3_reranker_chat_template.jinja"
+    env:
+      SM_VLLM_MODEL: "/opt/ml/model"
+      SM_VLLM_RUNNER: "pooling"
+      # Plain JSON, no literal-single-quote wrapping. standard-supervisor (>=0.1.15,
+      # pinned in docker/vllm/Dockerfile.amzn2023) shlex.quote()s each argv element
+      # before the supervisord round-trip, so the inner double quotes survive intact
+      # and json.loads succeeds. Wrapping in single quotes here would pass them
+      # through verbatim and break json.loads on the leading quote.
+      SM_VLLM_HF_OVERRIDES: '{"architectures":["Qwen3ForSequenceClassification"],"classifier_from_token":["no","yes"],"is_original_qwen3_reranker":true}'
+      SM_VLLM_DTYPE: "bfloat16"
+      SM_VLLM_MAX_MODEL_LEN: "10000"
+      SM_VLLM_GPU_MEMORY_UTILIZATION: "0.85"
+    test_cases:
+      - name: "rerank-basic"
+        route: "/v1/rerank"
+        content_type: "application/json"
+        request:
+          body:
+            model: "/opt/ml/model"
+            query: "What is the capital of France?"
+            documents:
+              - "Paris is the capital of France."
+              - "Berlin is the capital of Germany."
+              - "The Eiffel Tower is in Paris."
+              - "Tokyo is the capital of Japan."
+        validate: "json_field:results.0.relevance_score"