Check endpoint method in nemo-evaluator does not allow vlm endpoint type

[athitten]

Describe the bug

check_endpoint method in nemo-evaluator's utils does not allow endpoint_type="vlm". This causes the error:

    check_endpoint(
  File "/opt/Evaluator/packages/nemo-evaluator/src/nemo_evaluator/core/utils.py", line 348, in check_endpoint
    raise ValueError(f"Invalid endpoint type: {endpoint_type}")
ValueError: Invalid endpoint type: vlm

Steps/Code to reproduce bug

Can be reproduced with this evaluator snippet

from nemo_evaluator.api import check_endpoint, evaluate
from nemo_evaluator.api.api_dataclasses import (
    ApiEndpoint,
    ConfigParams,
    EvaluationConfig,
    EvaluationTarget,
)

# Configure the evaluation target
api_endpoint = ApiEndpoint(
    url="http://0.0.0.0:1235/v1/chat/completions/",
    type="vlm",
    model_id="megatron-model",
)
eval_target = EvaluationTarget(api_endpoint=api_endpoint)
eval_params = ConfigParams(top_p=0, temperature=0, limit_samples=2, parallelism=1)
eval_config = EvaluationConfig(type="ocrbench", params=eval_params, output_dir="results")

if __name__ == "__main__":
    check_endpoint(
        endpoint_url=eval_target.api_endpoint.url,
        endpoint_type=eval_target.api_endpoint.type,
        model_name=eval_target.api_endpoint.model_id,
    )
    evaluate(target_cfg=eval_target, eval_cfg=eval_config)                                                           

Expected behavior

Support for endpoint_type "vlm" needs to be added to check_endpoint utility to avoid error when evaluating on vlm benchmarks.

Additional context

Add any other context about the problem here.

[marta-sd]

Good catch. I now realized we also don't have embedding there.

Side note: we're refactoring endpoint specification and vlm type will be removed soon. We will have only the basic types (completions, chat, embedding) and additionally tags indicating what capabilities a given benchmark requires. What we currently call vlm is actually a chat endpoint that accepts images, so we will use chat type in near future.

[marta-sd]

Update: in PR #792 I have implemented the change described above. I'll now verify if the check method works correctly for multi-modal endpoints with the new code.

[marta-sd]

I was testing the new version with NeMo FW and I hit an error - Qwen VL model fails when there are no images in the request. We need to fix this on the Export-Deploy side

[marta-sd]

Created a ticket to track it: NVIDIA-NeMo/Export-Deploy#632

[marta-sd]

I have applied the changes from PR NVIDIA-NeMo/Export-Deploy#634 on top of 26.02.00 container:

diff --git a/nemo_deploy/multimodal/megatron_multimodal_deployable.py b/nemo_deploy/multimodal/megatron_multimodal_deployable.py
index 3dc60d27c..ff2dabb49 100644
--- a/nemo_deploy/multimodal/megatron_multimodal_deployable.py
+++ b/nemo_deploy/multimodal/megatron_multimodal_deployable.py
@@ -139,7 +139,7 @@ class MegatronMultimodalDeployable(ITritonDeployable):
             tokenizer=self.processor.tokenizer,
             image_processor=self.processor.image_processor,
             prompts=prompts,
-            images=images,
+            images=images if images and len(images) > 0 else None,
             processor=self.processor,
             max_batch_size=max_batch_size,
             random_seed=random_seed,
diff --git a/nemo_deploy/multimodal/megatron_multimodal_deployable_ray.py b/nemo_deploy/multimodal/megatron_multimodal_deployable_ray.py
index 645545c71..5a17a87a3 100644
--- a/nemo_deploy/multimodal/megatron_multimodal_deployable_ray.py
+++ b/nemo_deploy/multimodal/megatron_multimodal_deployable_ray.py
@@ -217,6 +217,12 @@ class MegatronMultimodalRayDeployable:
             if not isinstance(messages, list):
                 prompts = [messages]
 
+            # Normalize content: "content" as string -> list of one text part
+            for message in prompts:
+                if isinstance(message.get("content"), str):
+                    message["content"] = [{"type": "text", "text": message["content"]}]
+
+
             # Normalize image_url format to image format for consistent processing
             for message in prompts:
                 for content in message["content"]:

With these changes I was able to successfully check endpoint readiness as described in our docs:

python /opt/Export-Deploy/scripts/deploy/multimodal/deploy_ray_inframework.py \
  --megatron_checkpoint /workspace/qwen2_5_vl_3b_instruct/ \
  --host 0.0.0.0 \
  --port 8886

[pruprakash]

QA RCCA Analysis

Date: 2026-04-20
Analyst: AI QA Agent (issues-rca skill)
Issue: #714 — Check endpoint method in nemo-evaluator does not allow vlm endpoint type

---

1. Fix Reference

Fix PR: #792 — Refactored endpoint type specification to remove vlm as a standalone type; VLM/multimodal endpoints now use endpoint_type="chat" plus capability tags. Closed 2026-04-07.

Additionally, Export-Deploy PR #634 was applied to fix a related Qwen VL failure (tracked via Export-Deploy #632), ensuring no-image requests do not crash on the model side.

---

2. Root Cause

The check_endpoint() function in nemo_evaluator/core/utils.py maintained a hardcoded allowlist of valid endpoint types. This list included "completions" and "chat" but not "vlm". When any caller passed endpoint_type="vlm", the function raised ValueError: Invalid endpoint type: vlm at the type-validation step (line ~348 in utils.py before the fix), before even attempting to reach the target endpoint. The root problem was that the allowlist was never updated when VLM support was added to the broader system, creating an inconsistency between the types the rest of the codebase could produce and the types the validation function would accept. A secondary omission (noted by a maintainer in comments) was that "embedding" was also missing from the allowlist.

---

3. Trigger Config

Trigger conditions (OR — any single one suffices):

• Passing endpoint_type="vlm" to check_endpoint() (directly or via the evaluate API)
• Passing endpoint_type="embedding" to check_endpoint() (secondary gap noted in comments)

Deterministic? Yes — the ValueError was raised deterministically on every call with these types, regardless of model, hardware, or network state.

---

4. Nature of Bug

Primary classification: API/interface bug — a valid user-facing endpoint type (vlm) was rejected by the validation function before any network call was made

Impact scope: All users running VLM (vision-language model) evaluations via nemo_evaluator.api.check_endpoint or evaluate. Any code path that calls check_endpoint with endpoint_type="vlm" is broken.

NOT affected: Text-only evaluations using endpoint_type="completions" or "chat", training workflows, model export.

---

5. Functional Test Coverage

Verdict: PARTIAL

Test	File	Key Config	What it covers
test_check_endpoint_importable	evaluator/testcases/rcca/core/test_vlm_endpoint_type.py	Unit (no GPU)	check_endpoint is importable from public API
test_valid_endpoint_types_include_chat	same	Unit (no GPU)	Source inspection: "chat" is in the valid endpoint types allowlist
test_valid_endpoint_types_include_completions	same	Unit (no GPU)	Source inspection: "completions" is in the allowlist
test_valid_endpoint_types_include_embedding	same	Unit (no GPU)	Source inspection: "embedding" is in the allowlist (secondary #714 gap)
test_check_endpoint_does_not_raise_for_chat_type	same	Unit (mocked HTTP)	check_endpoint("chat") does not raise ValueError
test_check_endpoint_vlm_type_handled_gracefully	same	Unit (mocked HTTP)	"vlm" either accepted or raises a helpful migration error pointing to "chat"

---

6. Gaps and Limitations

Gap 1 — No live VLM endpoint test (out of scope — hardware/model constraint):
Running an actual check_endpoint call against a live VLM model requires a deployed multimodal endpoint (Qwen VL, LLaVA, etc.) and typically GPU hardware. This cannot run in standard CI.

Gap 2 — Mocked HTTP tests cannot validate network-level behavior:
The test_check_endpoint_does_not_raise_for_chat_type test mocks requests.get/post to bypass real HTTP calls. It validates type-validation logic only, not end-to-end connectivity or response parsing.

Gap 3 — Capability tags system is partially opaque:
PR #792 introduced capability tags to replace the vlm type, but the test cannot fully verify that all VLM-specific capability tags are correctly wired through the evaluation pipeline without a live model.

Overall gap assessment: Medium regression risk. The structural tests catch a regression to the allowlist (e.g., if "chat" is removed or the ValueError is re-introduced). Live multimodal routing validation requires a deployed VLM endpoint outside CI scope.

---

7. New Test Added

Field	Value
Test file	evaluator/testcases/rcca/core/test_vlm_endpoint_type.py
Test functions	test_check_endpoint_importable, test_valid_endpoint_types_include_chat, test_valid_endpoint_types_include_completions, test_valid_endpoint_types_include_embedding, test_check_endpoint_does_not_raise_for_chat_type, test_check_endpoint_vlm_type_handled_gracefully
QA repo	nmfw_tests (local: evaluator/testcases/rcca/core/)
PR	Pending
What it validates	check_endpoint allowlist includes chat/completions/embedding; chat type does not raise ValueError; vlm type handled gracefully
How it would catch a regression	If "chat" is removed from the allowlist or check_endpoint re-introduces a bare ValueError for valid types, test_valid_endpoint_types_include_chat and test_check_endpoint_does_not_raise_for_chat_type will fail immediately.

---

8. Conclusion

Issue #714 was caused by check_endpoint() having a hardcoded allowlist that excluded "vlm" and "embedding" endpoint types, raising ValueError for any VLM evaluation attempt before any network call was made. The fix (PR #792) refactored the endpoint type system to replace "vlm" with "chat" plus capability tags, and a related Export-Deploy fix (PR #634) addressed a downstream Qwen VL crash. A new six-function unit test suite has been added to validate the allowlist and that check_endpoint does not re-introduce the TypeError regression; live VLM endpoint validation remains out of scope for automated CI due to model hardware requirements.

---

Auto-generated by the issues-rca skill — QA RCCA Analysis v2