[Nemo FW] ImportError while evaluating on ocrbench from nvidia-vlmeval

[athitten]

Describe the bug

Evaluating qwen2.5-vl on ocrbench from nvidia-vlmeval inside the nemo fw container leads to the below Import error. Installed vlmeval inside nemo fw container via pip install nvidia-vlmeval

Traceback (most recent call last):
  File "<frozen runpy>", line 189, in _run_module_as_main
  File "<frozen runpy>", line 112, in _get_module_details
  File "/usr/local/lib/python3.12/dist-packages/vlmeval/__init__.py", line 26, in <module>
    from .api import *
  File "/usr/local/lib/python3.12/dist-packages/vlmeval/api/__init__.py", line 23, in <module>
    from .qwen_vl_api import QwenVLWrapper, QwenVLAPI, Qwen2VLAPI
  File "/usr/local/lib/python3.12/dist-packages/vlmeval/api/qwen_vl_api.py", line 23, in <module>
    from vlmeval.vlm.qwen2_vl.prompt import Qwen2VLPromptMixin
  File "/usr/local/lib/python3.12/dist-packages/vlmeval/vlm/__init__.py", line 20, in <module>
    from .aria import Aria
  File "/usr/local/lib/python3.12/dist-packages/vlmeval/vlm/aria.py", line 23, in <module>
    from .base import BaseModel
  File "/usr/local/lib/python3.12/dist-packages/vlmeval/vlm/base.py", line 17, in <module>
    from ..dataset import img_root_map, DATASET_TYPE
  File "/usr/local/lib/python3.12/dist-packages/vlmeval/dataset/__init__.py", line 45, in <module>
    from .mvbench import MVBench, MVBench_MP4
  File "/usr/local/lib/python3.12/dist-packages/vlmeval/dataset/mvbench.py", line 27, in <module>
    import cv2
  File "/usr/local/lib/python3.12/dist-packages/cv2/__init__.py", line 181, in <module>
    bootstrap()
  File "/usr/local/lib/python3.12/dist-packages/cv2/__init__.py", line 153, in bootstrap
    native_module = importlib.import_module("cv2")
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ImportError: libxcb.so.1: cannot open shared object file: No such file or directory

Steps/Code to reproduce bug

Mulitmodal(MM) or vision language model deployment with NeMO FW's MBridge will be available in a public container with 26.02 container release (2/19). Can share details offline on the internal containers that can be used to reproduce this in the meantime. The eval snippet used is as below after running pip install nvidia-vlmeval

from nemo_evaluator.api import check_endpoint, evaluate
from nemo_evaluator.api.api_dataclasses import (
    ApiEndpoint,
    ConfigParams,
    EvaluationConfig,
    EvaluationTarget,
)

# Configure the evaluation target
api_endpoint = ApiEndpoint(
    url="http://0.0.0.0:1235/v1/chat/completions/",
    type="vlm",
    model_id="megatron-model",
)
eval_target = EvaluationTarget(api_endpoint=api_endpoint)
eval_params = ConfigParams(top_p=0, temperature=0, limit_samples=2, parallelism=1)
eval_config = EvaluationConfig(type="ocrbench", params=eval_params, output_dir="results")

if __name__ == "__main__":
    evaluate(target_cfg=eval_target, eval_cfg=eval_config)

Expected behavior

A clear and concise description of what you expected to happen.

Additional context

Add any other context about the problem here.

[marta-sd]

@athitten looks like there's something wrong with the cv2 installation in the container (opencv-python package). When I do pip show opencv-python I can't find it, but at the same time I can import it with no issue (python -c "import cv2" command works).
Our package depends on cv2, doesn't detect it in the env and tries to install it, breaking the environment. To verify this hypothesis I installed it without cv2:

# content of requirements.txt
decord; platform_machine != 'arm64'
eva-decord; platform_machine == 'arm64'
gradio
huggingface_hub
imageio
matplotlib
numpy
omegaconf
openai
# opencv-python>=4.4.0.46  # commented out to use existing installation
openpyxl
pandas
pillow
portalocker
protobuf
python-dotenv
requests
rich
sentencepiece
setuptools
sty
tabulate
tiktoken
timeout-decorator
torch
torchvision
tqdm
transformers
typing_extensions
validators
xlsxwriter<=3.2.3  # pinning because pandas doesnt work with new xlsxwriter
nemo-evaluator>=0.1.51

pip install -r requirements.txt
pip install  --no-deps nvidia-vlmeval

Then I was able to run the evaluation snippet you provided without import errors.

@chtruong814 please advise how we can install a package that depends on opencv-python in a running 26.02. container without breaking the environment.

[marta-sd]

Update after discussing with the Automation team:
The issue will be fixed in a future version of the NeMo FW container (most likely 26.04). Also, we need to make sure we use uv for installation to correctly resolve all dependencies.

I was able to install the package successfully in the dev container with the following command: cd /opt/NeMo-FW/ && uv pip install nvidia-vlmeval

@athitten should we close the issue or you'd like to keep it for tracking?

[athitten]

Thank you @marta-sd , lets make sure to add this installation clearly when we add the docs for MM evaluation. We can close this and use #279 for tracking.

[pruprakash]

QA RCCA Analysis

Date: 2026-04-20
Analyst: AI QA Agent (issues-rca skill)
Issue: #715 — [Nemo FW] ImportError while evaluating on ocrbench from nvidia-vlmeval

---

1. Fix Reference

No dedicated fix PR was created. The issue was closed on 2026-02-19. The resolution involved ensuring opencv-python (cv2) and vlmeval were installed correctly inside the NeMo FW container. The key insight from the issue: the package must be installed via uv pip install opencv-python rather than plain pip install to properly register the shared library paths in the container's virtual environment. A plain pip install left cv2 in a location not on the Python path used by the container's vlmeval runner.

---

2. Root Cause

The nvidia-vlmeval package has a transitive dependency on opencv-python (cv2) for image processing tasks in benchmarks like OCRBench. The NeMo FW container (nvcr.io/nvidia/nemo nightly) did not include opencv-python in its image manifest. When vlmeval was installed via pip install nvidia-vlmeval inside the container, its __init__.py chain imported cv2 immediately, raising ImportError before any evaluation could start. The error propagated through vlmeval/api/__init__.py → vlmeval/api/qwen_*.py and was not caught, killing the evaluation process entirely.

---

3. Trigger Config

Trigger conditions (AND — all must be present):

• NeMo FW container without opencv-python pre-installed
• nvidia-vlmeval installed via pip install nvidia-vlmeval (not the full VLM eval container)
• Any VLM benchmark that imports vlmeval (e.g., OCRBench, any nvidia-vlmeval harness)

Deterministic? Yes. vlmeval/__init__.py imports cv2 at module load time — the import fails on every run when cv2 is absent.

---

4. Nature of Bug

Primary classification: Build/packaging bug — installation, import, or dependency failure (missing transitive dependency)

Impact scope: All users installing nvidia-vlmeval inside a NeMo FW container and running any VLM evaluation benchmark. The failure is at import time — no evaluation results are produced.

NOT affected: The dedicated VLM evaluation container (nvcr.io/nvidia/eval-factory/vlmeval) which pre-includes opencv-python. Text-only evaluation benchmarks that do not import vlmeval.

---

5. Functional Test Coverage

Verdict: COVERED

Test	File	Key Config	What it covers
test_cv2_importable_for_vlmeval	evaluator/testcases/rcca/infra/test_container_imports_smoke.py	Inside container	Hard-fail import check for cv2 (opencv-python)
test_vlmeval_importable	same	Inside container	Hard-fail import check for vlmeval
test_all_required_imports_summary	same	Inside container	Batch check including cv2 and vlmeval with pass/fail summary

These tests run inside the container and fail hard (not skip) if cv2 is absent, providing a release gate for VLM evaluation capability.

---

6. Gaps and Limitations

Gap 1 — Install method not tested:
The fix requires uv pip install opencv-python specifically. The test only checks if cv2 is importable — it does not verify the install was done via uv rather than plain pip. A future container update using plain pip might pass the test locally but fail in some edge-case path.

Gap 2 — No OCRBench end-to-end test:
Running an actual OCRBench evaluation requires VLM model weights, GPU, and significant runtime. This is outside CI scope.

Overall gap assessment: Low regression risk. The smoke import test directly catches the failure condition from this issue (cv2 not importable) on every container run.

---

7. New Test Added

Field	Value
Test file	evaluator/testcases/rcca/infra/test_container_imports_smoke.py
Test functions	test_cv2_importable_for_vlmeval, test_vlmeval_importable, test_all_required_imports_summary
QA repo	nmfw_tests (local: evaluator/testcases/rcca/infra/)
PR	Pending
What it validates	cv2 (opencv-python) and vlmeval are importable inside the container
How it would catch a regression	test_cv2_importable_for_vlmeval calls pytest.fail() with install instructions if cv2 is missing — catches any future container that omits the package.

---

8. Conclusion

Issue #715 was caused by opencv-python (cv2) missing from the NeMo FW container while vlmeval requires it as a transitive dependency at import time, making all VLM evaluation benchmarks completely unusable. The fix was ensuring cv2 is installed via uv pip install opencv-python inside the container. A new container smoke-import test hard-fails if cv2 or vlmeval is missing, acting as a release gate; regression risk is low.

---

Auto-generated by the issues-rca skill — QA RCCA Analysis v2