AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' when serving Cosmos3-Nano/Super Reasoner via vLLM

[berlin088]

Attempting to serve both nvidia/Cosmos3-Nano and nvidia/Cosmos3-Super in Reasoner mode via vLLM fails with an identical error during vision-encoder initialization, regardless of model size.

Environment:

GPU: NVIDIA B200 (180GB VRAM)
CUDA: 12.8, Driver 570.211.01
vllm: 0.19.1
vllm-cosmos3: 0.1.0 (installed from packages/vllm-cosmos3 in this repo)
Python: 3.12
OS: Ubuntu 24.04

Command run:

bash
vllm serve /workspace/cosmos3-nano
--hf-overrides '{"architectures": ["Cosmos3ReasonerForConditionalGeneration"]}'
--async-scheduling
--allowed-local-media-path /
--port 8000

Behavior: Model weights load successfully (all shards complete). Failure occurs during a subsequent vision-encoder forward pass, in qwen3_vl.py's vit_flash_attn_wrapper, which calls into vllm.vllm_flash_attn.cute.interface, ultimately failing at:

File ".../vllm_flash_attn/cute/utils.py", line 115
AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma'

Reproduced identically on both Cosmos3-Nano and Cosmos3-Super — confirms this is not size-dependent.

Attempted fixes (none resolved it):

export VLLM_ATTENTION_BACKEND=FLASH_ATTN
nvidia-cutlass-dsl versions 4.4.1, 4.6.2, and 4.7.1 (default installed version)
Uninstalling flashinfer-python

Any guidance on the correct cutlass-dsl version pairing, or whether this vision-attention path requires a different flag/config to bypass, would be appreciated.

[lfengad]

Thanks for the detailed report. This is a CUTLASS DSL version-skew issue rather than a Cosmos3 model-size issue.

For the CUDA 12.8 path, our tested combination is:

• vllm==0.19.1
• nvidia-cutlass-dsl==4.4.2
• nvidia-cutlass-dsl-libs-base==4.4.2

vLLM 0.19.1 currently declares nvidia-cutlass-dsl>=4.4.0.dev1, so a standalone installation can resolve to a newer, API-incompatible CUTLASS DSL release. The Cosmos Framework lockfile resolves exactly 4.4.2, and we verified that this version provides cutlass.cute.core.ThrMma. None of the versions listed in the report is the tested 4.4.2.

Could you please try the exact version in a clean environment, or reinstall the matching pair together?

python -m pip uninstall -y \
  nvidia-cutlass-dsl \
  nvidia-cutlass-dsl-libs-base \
  nvidia-cutlass-dsl-libs-cu13

python -m pip install "nvidia-cutlass-dsl==4.4.2"

You can verify the environment with:

python - <<'PY'
from importlib.metadata import version
import cutlass.cute.core as core

print("dsl:", version("nvidia-cutlass-dsl"))
print("base:", version("nvidia-cutlass-dsl-libs-base"))
print("ThrMma:", hasattr(core, "ThrMma"))
PY

The expected output is 4.4.2, 4.4.2, and True.

As a temporary fallback, --mm-encoder-attn-backend TORCH_SDPA bypasses the vision FlashAttention path:

vllm serve /workspace/cosmos3-nano \
  --hf-overrides '{"architectures": ["Cosmos3ReasonerForConditionalGeneration"]}' \
  --mm-encoder-attn-backend TORCH_SDPA \
  --async-scheduling \
  --allowed-local-media-path / \
  --port 8000

VLLM_ATTENTION_BACKEND does not specifically select the multimodal vision-encoder backend, so it does not bypass this path. TORCH_SDPA is expected to be slower and is mainly useful as a workaround or diagnostic.

This also exposes a reproducibility gap in our standalone vllm-cosmos3 installation path. We should tighten the dependency/example so it cannot float away from the tested CUTLASS DSL version. Please let us know whether the exact 4.4.2 pair resolves the startup failure.