`1.3.0rc15`: trtllm-serve hangs in fused_moe/quantization.py during cold-start on GB10 / sm_121 — `1.3.0rc12` works with identical config

[zentradev-rabih]

Summary

Loading Nemotron-3-Super-120B-A12B-NVFP4 via trtllm-serve with the same launch args and the same extra_llm_api_options yaml on 1.3.0rc12 and 1.3.0rc15:

Image	Cold-start	Outcome
nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc12	~4 min	✅ serves /v1/models
nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc15	18+ min, never serves	❌ worker spins on NVIDIA-driver ioctls in fused_moe/quantization.py:2769, progress bar stuck at Loading weights concurrently: 0/1288

Reverting only the image tag back to rc12 immediately restores normal cold-start and serving.

Environment

• GPU: NVIDIA GB10 (Grace-Blackwell, sm_121, integrated GPU, unified-memory architecture)
• OS: DGX OS 7 (Ubuntu 24.04 aarch64)
• Model: Nemotron-3-Super-120B-A12B-NVFP4 (MoE; hybrid Mamba/SSM + attention; NVFP4 quantized)
• Container image tested: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc15
• Backend: pytorch (AutoDeploy via trtllm-serve)

Repro

Launch command (identical args work on 1.3.0rc12):

docker run --rm \
    --gpus all --ipc host --network host \
    --ulimit memlock=-1 --ulimit stack=67108864 \
    -e TLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
    -v /path/to/Nemotron-3-Super-120B-A12B-NVFP4:/model:ro \
    -v /path/to/extra-llm-api-config.yml:/extra-llm-api-config.yml:ro \
    nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc15 \
    trtllm-serve /model --host 0.0.0.0 --port 8080 \
      --served_model_name nemotron-super-120b-nvfp4 \
      --max_batch_size 8 --tp_size 1 --ep_size 1 \
      --max_num_tokens 8192 --max_seq_len 1048576 \
      --trust_remote_code \
      --reasoning_parser nano-v3 --tool_parser qwen3_coder \
      --extra_llm_api_options /extra-llm-api-config.yml

extra-llm-api-config.yml:

kv_cache_config:
  dtype: fp8
  enable_block_reuse: false
  free_gpu_memory_fraction: 0.9
  mamba_ssm_cache_dtype: float16
  mamba_ssm_stochastic_rounding: true
  mamba_ssm_philox_rounds: 5
moe_config:
  backend: CUTLASS
cuda_graph_config:
  enable_padding: true
  max_batch_size: 8
enable_chunked_prefill: true

(No speculative decoding / MTP. tp_size=1, ep_size=1.)

Observed behavior

Boot reaches this point and then makes no further visible progress for 18 minutes:

[TensorRT-LLM] TensorRT LLM version: 1.3.0rc15
…
Loading safetensors weights in parallel: 100%|██████████| 17/17 [00:15<00:00,  1.13it/s]
[TRT-LLM] [W] [_torch] [load_weight_shard] Skipping device transfer from cpu to cuda on integrated GPU to conserve shared memory.
Loading weights concurrently:   0%|          | 0/1288 [00:00<?, ?it/s]
/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/modules/fused_moe/quantization.py:2769: UserWarning: TypedStorage is deprecated. […]
  dst_base = dst_w3_w1_weight.storage().data_ptr()

No further log lines, no errors emitted. During the hang:

• CPU: the mpi4py.futures.server worker holds 99.9 % of one core; accumulated ~15 min of CPU time on a single thread
• GPU: 0 % utilization throughout (nvidia-smi)
• Block I/O: stable at 51.4 GB read (all weights already on disk; not I/O bound)
• Memory: container RSS 19 GB / 119 GB unified; no swap thrashing (vmstat shows si≈0, so=0)

strace of the busy worker thread

ioctl(23, _IOC(_IOC_READ|_IOC_WRITE, 0x46, 0x2a, 0x20), 0xf0336fffbf50) = 0
ioctl(23, _IOC(_IOC_READ|_IOC_WRITE, 0x46, 0x2a, 0x20), 0xf0336fffbf50) = 0
ioctl(23, _IOC(_IOC_READ|_IOC_WRITE, 0x46, 0x2a, 0x20), 0xf0336fffbf50) = 0
…thousands of identical calls, all returning 0…

0x46 is the NVRM character-device magic; 0x2a is a memory-handshake ioctl. All calls succeed but the caller never reaches a satisfying condition.

Likely path

The log line Skipping device transfer from cpu to cuda on integrated GPU to conserve shared memory is the GB10-specific code path that avoids the cpu→cuda copy because unified memory makes it redundant. The hang then appears inside the NVFP4 fused-MoE weight-quantization path entered from there (_torch/modules/fused_moe/quantization.py:2769).

Why this looks like an rc15 regression, not a config issue

• The same yaml and the same flags boot on 1.3.0rc12 in ~4 minutes and serve traffic stably at max_batch_size=8
• Switching only the container tag rc15 → rc12 restores normal cold-start; no other change required
• Nothing exotic in the config: no MTP, no TP, no EP, default pytorch backend

Worth noting: 1.3.0rc15 does not hit the sm_121 PTXAS / Triton codegen error that previously blocked 1.3.0rc14 on this same hardware — so the rc15 boot path on GB10 has clearly moved, just not yet to a working state.

What would help

1. Confirmation whether the GB10 / sm_121 integrated-GPU NVFP4 fused-MoE path is exercised in CI for rc15
2. Whether moe_config.backend: TRTLLM (vs CUTLASS) is expected to work-around it
3. Whether mamba_ssm_cache_dtype: bfloat16 (vs float16) makes a difference
4. Whether --enable_chunked_prefill off changes anything

Happy to capture py-spy dump, a longer strace, or run additional repro variants — let me know what's most useful.

[zentradev-rabih]

Context — why this matters to us (added per reviewer benefit):

We were specifically attempting rc15 to validate the mamba-cache + Scheduler-V2 fixes documented in the release notes:

• [None][feat] Improve memory calculation for mamba hybrid models when block reuse is off #13549 — Improve memory calculation for mamba hybrid models when block reuse is off
• [https://nvbugs/6094224][fix] Fix mamba disagg issues when conc > mbs #13274 — Fix mamba disagg issues when conc > mbs
• [None][fix] Clear stale Scheduler V2 request state #13592 — Clear stale Scheduler V2 request state
• [https://nvbugs/6115271][fix] Fix stale TRTLLM attention backend metadata reuse across request shape transitions #13696 — Fix stale TRTLLM attention backend metadata reuse across request shape transitions

On rc12 with the same Nemotron-3-Super-120B-A12B-NVFP4 setup, we currently observe two production-impacting symptoms at moderate concurrency:

1. Engine event-loop thread dies under mamba-cache pressure — the HTTP shell keeps answering 200 OK with empty SSE bodies (no error), so the failure is invisible to upstream proxies. We work around it with an external 30-second 1-token liveness probe that flips /api/health to degraded+503 when the engine stops emitting tokens.
2. Cancellation leaks request slots — when a client disconnects mid-stream, the scheduler doesn't always release the slot, so the next ~N requests at the batch ceiling hang waiting for a slot that never frees.

The four rc15 PRs above all read like direct fixes for those two symptoms, which is why we tried to upgrade. The cold-start hang reported in this issue is currently blocking us from reaching the /v1/models endpoint at all on rc15, so we can't confirm whether those fixes resolve our production symptoms.

If there's a way to validate rc15 on GB10 / sm_121 without going through the fused_moe/quantization.py path that's hanging (e.g., a moe_config.backend knob or a build flag), that would unblock us. Otherwise we're holding at rc12 with the workarounds above pending a fix.

[ssam18]

The hang site (rc15 line 2769, HEAD line 2806) sits inside a per-expert dict-accumulation pattern (tmp_cutlass_w3_w1_weights, keyed off dst.storage().data_ptr()) that wasn't there at rc12. On integrated GPU the dst stays host/UMA-backed because of the "Skipping device transfer..." branch you log just above, and calling .storage().data_ptr() on that path goes through NVRM memory ops, which is the 0x46/0x2a ioctl loop your strace is sitting in. Best bisect candidates between rc12 and HEAD on this file: f15af81 (EPLB host memory reduction) and 1f5339f (shared-expert index fix).

If you need to unblock today, try moe_config.backend: TRTLLM. The hanging code is CUTLASS-specific (the class is CUTLASS-prefixed and the dict is literally tmp_cutlass_w3_w1_weights), so switching backends should skip this entirely. Can't promise TRTLLM is feature-equivalent for NVFP4 MoE on Nemotron-3-Super-120B but it's the cheapest thing to try. The bfloat16 mamba cache and chunked-prefill-off knobs are on unrelated code paths so they probably won't help with a load-time hang.

[zentradev-rabih]

Thanks for the very specific diagnosis — much appreciated.

I tried moe_config.backend: TRTLLM (only that knob changed; CUDA-graph + chunked-prefill + mamba SSM settings left as before, MTP still off). It exits hard at executor-worker startup with:

NotImplementedError: TRTLLMGenFusedMoE does not support SM120 and above.

Full traceback (trimmed to the relevant frames):

File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/commands/serve.py", line 331, in launch_server
    llm = PyTorchLLM(**llm_args)
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/llm.py", line 1533, in _build_model
    self._executor = self._executor_cls.create(…)
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/proxy.py", line 402, in _start_executor_workers
    raise RuntimeError("Executor worker returned error")
…
NotImplementedError: TRTLLMGenFusedMoE does not support SM120 and above.

So on GB10 / sm_121 in 1.3.0rc15 both NVFP4 MoE backends appear blocked:

• moe_config.backend: CUTLASS — hangs in fused_moe/quantization.py:2769 (the original report)
• moe_config.backend: TRTLLM — explicitly NotImplementedError for SM120+

Reverted to rc12 for now. The f15af81 / 1f5339f bisect targets you flagged are still the most likely culprits for the CUTLASS path on integrated/UMA GPUs — happy to test a candidate patch or a custom build against this hardware if that would help narrow it down. Let me know.

[zentradev-rabih]

Re-framing — to be explicit about the production cost driving this:

The original report focused on the rc15 cold-start hang, but that's the symptom blocking the upgrade, not the underlying production pain. Surfacing the real problem so the priority is clear:

Where we're stuck on Nemotron-3-Super-120B-A12B-NVFP4 + GB10 today:

1. With MTP on (speculative_config: num_nextn_predict_layers: 3): each request consumes 1 main + 3 draft = 4 mamba slots, against a pool of 8. Engine reaches slot exhaustion at ~4–6 concurrent users, after which the event-loop thread dies under the slot pressure — HTTP shell stays up emitting empty SSE bodies, so it's silent.
2. Workaround we run in production: disable MTP entirely. Throughput on this model drops roughly 30 tok/s → 15–25 tok/s (≈30–50 %). On a 120B model that latency hit is meaningful for every interactive user, and it's been steady since we adopted Nemotron-3-Super.
3. The supposed fix is rc14 [None][fix] Fix Mamba cache correctness under MTP + CUDA-graph padding #13151 (per the release notes for the MTP+mamba interaction), but rc14 has the separate sm_121 PTXAS/Triton codegen issue, so we cannot adopt it on GB10.
4. rc15 was the next viable target — release notes ([None][feat] Improve memory calculation for mamba hybrid models when block reuse is off #13549, [https://nvbugs/6094224][fix] Fix mamba disagg issues when conc > mbs #13274, [None][fix] Clear stale Scheduler V2 request state #13592, [https://nvbugs/6115271][fix] Fix stale TRTLLM attention backend metadata reuse across request shape transitions #13696) all read like direct fixes for our scheduler / mamba behavior. We attempted the upgrade and hit:
   • moe_config.backend: CUTLASS → the cold-start ioctl-loop hang reported in this issue
   • moe_config.backend: TRTLLM → NotImplementedError: TRTLLMGenFusedMoE does not support SM120 and above (comment above)

Net: on the only NVFP4 model we're shipping, no MoE backend in rc15 works on GB10, the slot-exhaustion bug remains live on rc12, and the only known mitigation costs us a real chunk of throughput on every request.

What we'd actually find unblocking (in rough priority order):

1. A working NVFP4 fused-MoE path on sm_120+ in a near-term release — either CUTLASS without the integrated-GPU hang, or TRTLLMGenFusedMoE extended to SM120+
2. Or: a backport of [None][fix] Fix Mamba cache correctness under MTP + CUDA-graph padding #13151 (MTP + mamba slot accounting) to a release that actually loads on GB10
3. Or: confirmation that there's an existing config knob to avoid the per-expert host-tensor accumulation path on UMA-backed dst (so we can stay on rc15 without the CUTLASS hang)

Happy to be the GB10 / Nemotron-3-Super reproducer for any of those — we can build/test custom images on this hardware and feed back results quickly.

[rehan243]

Fwiw, we hit a similar issue with trtllm-serve on our end when loading a quantized MoE model on GB10 hardware, turns out it was related to the fused_moe kernel and quantization. We were using 1.3.0rc12 at the time, and switching to a different quantization scheme fixed it for us — specifically, we changed the quantization_bit param from 4 to 8 in our model config. Not saying that's the same root cause here, but it's worth checking if the quantization settings are identical between rc12 and rc15. What's your extra_llm_api_options yaml look like, specifically the parts related to quantization?