MiniMax H3 reproducibly causes `GPU is lost` / TDR black-screen on RTX 5070 Ti when the system has 64 GB RAM; identical workload is stable when Windows is capped to 32 GB

[neko629]

Summary

On a single-GPU Windows box, MiniMax H3 video generation reliably kills the GPU (nvidia-smi: GPU is lost, requires reboot) after 1–8 generations. The same machine, same GPU, same driver, same ComfyUI build is completely stable in two control conditions:

• Cap Windows to 32 GB (bcdedit /set removememory 32768, both DIMMs physically installed, XMP on): 29 consecutive H3 generations, zero incidents.
• Run a comparable non-comfy_kitchen model at full 64 GB (Wan 2.1 14B fp16, 31.3 GB DiT, same DynamicVRAM streaming pressure, same ~300 s/run, same back-to-back queueing): 10 consecutive generations, zero incidents.

So the failure appears to require (a) H3 / comfy_kitchen quantized inference and (b) the full 64 GB being visible to the OS. Neither alone reproduces it.

Environment

GPU	RTX 5070 Ti (Blackwell, 16 GB)
Driver	610.47 and 610.88 — both affected
OS	Windows 11 Enterprise 26200
CPU / RAM	i5-12400F / 2×32 GB DDR4, tested at 3200 and 2666
ComfyUI	0.30.1 and 0.31.1 — both affected
comfy-kitchen	0.2.26 and 0.2.28 — both affected
comfy-aimdo	0.4.11 and 0.4.13 — both affected
torch	2.9.1+cu130
Model	minimax_h3_fl2va_int8_convrot (31.7 GB) + qwen3vl_32b_minimax_h3_nvfp4_awq TE; DynamicVRAM streaming (model does not fit in 16 GB VRAM)

Symptom

Mid-sampling the GPU disappears from the bus:

nvidia-smi: Unable to determine the device handle for GPU0: 0000:01:00.0: GPU is lost.
            Reboot the system to recover this GPU

Windows WER records, every time:

LKD_0x141_Tdr:6_IMAGE_nvlddmkm.sys_Blackwell
LKD_0x1B8_NV_Blackscreen_Blackbox_dxgkrnl!DxgCreateLiveDumpWithDriverBlob

No WHEA errors, no minidump. The OS survives (a scripted shutdown /r works), only the display driver is dead.

Telemetry at the moment of failure

10-second-resolution sampling across 8 separate crashes shows nothing anomalous right up to the last sample:

GPU temp        74–88 °C          (often falling just before the crash)
GPU power       234–280 W         (TDP 300 W, never pegged)
GPU util        100 %, dropping to 58–80 % in the final sample only
VRAM            10.5–15.9 GB      (never OOM)
Free system RAM 11–36 GB free     (never under pressure)
Pagefile peak   0–37 MB           (essentially untouched)

Memory pressure, thermals and power are all ruled out by direct measurement.

Python traceback is not a reliable fault location

The reported location differs every time, and in one case it is torch.ones(()) — an operation that cannot itself hang a GPU. This is consistent with the async-error warning ComfyUI already prints; listing them only to show the spread:

# ComfyUI 0.30.1
comfy/memory_management.py:32  read_tensor_file_slice_into
    destination2._params.copy_from(..., non_blocking=True)
  -> comfy_kitchen/tensor/base.py:86 copy_from

# ComfyUI 0.31.1, cuda backend
comfy_kitchen/backends/cuda/__init__.py:1586  quantize_and_rotate_rowwise
  -> comfy_kitchen/tensor/int8_utils.py:72 _rotate_activation
     x_rotated = torch.matmul(x_grouped, h)

# ComfyUI 0.31.1, triton backend forced first
comfy_kitchen/backends/triton/quantization.py:1055  int8_linear
  -> comfy_kitchen/tensor/int8_utils.py:72 _rotate_activation   (same shared helper)

# ComfyUI 0.31.1, nvfp4 weights
comfy/quant_ops.py:132  quantize
    scale = torch.ones((), device=tensor.device, dtype=torch.float32)

All four end in torch.AcceleratorError: CUDA error: unknown error.

Ruled out by experiment (each tested to the point of a crash or a clean run set)

• NVIDIA driver: 610.47 and 610.88 both crash
• ComfyUI: 0.30.1 and 0.31.1 both crash
• comfy-kitchen 0.2.26→0.2.28, comfy-aimdo 0.4.11→0.4.13: both crash
• --disable-pinned-memory: still crashes (run 6)
• --disable-async-offload (NUM_STREAMS=0, prefetch queue removed): still crashes (run 6)
• comfy_kitchen backend: cuda and triton (forced via set_backend_priority(["triton","cuda","eager"])) both crash
• Quantization: int8_convrot, pruned_int8_convrot, and a third-party nvfp4 build all crash
• Sampler steps 4 / 6 / 8; samplers dual_clock_euler and er_sde; 0.4 MP and 0.7 MP — all crash
• RAM: MemTest86 v11.7, 4/4 passes + row-hammer over the full 68.5 GB at 3192 MT/s — PASS, 0 errors
• RAM speed: XMP off (DDR4-2666) still crashes (run 15)
• Model size / staging volume: pruned build (20.0 GB staged, 24 GB RAM free at crash) crashes too
• Thermals, power, PSU, motherboard, GPU hardware: same hardware ran 41 consecutive clean H3 generations before the RAM upgrade

The two variables that do change the outcome

1. RAM visible to Windows. 32 GB visible → 29 clean runs. 64 GB visible → crashes within 1–8 runs, across every software configuration above.
2. Whether the model goes through comfy_kitchen quantized inference. Wan 2.1 fp16 at 64 GB: 10 clean runs under equivalent streaming load.

For reference, the measured difference between the two RAM conditions is that at 32 GB the 31.7 GB weight file is re-read from disk every step (sustained 600–976 MB/s), whereas at 64 GB it is served from the OS page cache (0 MB/s disk). ComfyUI-side constants that scale with total RAM are cache_ram = min(10, max(2, total_ram*0.10/1024)), cache_ram_inactive = min(128, total_ram/1024) (main.py) and MAX_PINNED_MEMORY = ram * 0.40 (model_management.py).

Repro

1. 16 GB VRAM Blackwell card, ≥64 GB system RAM, Windows.
2. Load minimax_h3_fl2va_int8_convrot + qwen3vl_32b_minimax_h3_nvfp4_awq (DynamicVRAM engages; model does not fit VRAM).
3. Queue 10+ generations at 0.7 MP / 243 frames back to back without restarting ComfyUI.
4. GPU is lost within 1–8 runs.
5. bcdedit /set removememory 32768 + reboot → same queue completes 29/29.

Question

Is there a known path by which the DynamicVRAM / comfy_kitchen quantized path can wedge the GPU, and is there a supported way to make ComfyUI behave as if the machine had less RAM (i.e. pin cache_ram / MAX_PINNED_MEMORY to the smaller values) without physically hiding memory from the OS?

I have 8 crash captures with 10-second telemetry, the full control-experiment matrix, and Windows event logs, and can supply any subset on request.

[coderabbitai]

🔗 Related PRs

#15456 - Free other models' memory before retrying handles_tiling VAE decode [open]
#15474 - fix(xpu): VAE no_grad + execution-layer VRAM prep, H3 first-forward synchronize, .gguf extension support [open]

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[neko629]

Update: tested the PR #15474 approach on CUDA — negative result

CodeRabbit linked #15474 (fix(xpu): ... H3 first-forward synchronize ...), whose description matches this failure almost exactly, just on a different vendor stack:

queued async weight uploads race the Level Zero driver and intermittently fail with UR_RESULT_ERROR_DEVICE_LOST on the first sampling step

Same shape here: async DynamicVRAM weight streaming → CUDA error: unknown error → nvidia-smi: GPU is lost → LKD_0x141_Tdr black-screen, intermittent, early in sampling, also on a 16 GB card that cannot hold the H3 DiT.

I ported it to CUDA and ran two variants. Neither helped.

Variant A — direct port of #15474 (single sync before the block loop)

# comfy/ldm/minimax/model.py, before make_prefetch_queue(...)
if device.type == "cuda":
    torch.cuda.synchronize(device)

Crashed on run 3 of a 20-run queue (runs 1–2 clean at 170.2 s / 164.0 s). Historical crash positions on this box without any patch are runs 1, 2, 2, 2, 6, 15 — so run 3 is inside the existing distribution, no signal.

Variant B — full serialization (sync after every prefetch_queue_pop)

The actual async uploads happen inside the loop, once per block, so a single pre-loop sync cannot cover them:

prefetch_queue = comfy.model_prefetch.make_prefetch_queue(...)
for i, block in enumerate(self.blocks):
    comfy.model_prefetch.prefetch_queue_pop(prefetch_queue, device, block)
    if device.type == "cuda":
        torch.cuda.synchronize(device)     # ~200 syncs per generation (50 blocks x 4 steps)
    h = block(...)

Crashed on run 6 of a 20-run queue (runs 1–5 clean). Patch execution verified: .pyc newer than source, ComfyUI started after both, device.type == "cuda" holds (device = video_x.device).

The interesting part: full serialization cost nothing

no patch    162.86 / 163.98 / 170.16 / 175.04 / 192.74 s
variant B   170.93 / 162.16 / 170.26 / 168.80 / 170.89 s

Adding ~200 forced synchronizes per generation produced no measurable slowdown. If there were substantial in-flight async uploads overlapping compute, forcing a sync at every block would have cost real time. Zero cost suggests there is little upload/compute overlap at that granularity to begin with — which argues against an async-upload race being the mechanism here, independent of whether the sync is placed correctly.

So #15474 appears to address a Level Zero-specific race rather than the CUDA failure in this report.

One structural note that may still be relevant

comfy.model_prefetch is used by exactly two models in the tree:

comfy/ldm/minimax/model.py
comfy/ldm/lightricks/av_model.py

Both are models I have seen black-screen this GPU. Wan 2.1 fp16, which does not use model_prefetch, ran 10/10 clean under equivalent streaming pressure at full 64 GB in the control experiment above. That correlation may be coincidental — H3 and LTX-2 AV are also the only two that combine huge DiTs, joint audio/video, and comfy_kitchen quantized inference — but it is the cleanest structural split I have found between the models that crash and the one that does not.

Additional exclusions since the original report

• Driver: black screens also occurred on the 595.x branch (2026-05-28 and 2026-05-31, before 610.47 was installed), so this is not a 610.x regression.
• --enable-triton-backend alone does not move dispatch off the CUDA backend (registry.py default priority is ["cuda", "triton", "eager"]). Forcing set_backend_priority(["triton", "cuda", "eager"]) does move it — verified by the traceback shifting to backends/triton/quantization.py — and it still crashes, both paths converging on the shared comfy_kitchen/tensor/int8_utils.py::_rotate_activation.
• Quantization format: int8_convrot, pruned_int8_convrot, and a third-party nvfp4 build all crash. The nvfp4 crash surfaced at comfy/quant_ops.py:132, on torch.ones((), device=...) — an operation that cannot itself hang a GPU, confirming the reported traceback location is only where the async error surfaced, not the fault site.

Still reproducible on demand; happy to run any instrumented build.

[Vicullum]

I'm having the same issue with the minimax_h3_ref2va_pruned_int8_convrot.safetensors model. 5090 with 64GB of ram, 610.88 driver. It happens randomly but roughly 25% of the time my screen will just go completely black as if the driver crashed and at that point there's nothing more I can do but hard reset my computer. Using the --disable-async-offload --disable-smart-memory flags did nothing.

[neko629]

Cross-linking #14714 — this looks like the same bug from a different angle, and a comment there contains what may be the key data point.

#14714 (RTX 5090, driver 610.47 DCH, WAN 2.2): cudaErrorIllegalAddress at offload_stream.synchronize(), "often complete GPU black screen, only option, hard reset power off PC. Nothing logged in comfyui log". Startup banner identical to mine:

Using async weight offloading with 2 streams
Enabled pinned memory
DynamicVRAM support detected

Their exclusion list also overlaps almost completely with mine: --disable-async-offload, --disable-dynamic-vram, --disable-smart-memory all fail to solve it; new PSU; GPU verified stable under hours of AI Toolkit training.

The comment by @Windecay in #14714 is the part I want to highlight:

5090 32GB + 192GB RAM
ComfyUI 0.27.0, pytorch 2.9.1+cu130, driver 591.74 studio
1920x1088
use bernini fp8 scaled  : PASS
use bernini int8 convrot: FAILED

Different GPU, different driver branch, different model, 192 GB RAM — and the same split I hit independently: int8_convrot crashes, fp8_scaled does not.

Two corrections to my own report in light of this:

1. RAM size is probably not the mechanism, only a trigger-rate modifier. @Windecay reproduces it with 192 GB. My 32 GB-visible / 64 GB-visible split is real and reproducible on my box (29 clean vs crash within 1–6), but it should not be read as "this needs ≥64 GB".
2. Driver range is wider than I stated. Combining reports: 591.74, 595.x (my own pre-610 crashes on 2026-05-28 / 05-31), 610.47, 610.88. Not a single-branch regression.

What the three reports have in common:

	#14714 (trick72)	#14714 comment (Windecay)	#15488 (this)
GPU	RTX 5090	RTX 5090	RTX 5070 Ti
RAM	32 GB	192 GB	64 GB
Driver	610.47	591.74	610.47 / 610.88
Model	WAN 2.2	Bernini	MiniMax H3
Quantization	—	int8_convrot fails, fp8_scaled passes	int8_convrot / pruned_int8_convrot / nvfp4 all fail
Symptom	cudaErrorIllegalAddress → black screen, hard reset	—	CUDA unknown error → GPU is lost → LKD_0x141 TDR
async offload off	doesn't help	—	doesn't help
DynamicVRAM	on	on	on

Common denominators across all three: Blackwell, DynamicVRAM + async weight offload + pinned memory, a model too large for VRAM, and comfy_kitchen quantized inference.

I am downloading minimax_h3_fl2va_pruned_fp8_scaled now to test the fp8_scaled leg on H3, since that is the one format independently reported as passing. Will report back either way. Note that in my testing nvfp4 — also not convrot — still crashed, so "not convrot" may not be sufficient on its own.

[yaotech2023]

Same problem, I'm using 5070ti with 32GB ram.