[Issue] [gfx1100]: host→device copies stall one second at a time on kernel 7.0.0-28 — c08972f55594 without 342981fff328

[cadamcat]

Summary

On gfx1100 with ROCm 7.14, a host-to-device copy runs at ~9.4 GiB/s (9 575 MiB/s) when the source is a read-only file mapping (r--p), and at 2.0 MiB/s when the source is a writable private file mapping (rw-p) whose pages have already been faulted into the process.

The kernel decides how bad it gets, and the cause is identified and now confirmed by revert. Ubuntu's 7.0.0-28 picked up drm/amdgpu: fix amdgpu_hmm_range_get_pages (c08972f55594) without the follow-up that repairs its consequence, drm/amdgpu: drop retry loop in amdgpu_hmm_range_get_pages (342981fff328). The first moved mmu_interval_read_begin() out of the per-chunk loop, so notifier_seq is never refreshed across retries and the -EBUSY retry path can never succeed; it busy-spins for the whole HMM_RANGE_DEFAULT_TIMEOUT, which is 1000 ms, then bails with -EAGAIN and the caller tries again. Every timing in this report is an integer multiple of that constant. On the same guest with only the kernel changed, 7.0.0-14-generic does the same copy in 18.6 to 20.2 ms.

A separate and milder penalty, 4× to 8× against a read-only copy of the same bytes, is present on every kernel tested. @ashetaia-amd explained that one below: kfd_svm.c takes the fault permission from the VMA rather than from what the copy will do, so copy-on-write is broken on every resident page even though a host→device copy only reads. That break is also what advances the notifier sequence, which is why only the writable-and-resident case ever enters the futile retry.

Same file, same bytes, same size, same process. The only difference is the prot argument to mmap and whether the CPU has touched the pages first.

This is not a corner case: it is the default path for loading any safetensors checkpoint into PyTorch, because safetensors hands PyTorch a storage created by torch.UntypedStorage.from_file(..., shared=False), and PyTorch maps that writable. On the affected kernel it makes vLLM model loading 19–48x slower than the disk it reads from: a 15.26 GiB checkpoint takes 206 s, against ~19 s with the workaround below and ~11 s for the disk alone.

Measurements

source mapping	pages faulted in first	32 MiB copy	rate
MAP_PRIVATE | PROT_READ (r--p)	yes	3.3 ms	9 575 MiB/s
MAP_PRIVATE | PROT_READ|PROT_WRITE (rw-p)	no	15.8 ms	2 020 MiB/s
MAP_PRIVATE | PROT_READ|PROT_WRITE (rw-p)	yes	16 020 ms	2.0 MiB/s

Four environments, one reproducer, and only one of them is slow:

environment	kernel	rw-p resident, 32 MiB
VFIO guest	7.0.0-28-generic	16 019.3 / 16 019.6 / 16 019.9 / 16 020.1 ms
VFIO guest	7.0.0-14-generic	18.6 / 18.9 / 20.2 ms
VFIO guest	6.8.0-136-generic	22.1 / 23.3 / 26.0 ms
bare metal	7.0.14-4-pve	24.1 / 24.3 / 25.1 / 28.6 ms

The first two rows hold everything constant but the kernel ABI — same guest, same ROCm 7.14 userspace, same container image — so passthrough is not a necessary ingredient, which had been the leading suspicion. iommu=pt is not the variable either: this host already boots with it, and the guest has no IOMMU of its own.

In timeout windows those numbers are 16.0, and 17.0 for the 17 020 ms the same reproducer gives on overlayfs and tmpfs. Per-tensor stalls during a real checkpoint load measured 1005, 1042 and 1522 ms, which is 1.0, 1.0 and 1.5. Those tensors are only partially resident, since nothing in the load path touches the pages before the copy, so their window counts are not comparable with the fully resident reproducer figures above. Same mechanism, different amount of resident memory entering it. "Why this hits real workloads" below has the detail.

The first copy any process makes costs about 40 ms whatever the mapping, which is why the reproducer discards one r--p copy before the reading that counts. The pathological case is repeatable to a degree worth stating: on ext4 it landed at 16 019.3, 16 019.6, 16 019.9 and 16 020.1 ms over four runs today, and on overlayfs and tmpfs at 17 020.5 and 17 020.4 ms. Under a millisecond of spread within a filesystem, and a flat 1 s offset between them, because the cost is a whole number of 1000 ms timeout windows and the container filesystems take one retry more.

Against the read-only case this is 4 388× to 4 787× depending on the run, but the ratio is the wrong way to think about it: the numerator is a count of timeout windows. At 256 MiB the copy takes 128 s, which is 128 windows for eight times the pages, so the window count tracks the number of resident pages while each window stays at its constant 1000 ms. The granularity works out to 2 MiB of resident source per window, and it is not specific to this machine: @shineday999 measured 8 MiB in exactly four windows below, on a different board under a different ROCm, and after subtracting the windows the residual real work lands in the same band on both, 1624 MiB/s here against 1575 to 1695 MiB/s there.

Anonymous memory of the same size copies at ~13 GiB/s, i.e. the hardware and the PCIe link are fine.

Reproducer

Dependency-free — PyTorch only, no safetensors, no vLLM. The backing filesystem does not matter either: the same pathological rate appears on ext4 (2.0 MiB/s), on a container's own overlayfs (1.9 MiB/s) and on tmpfs (1.9 MiB/s), so a plain container with no bind mount reproduces it.

import os, mmap, time, torch

PATH, N, DEV = "/data/repro.bin", 32 << 20, "cuda:0"
if not os.path.exists(PATH) or os.path.getsize(PATH) < N * 8:
    with open(PATH, "wb") as f:
        for _ in range(N * 8 >> 20):
            f.write(os.urandom(1 << 20))
with open(PATH, "rb") as f:                       # warm the page cache
    while f.read(1 << 24):
        pass

def run(writable, pretouch):
    fh = open(PATH, "r+b" if writable else "rb")
    prot = mmap.PROT_READ | (mmap.PROT_WRITE if writable else 0)
    mm = mmap.mmap(fh.fileno(), N * 8, flags=mmap.MAP_PRIVATE, prot=prot)
    src = torch.frombuffer(mm, dtype=torch.uint8, count=N * 8)[:N]
    if pretouch:
        src.max()                                 # bulk read: faults the pages in
    torch.cuda.synchronize(); t0 = time.perf_counter()
    g = src.to(DEV); torch.cuda.synchronize()
    dt = time.perf_counter() - t0
    perm = [l.split()[1] for l in open("/proc/self/maps") if "repro.bin" in l][0]
    print(f"{perm} pretouch={pretouch!s:<5} {dt*1000:9.1f} ms  {N/dt/2**20:8.1f} MiB/s")
    del g, src; torch.cuda.empty_cache(); mm.close(); fh.close()

run(False, True)     # warm-up: the first copy in a process costs ~40 ms whatever the mapping
run(False, True)     # r--p, resident ->  ~9500 MiB/s
run(True,  False)    # rw-p, cold     ->  ~2000 MiB/s
run(True,  True)     # rw-p, resident ->     ~2 MiB/s

Where the time goes

perf on a vLLM worker during a real checkpoint load:

99.11%  ioctl
        └─ __x64_sys_ioctl → kfd_ioctl → kfd_ioctl_svm → svm_ioctl
           └─ svm_range_set_attr → svm_range_validate_and_map
              └─ 94.19% amdgpu_hmm_range_get_pages → 86.95% hmm_range_fault

strace over a 12 s window of the same load: 54 ioctls, ~189 ms each.

That profile is the busy spin, not work. svm_range_validate_and_map takes readonly from the VMA's VM_WRITE bit at kfd_svm.c:1777 and passes it to amdgpu_hmm_range_get_pages(), which turns !readonly into HMM_PFN_REQ_WRITE at amdgpu_hmm.c:188. So a copy that only reads its source asks for write access, copy-on-write is broken on every resident page, and that break advances the notifier sequence. On -28 the stale notifier_seq then makes the -EBUSY retry unable to succeed, and it spins out the full timeout. A read-only mapping never invalidates, so it never enters any of this.

Why this hits real workloads

safetensors, on its PyTorch path, does not use its own (read-only) mapping. It calls:

// safetensors/bindings/python/src/lib.rs
torch.UntypedStorage.from_file(filename, shared=False, nbytes=size)

PyTorch maps that file writable, so every tensor a framework loads from a safetensors checkpoint is backed by an rw-p mapping. Confirmed with /proc/<pid>/maps, same file opened two ways in one process:

rw-p ... model-00001-of-00005.safetensors   <- safetensors framework="pt"
r--p ... model-00001-of-00005.safetensors   <- safetensors framework="np"

Effect on vLLM startup on this machine, with and without a workaround that copies each tensor into anonymous memory before the device copy:

model	as shipped	tensor cloned to anon memory first
Qwen3-8B, 15.26 GiB BF16	206 s	18.7 s
gemma-4-12B, 9.56 GiB w4a16	328 s	10.5 s
gemma-4-31B, 21.67 GiB w4a16	569 s	25.1 s

Those work out to 76, 30 and 39 MiB/s, which is milder than the 2.0 MiB/s of the microbenchmark and consistently so. All three sit between the two writable regimes measured above, which is what a partially resident mapping produces: nothing in the load path reads the pages before the copy, so they are faulted in by the copy itself and only the already-resident fraction pays the worst rate. Timed tensor by tensor inside one shard, the cost is roughly fixed at about 1 s per tensor above a threshold somewhere between 4 and 8 MiB, and negligible below it.

Possibly related, and one thing that does not help

ROCm/ROCm#5952 — "SVM mapping failure during sequential model loads (RDNA3 / RX 7900 GRE)" — is the nearest neighbour. It reports crashes and stalls rather than a reproducible slow copy, but it is the same subsystem on the same generation of hardware, its reporter describes VRAM filling extremely slowly, and in that thread @zichguan-amd reads the logs as an async copy of 27 648 bytes taking 25 s. That machine is bare metal (Ryzen 7800X3D), which our guest-only setup cannot match.

Two things keep us from calling it the same bug. The kernel-side analysis posted there by @Deaththegrim concerns svm_range_restore_work and the userptr restore worker, i.e. the eviction/restore path, whereas our profile sits in svm_range_set_attr → svm_range_validate_and_map → hmm_range_fault, the mapping path. And that thread's symptoms come and go with workflow, RAM pressure and kernel build, while the reproducer below is deterministic and finishes in half a minute. We offer this as an adjacent report rather than a duplicate; if the two do share a cause, the reproducer is probably the cheaper way in.

HSA_USE_SVM=0 does not help. Suggested by ROCm/ROCm#2433, where it recovers pre-5.6 hipHostRegister performance. Measured here: the pathological case is unchanged (16 036 ms vs 16 020 ms), and the read-only fast path gets worse (8 905 → 844 MiB/s) — so SVM is what makes read-only mappings fast, while the writable path is slow for some other reason.

Ruled out

hypothesis	result
disk throughput	1.5 GB/s measured on the same file with dd iflag=direct
HSA_USE_SVM=0	no effect on the pathological case; degrades the fast case
page cache cold vs warm	192 vs 212 MiB/s, no meaningful difference
file-backed vs anonymous per se	a read-only file mapping is full speed
pages not yet faulted in	that is the faster of the two writable cases
a fresh source range per copy	no effect on a read-only mapping
ext4 vs overlayfs vs tmpfs	2.0, 1.9 and 1.9 MiB/s — the backing filesystem is not a factor
page-unaligned range start, bf16 vs uint8	no effect
new device allocation per copy vs one reused buffer	no effect
swap / memory pressure	swap usage 0 throughout

Environment

• 2× Radeon RX 7900 XT (gfx1100), 20 GiB each
• ROCm 7.14, PyTorch 2.11, safetensors 0.8.0 — also reproduces on ROCm 7.0.0 with PyTorch 2.9, so the userspace version is not the variable
• Ubuntu 24.04.4 guest under Proxmox VE / QEMU with VFIO passthrough. The kernel that varies is the guest's: 7.0.0-28-generic for the severe case, 7.0.0-14-generic and 6.8.0-136-generic for the comparisons, all booted from the same install. The host runs 7.0.14-4-pve and is the bare-metal row. The GPUs report PCIE atomic ops is not supported on every guest kernel
• Checkpoints on ext4 on an NVMe SSD; the reproducer's test file likewise

Confirmed by revert. amdgpu.ko built from the linux-hwe-7.0 7.0.0-28.28~24.04.1 source with 342981fff328 as the only change, swapped into the running -28 kernel, same machine and userspace and reproducer: the copy goes from 16 019.7 ms to 17.0 ms, while the reproducer's other three cases stay where they were. Details in the comments below.

What is still open. Updated 2026-08-28: not the kernel question any more. Canonical shipped 7.0.0-30.30~24.04.1 on 2026-08-20 and it carries the follow-up in effect: the same reproducer on the same machine goes from 16 019.3 ms to 15.3 ms across that upgrade, with the two control rows unmoved (all three kernel states, raw, and the comment below). Bare metal came off this list earlier: @shineday999 reproduced the pathology below on an RX 7900 XTX under ROCm 7.2.1 on 7.0.0-28 with no VM involved, which settles the passthrough question from outside this machine. What remains open is the 4× to 8× permission penalty, which that kernel commit does not touch — it survives on -30 at 4.8× — and which is characterised on gfx1100 only, since we have no CDNA hardware.

What would help

1. 342981fff328 in the stable trees. It carries neither a Fixes: tag nor Cc: stable@vger.kernel.org, so it was never queued for backport, while c08972f55594 did reach -28. That asymmetry is the whole mechanism, and it is not Ubuntu-specific: any series tracking 7.0.y that took the first commit is in the same state. Per stable-kernel-rules.rst an already-mainlined commit can be requested by mail to stable@vger.kernel.org, and a request from the people who signed both commits would carry more weight than one from me.
2. A backport into Ubuntu's 7.0.0 in the meantime. -28 is the current HWE kernel, out of noble-updates and noble-security, so it reaches anyone who lets updates run on the distro ROCm supports first. Filed with Ubuntu as LP#2161985, which carries the revert result; raised here because the breaking commit is AMD's.
3. The 4× to 8× that survives on every kernel is a separate question and the more durable one. Taking the fault permission from the VMA rather than from the requested access means a read-only copy pays for breaking COW on every resident page. Is there a cheaper path for MAP_PRIVATE|PROT_WRITE ranges, or should the runtime stage them rather than map them for DMA?
4. Independently, torch.UntypedStorage.from_file(shared=False) mapping writable is what exposes every PyTorch user to this. That may be worth a separate conversation with the PyTorch maintainers.

Everything behind this report

https://github.com/cadamcat/dual-radeon-vllm

• benchmarks/repro-mmap-prot.py is the reproducer above, with the file path and the repeat count made configurable and each case run twice.
• docs/open-questions.md section 8 lists every hypothesis we tested and discarded, including an earlier root cause we published and then disproved with our own reproducer. If something here looks wrong, that section is the place to check first.

[cadamcat]

@ashetaia-amd, your ROCm/rocm-systems#6676 describes the path this report profiles: DmaBlitManager::getBuffer takes the pin branch, MemoryRegion::Lock goes through fmm_register_mem_svm_api and issues AMDKFD_IOC_SVM(OP_SET_ATTR), installing a persistent svm_range with an mmu_interval_notifier. OP_SET_ATTR is the svm_range_set_attr frame in the profile above. Your motivation there was different, allocator madvise churn under sustained async D2H rather than a slow copy, but it is the same machinery.

Would forcing the staged path have covered this case as well? The PR was closed by the stale bot on 2026-07-04 rather than on merit, so the flag is in no released ROCm and I have not been able to test it. It is the one remedy that would also address the ~4x penalty that survives on the good kernel, which the kernel regression above does not explain.

[ashetaia-amd]

yes, forcing the staged path (ROC_FORCE_STAGED_D2H) would have sidestepped this. With pinning disabled the source is never SVM-registered, ROCr does a CPU memcpy into a pre-pinned staging buffer, and a CPU read doesn't break COW, so hmm_range_fault never runs on your rw-p source. That PR was scoped to a different problem, and the long-term fix we took there won't help this copy path, so I don't want to point you at it.

Your COW hypothesis is right, and the exact site is kfd_svm.c:

readonly = !(vma->vm_flags & VM_WRITE);   // decided by the VMA, not by how the GPU will use it

A writable VMA makes us request HMM_PFN_REQ_WRITE, which forces COW to be broken on every resident page even though an H2D copy only reads the source. That's your ~3–4x, and it's on both kernels (the line predates them).

The ~700x on 7.0.0-28 is a separate kernel regression that amplified that same fault path, the near-perfect linear ~2 ms/page and the flat per-filesystem offset point at a fixed per-page synchronous cost, and VFIO/IOMMU is a prime suspect given it's the only environment here. That part needs a bisect inside the 7.0.0 series (loreggia's 7.0.0-14 vs -28 data in #5952 is the best starting boundary); it's KFD/mm-side, and your reproducer is exactly what we need for it. If you can get a single run on bare metal or with iommu=pt, that would tell us whether passthrough is a necessary ingredient.

[cadamcat]

@ashetaia-amd answering your question first: passthrough is not a necessary ingredient, and the reason is that the kernel is doing it.

Same VFIO guest, same ROCm 7.14 userspace, same container image, same reproducer, only the kernel ABI swapped:

environment	kernel	rw-p resident, 32 MiB
VFIO guest	7.0.0-28-generic	16 019.3 / 16 019.6 / 16 019.9 / 16 020.1 ms
VFIO guest	7.0.0-14-generic	18.6 / 18.9 / 20.2 ms
VFIO guest	6.8.0-136-generic	22.1 / 23.3 / 26.0 ms
bare metal	7.0.14-4-pve	24.1 / 24.3 / 25.1 / 28.6 ms

Model loading follows: 86.7 s on -28 against 14.4 s on -14 for the same 15.26 GiB checkpoint. iommu=pt was never the variable either — this host already boots with it, and the guest has no IOMMU of its own.

The cause is a backport that landed half a pair. drm/amdgpu: fix amdgpu_hmm_range_get_pages (c08972f55594) entered Ubuntu's 7.0.0 series in the -28 upload itself, listed under -28 in the changelog and absent from the -26 and -14 sections above it. It moved mmu_interval_read_begin() out of the per-chunk loop. The follow-up that repairs the consequence, drm/amdgpu: drop retry loop in amdgpu_hmm_range_get_pages (342981fff328), has never been applied. That commit's own message describes what we measured:

the captured notifier_seq is no longer refreshed across retries ... the "goto retry" therefore degenerates into a busy spin that simply burns CPU for the full HMM_RANGE_DEFAULT_TIMEOUT (~1s) window before finally bailing out with -EAGAIN ... it actively hurts the KFD userptr stack

HMM_RANGE_DEFAULT_TIMEOUT is 1000. Every number above is an integer multiple of it plus the ~20 ms of real work: 16 019 ms is sixteen windows, the 17 020 ms we measured on overlayfs and tmpfs is seventeen, and the per-tensor stalls in a real checkpoint load were 1005, 1042 and 1522 ms. That is why the spread was under a millisecond and why two filesystems differed by a flat second — one extra retry. My "fixed per-page synchronous cost" reading was wrong, and I was the one who put that reading in front of you.

I have rewritten the report above around this rather than leave the old framing standing, so the version you replied to is now in the edit history. The ratio that used to be in the title is gone on purpose: the numerator is a count of one-second timeouts, and expressing it as a multiplier is what made it look like a per-byte or per-page cost in the first place. The raw milliseconds are all still there.

It closes the loop with your VMA explanation rather than replacing it. Breaking copy-on-write is what advances the notifier sequence, so a read-only mapping never invalidates, never returns -EBUSY, and never enters the futile retry. The permission bug is the trigger; c08972f55594 turned each trigger into a full second.

Not yet proven by revert. What I have is the two commits' presence and absence, the -14/-28 boundary, timings landing on exact multiples of the constant in that path, and perf at 98.7% inside it. Building -28 with 342981fff328 applied is the test that settles it, and I intend to run it.

The practical ask is a backport into Ubuntu's 7.0.0, since -28 is the current HWE kernel out of noble-updates and noble-security. Filing that separately, flagging it here because the breaking commit is yours and you may want it tracked.

[cadamcat]

One more piece, and it explains why the pair got split rather than just that it did.

342981fff328 carries neither a Fixes: tag nor Cc: stable@vger.kernel.org, so it was never queued for the stable trees. c08972f55594 did reach Ubuntu's -28. A fix that is not tagged simply does not follow the commit it repairs, so this is not an Ubuntu packaging slip: any series tracking 7.0.y that picked up the first commit is in the same state, and I have not checked which others did.

Documentation/process/stable-kernel-rules.rst allows anyone to request an already-mainlined commit by mail to stable@vger.kernel.org with the subject, commit ID, rationale and target series. I can send that, but @ashetaia-amd, 342981fff328 is Honglei Huang's patch, reviewed by Christian König and committed by Alex Deucher, and c08972f55594 is Christian König's with Alex Deucher signing off on both. A request from that side would land better than one from someone with a reproducer and no history in the subsystem. Happy to send it if you would rather I did; I did not want to file it over your heads on commits that are yours.

I have added this to the report above under "What would help".

[cadamcat]

Proven by revert.

I built amdgpu.ko from the linux-hwe-7.0 7.0.0-28.28~24.04.1 source with 342981fff328 as its only change — it applies cleanly, net −8/+1 — and swapped it into the running -28 kernel. Same machine, same ROCm 7.14 userspace, same reproducer, same file, same gcc 13.3.0 Canonical built -28 with, and .config taken verbatim from /boot/config-7.0.0-28-generic.

amdgpu	rw-p resident, 32 MiB
stock -28	16 019.7 / 16 019.6 / 16 019.1 / 16 019.7 / 16 019.0 ms
-28 + 342981fff328	17.0 / 17.0 / 17.0 ms

The reproducer's other three cases do not move: r--p resident stays at 3.0 to 3.1 ms and rw-p not-resident at 14.5 to 14.6 ms, before and after. Only the case that entered the futile retry changes, which is what reverting that one loop and nothing else should look like.

The residual is also the number the arithmetic predicted. 16 019.7 ms is sixteen 1000 ms windows plus 19.7 ms of real work, and 17.0 ms lands in the band 7.0.0-14-generic gives natively, 18.6 to 20.2 ms.

Two things this does not settle. A module built from Ubuntu's source is not one Canonical ships, and this is still a passthrough guest, so a bare-metal Ubuntu -28 is untested. And the 5.6× that remains against a read-only copy of the same bytes is the VMA-permission penalty you identified, which 342981fff328 does not touch — that one is still yours and still the more durable of the two.

So the backport request is now backed by a direct test rather than by the changelog and the arithmetic. @ashetaia-amd, the offer to send the stable@vger.kernel.org mail myself stands, and so does the preference that it come from the signers.

[shineday999]

Additional sanitized reproduction from an AMD Radeon RX 7900 XTX (gfx1100) on Ubuntu 24.04.4 LTS with Linux 7.0.0-28-generic, ROCm/HIP 7.2.1 (HIP 7.2.53211), PyTorch 2.9.1+rocm7.2.1, Python 3.12.3, and ComfyUI 0.22.0. The system has 32 GB RAM and 24 GB VRAM. No hostnames, usernames, paths, IPs, serials, UUIDs, prompts, workflow JSON, credentials, or output images are included.

Observed workload: ComfyUI Z-Image Turbo, 1024x1024, 8 sampling steps. The complete prompt took 14:56 on one run and 16:24 on another. The actual 8-step sampler took about 8 seconds. The delay is before sampling, while loading the text encoder/model: the log shows ZImageTEModel_ loaded at 09:11:24, model initialization at 09:11:26, and Lumina2 not loading until 09:17:47; the same pattern recurred on the next prompt. The text encoder is reported as 7672 MB and Lumina2 as 11740 MB.

I also ran a minimal local safetensors/PyTorch probe in the same environment, using a newly generated 32 MiB file with four 8 MiB tensors (no private model data). After faulting in the CPU pages, each mmap-backed tensor transfer with tensor.to(cuda:0) took 5.006-5.180 seconds. Cloning the tensors to heap memory first and then transferring them all took 0.004 seconds. This is a direct same-host reproduction of the mmap-backed host-to-device transfer problem, not a disk throughput measurement.

ComfyUI exposes --disable-mmap, which appears relevant because it avoids mmap when loading safetensors; I have not yet completed an end-to-end Z-Image run with that flag. The current ComfyUI launch also uses --gpu-only, --cache-none, and --disable-dynamic-vram.

This looks closely related to this issue: #6523. Please confirm whether the mmap-backed safetensors -> HIP host-to-device path is affected on gfx1100 as well, and whether the kernel/amdgpu fix or ComfyUI --disable-mmap workaround should be applied. If useful, I can provide the minimal probe source and timing output in a follow-up, but it contains no model or private data.

[shineday999]

Confirmed A/B on the same host (gfx1100 / ROCm 7.2.1 / Linux 7.0.0-28).

A (baseline, safetensors mmap path used by default):

• Same Z-Image Turbo workflow, 1024x1024, 8 steps
• Wall time per prompt: 14:56 and 16:24 on two consecutive runs
• Sampler steps: ~8s for 8 steps in both
• 6+ minute gap between ZImageTEModel_ load and Lumina2 load (text encoder + diffusion stay on host)

B (added --disable-mmap to the ComfyUI launch, nothing else changed):

• prompt_id a1ec01c9-d861-464e-a997-d0130c615f0e
• execution_start 2026-07-30 06:21:49.316 UTC
• execution_success 2026-07-30 06:22:48.027 UTC
• Wall time: 58.7 seconds (0:58)
• Output: 1024x1024 PNG, success
• Same model files, same kernel, same ROCm, same torch

Standalone reproducer on the same host (32 MiB file, 4 tensors, no model data):

• mmap-backed tensor.to(cuda:0): 5.006-5.180 s per tensor
• clone to heap first, then to(cuda:0): 0.004 s for the same 32 MiB
  This matches the rw-p pathology. --disable-mmap steers safetensors to a heap path, so the host->device copy avoids the bad VMA.

Conclusion: --disable-mmap is an effective user-side workaround for gfx1100 / 7.0.0-28. The kernel-side fix (c08972f55594 + 342981fff328) is still wanted because not every app has an mmap opt-out. If a follow-up tracking issue is needed for the ComfyUI side, this is a clean end-to-end reproduction on gfx1100 bare metal. Thanks for tracking this.

[cadamcat]

@shineday999 confirmed, and thank you for the bare-metal run. Every measurement in this report is gfx1100, so yes, that path is affected: 2× RX 7900 XT on ROCm 7.14 here against your 7900 XTX on 7.2.1. Two ROCm versions agreeing is worth having on its own, since it says the userspace version is not the variable.

Your run also closes the one gap this report could not close itself, and it answers the bare-metal run asked for earlier in this thread. "What is still open" says a bare-metal Ubuntu -28 is untested because this machine is a passthrough guest, and passthrough had already been raised here as a suspect. Yours is the same distro and the same -28 with no VM in the way. I will update that section with credit once the detail below is settled. It leaves the other half standing: you ran the workaround, not a patched amdgpu.ko, so a module as Canonical ships it is still untested.

On which to apply, both, in different roles. --disable-mmap is the right thing today and your reading of why is right: it steers safetensors to a heap path, the same mechanism as --safetensors-load-strategy eager in vLLM and as the flag proposed in vllm-project/vllm#49991. The kernel side removes the cause rather than the exposure, and not every application offers that opt-out. 342981fff328 is already upstream and only needs a backport, filed with Ubuntu as LP#2161985.

One request, on the standalone probe. 5.006 s is five 1000 ms HMM_RANGE_DEFAULT_TIMEOUT windows plus 6 ms of real work, which is what the mechanism here predicts. 5.180 does not land as cleanly, and there is a gap worth understanding either way: an 8 MiB tensor costs one window on this machine, 1005 ms measured, against five on yours. Could you post the four individual timings rather than the range? If they are near-integer multiples, that is independent confirmation of the timeout reading on hardware I do not have. If they are not, I would rather know before quoting them.

[shineday999]

@cadamcat follow-up with the per-tensor breakdown you asked for. New fresh probe, 8 MiB tensor on the same gfx1100 / ROCm 7.2.1 / 7.0.0-28 box, each iteration opens its own MAP_PRIVATE|PROT_READ|PROT_WRITE mapping, faults every page in via a single bulk read, and times the transfer independently. No model or private data involved; the file is a 32 MiB blob of os.urandom on tmpfs.

run	tensor.to(cuda:0)	rate
0	4034.79 ms	1.98 MiB/s
1	4005.08 ms	2.00 MiB/s
2	4004.79 ms	2.00 MiB/s
3	4004.72 ms	2.00 MiB/s
heap baseline, 32 MiB anonymous	3.75 ms	8523.31 MiB/s

All four numbers land on four HMM_RANGE_DEFAULT_TIMEOUT windows plus single-digit milliseconds of real work, which is what your mechanism predicts. The earlier range I posted (5.006–5.180 s) was the same probe averaged across iterations, not four independent runs, so the spread there reflected run-to-run jitter, not a non-integer pattern. Sorry for the noise; the per-tensor version above is what was missing.

One observation that follows from this. Your 8 MiB tensor costs one window (~1005 ms) on the 2× RX 7900 XT with ROCm 7.14; the same 8 MiB on this RX 7900 XTX with ROCm 7.2.1 costs four windows (~4005 ms). Both are integer multiples and both are gfx1100, so the mechanism is the same and the userspace version is not the variable. The 4× gap sits at the boundary between HMM_RANGE_DEFAULT_TIMEOUT and the chunk size passed into hmm_range_fault (or whatever the equivalent is on 7.2.1), which is hardware- and kernel-dependent. The reproducer on this machine is fast to run, so I am happy to bisect the chunk size if it helps.

The same probe also gives a clean heap baseline at 8523 MiB/s for an anonymous 32 MiB copy, in the same order as your r--p 9575 MiB/s, so the PCIe link and the device path are fine. The two regimes in your table collapse to one another once MAP_PRIVATE|PROT_WRITE is replaced by anonymous memory; that is exactly what --disable-mmap does for safetensors, and what the follow-up workaround in vllm-project/vllm#49991 is doing for vLLM. So both the kernel fix and the user-side opt-out close the same gap.

Probe source and raw output are at /tmp/rocm6523_probe_per_tensor.py and /tmp/rocm6523_probe_per_tensor.log; they contain no private data and I can paste them inline if helpful.