[Issue]: "operation cannot be performed in the present state" — on RCCL 2.30.4 the hostcall declaration survives NDEBUG and a device image with zero __ockl_* symbols

[cadamcat]

Following up on #6074, where @harkgill-amd narrowed this down to a PCIe atomics
dependency introduced in RCCL with ROCm 7.2.1, and said the open question is whether
to keep atomics as an explicit dependency or remove it.

This report is about where the dependency actually lives, because it turns out not
to be in the code. On RCCL 2.30.4 the failure survives a build with zero
hostcall-calling instructions, which means removing the device-side assert() would
not be sufficient. That seemed worth its own issue rather than a comment.

The part that bears on the keep-or-remove decision

We built RCCL 2.30.4 (ROCm/rocm-systems, projects/rccl) with
add_compile_definitions(NDEBUG), for seven architectures. It still fails, at
hipify/src/enqueue.cc:2118. Static inspection of the linked device image:

check	result
__assert_fail	0 — NDEBUG did its job
__ockl_fprintf	0 — COLLTRACE is gone from 2.30.4 entirely
__ockl_* of any kind	0 — no hostcall-calling code remains at all
hidden_hostcall_buffer in kernel metadata	3 — one per ncclDevKernel_Generic_{1,2,4}

So the metadata declaration alone is enough for ROCr to refuse dispatch. Removing
every instruction that could use hostcall does not help while the kernel still
declares the implicit argument.

Bisected to the link step:

device_build/common.o             (before device link)  hostcall = 0
device_build/gfx1100/device.elf   (after device link)   hostcall = 3

2.30.4 replaced the ordinary HIP fat-binary path with its own device linker
(tools/rccl-device-compile --link, which also hand-patches SGPR/VGPR fields in the
kernel descriptor). That tool contains no hostcall or hidden_ strings, so the
declaration appears to be emitted during the dispatcher link, conservatively, for
kernels that demonstrably never call hostcall. Ruled out with trivial-kernel probes
(all give hostcall = 0): plain whole-program compile, -fgpu-rdc,
-mcode-object-version=4, and -fgpu-rdc + COv4. So it is neither RDC as such nor
the code-object version.

If the decision is to drop the dependency, the device-link step looks like the place
it has to be dropped. One workaround we have not tried: strip the implicit-argument
entry from the kernel metadata after linking, since RCCL's own tool already rewrites
that region.

What works today, for anyone arriving here from a search engine

If the cards are passed through to a QEMU/KVM guest, check the passthrough form
before rebuilding anything. Passing the GPU as a single function rather than
alongside its audio function gives the guest AtomicOp completer support, and with
atomics present stock RCCL needs no patch at all, 2.30.4 included. Measured here on
2026-08-23, one line of VM configuration and nothing else changed:
https://github.com/cadamcat/dual-radeon-vllm/blob/main/docs/vfio-atomics.md

Where atomics cannot be had, RCCL 2.27.7 rebuilt with NDEBUG does remove the
declaration, and multi-GPU works: gemma-4-31B at 43.2 tok/s under vLLM TP=2 on
2× RX 7900 XT, both GPUs at 265 W synchronised. 2.30.4 is not fixable this way, so
2.27.7 is the only rebuild route we have verified. Build script, deployment steps and
prebuilt libraries: https://github.com/cadamcat/dual-radeon-vllm

A reproducer that needs no RCCL

Useful for triage, and for telling this failure apart from the several other things
that produce the same message. 30 lines of HIP, one GPU is enough. Two kernels with
identical launch shape; the only difference is that the second needs a hostcall
because it calls printf:

devices=2
--- dev0 ---
[plain    ] launch:no error               sync:no error
[hostcall ] launch:the operation cannot be performed in the present state sync:the operation cannot be performed in the present state
--- dev1 ---
[plain    ] launch:no error               sync:no error
[hostcall ] launch:the operation cannot be performed in the present state sync:the operation cannot be performed in the present state

Source: diagnose/hipgate3.cpp in the repository above. Driver-level confirmation
with AMD_LOG_LEVEL=4, at the moment RCCL fails:

ncclDevKernel_Generic
rocvirtual.cpp:4208  Pcie atomics not enabled, hostcall not supported
rocvirtual.cpp:4636  AQL dispatch failed!
                     -> hipErrorIllegalState

Telling affected machines apart from unaffected ones

This may be why the failure has been awkward to reproduce on request: whether a
machine shows it is decided entirely by the root port, not by the GPU, the slot width
or the RCCL build. One command, no ROCm needed:

dmesg | grep "PCIE atomic"
# affected:     amdgpu 0000:01:00.0: PCIE atomic ops is not supported
# not affected: no output

Grep the exact phrase; a bare grep -i atomic also matches unrelated
DMA: preallocated ... pool for atomic allocations lines from early boot.

On the machine in this report:

[    9.044936] amdgpu 0000:01:00.0: PCIE atomic ops is not supported
[    9.568264] amdgpu 0000:02:00.0: PCIE atomic ops is not supported

0000:00:1c.0: AtomicOpsCap: Routing- 32bit- 64bit- 128bitCAS-

The 32bit- 64bit- on that root port is the upstream cause of the amdgpu line. pci_enable_atomic_ops_to_root() requires 32- and 64-bit AtomicOp completer support on the root port above the device, and AtomicOp routing only on the switch ports in between; a root port's own Routing bit describes peer-to-peer between root ports and is not part of the test. Whether QEMU's emulated root port advertises completer support depends on how the device below it is passed, not on the host CPU. vfio_pci_enable_rp_atomics() in hw/vfio/pci.c sets it up automatically and has since v8.1.0, and gives up on pdev->devfn or QEMU_PCI_CAP_MULTIFUNCTION among other conditions. Proxmox's default hostpciX: 0000:0b:00 form hands the GPU over together with its audio function and takes that branch, which is what produced the 32bit- 64bit- above; passing the same card as 0000:0b:00.0 gives the guest 32bit+ 64bit+ and the GPU AtomicOpsCtl: ReqEn+, measured here on 2026-08-23 with nothing else changed. So a passthrough Instinct is affected when it is passed multifunction and not when it is passed as a single function -- a configuration rather than a property of virtualisation. The bare-metal reports in #6074 and vllm-project/vllm#38587 are single-NUMA desktops (7800X3D and 13900K) with 2× 7900 XTX.

Edited twice. This paragraph first said Routing- on the root port was the cause, which is the wrong capability bit (correction and evidence). It then said every QEMU root port declines completer support regardless of what sits behind it, and drew from that the claim that this reaches any VFIO passthrough guest including virtualised Instinct. That is wrong for a single-function passthrough, and the A/B is one line of VM configuration: vfio-atomics.md. The severity claim is narrowed accordingly rather than withdrawn, since the multifunction form is the Proxmox default.

Two conclusions of ours that were wrong

Recorded because they cost us several days and may be costing others the same.

1. "It is an RCCL regression between 2.27.7-b38 and -b43." Plausible, since a
   downgrade does fix it, but building b38 from source on the newer runtime fails the
   same way. The version correlation is real, the causation is not.
2. "ROCm ≥ 7.13's HIP runtime gates kernel submission in VFIO guests." Disproved by
   a minimal probe: hipExtLaunchKernel with a completion event, 4 KB kernargs and
   cross-stream waits all pass on the same runtime. Submission is fine; only
   hostcall-carrying kernels are refused.

13 hypotheses were tested and 12 eliminated (kernel version, iommu=pt, event flags,
memory, NUMA, SDMA, P2P, cuMem, shared-memory size, transport layer, and others).
The matrix is in docs/root-cause.md in the repository.

Two smaller asks

• On 2.27.7, we did not test COLLTRACE=OFF on its own, which is the supported knob
  rather than a patch; NDEBUG happens to remove both sources at once. If disabling
  COLLTRACE alone is sufficient there, that is the cleaner answer and we would be glad
  to hear it.
• hipErrorIllegalState at enqueue.cc sends people hunting through RCCL versions and
  environment variables for days. A message along the lines of "hostcall unavailable:
  PCIe atomics disabled" would end that search immediately, whatever is decided about
  the dependency itself.

Environment

• 2× Radeon RX 7900 XT (gfx1100), cross-die, PCIe 3.0, no P2P
• Host: Threadripper 1950X / X399; Proxmox VE + QEMU 11.0.2, VFIO passthrough
• ROCm 7.14, PyTorch 2.11, vLLM 0.23. Also reproduced on ROCm 7.13 and 7.0.0
• Last working combination before the rebuild: the ROCm 7.0.0 image, whose RCCL 2.26.6
  device kernels predate the hostcall dependency

[adderek]

A rebuilt RCCL with NDEBUG now works for me (or so it seems).
I am using llama.cpp and there I can split models (-sm layer with around 8% cost, -sm row around 30%). I need to check experimental -sm tensor in llama.cpp which should be using RCCL (which seems fixed for me now).
Let AI summarize (it did most of the boring work anyway ;) )

Investigated and written up with the help of an AI coding assistant (Claude). Every
command below was run on real hardware and all output is pasted verbatim; the
analysis is mine to defend, but flagging the provenance so the reasoning gets the
scrutiny it deserves.

---

Answering the COLLTRACE=OFF question: not sufficient on 2.27.7, and assert() is why

You asked whether disabling COLLTRACE on its own is enough on 2.27.7, since it is the
supported knob rather than a patch. Tested on a third, independent machine — the
answer is no, and the reason is worth recording.

Environment

• 2× RX 7900 XTX (gfx1100), bare metal, no VFIO
• ASRock B550 PG Riptide, Ryzen 9 PRO 3900 (Matisse), Arch Linux, kernel 7.1.2-zen3-1-zen
• ROCm 7.2.4, RCCL 2.27.7 (distro package rccl 7.2.4-1)
• IOMMU entirely disabled (no IVRS table, zero IOMMU groups) — so iommu=pt is
  a no-op here, and RCCL's init warning about it is a red herring on this box

This machine is only partially affected, which may be a useful test case: one
GPU has PCIe atomics and the other does not.

amdgpu 0000:06:00.0: PCIE atomic ops is not supported     <- PCIE3, behind the B550 chipset
                                                          <- no line at all for 0000:0b:00.0
                                                             (PCIE1, CPU-direct)

Your hipgate3-style probe reproduces per-GPU rather than on both:

=== GPU 0 (gfx1100) ===          <- chipset-attached
  hostcall (printf)    launch+sync: OK    lastError: the operation cannot be performed in the present state
  plain compute        launch+sync: OK    lastError: OK
  => hostcall BROKEN, ordinary compute WORKS

=== GPU 1 (gfx1100) ===          <- CPU-direct
  [device] hostcall printf reached the host
  hostcall (printf)    launch+sync: OK    lastError: OK
  => hostcall WORKS, ordinary compute WORKS

Two details from that, in case they help triage:

• hipDeviceSynchronize() returns success even though the dispatch was rejected.
  The failure shows up only in hipGetLastError(), and the printf output silently
  never appears. That makes it easy to miss outside RCCL.
• Only hostcall-declaring kernels are refused. Ordinary compute dispatches and returns
  correct results on the same GPU, which is why single-GPU inference on that card is
  completely healthy.

Static counts

Method — unbundle the linked device image and count markers:

objcopy -O binary --only-section=.hip_fatbin librccl.so.1.0 fb.bin
/opt/rocm/llvm/bin/clang-offload-bundler --type=o --input=fb.bin --unbundle \
  --targets=hipv4-amdgcn-amd-amdhsa--gfx1100 --output=dev.co
strings dev.co | grep -c hidden_hostcall_buffer

build	__assert_fail	__ockl_fprintf	__ockl_hostcall	hidden_hostcall_buffer	collectives
distro rccl 7.2.4-1 as shipped	5	3	0	6	fail
-DCOLLTRACE=OFF	4	3	0	3	fail
-DCOLLTRACE=OFF -DCMAKE_CXX_FLAGS=-DNDEBUG	0	0	0	0	pass

COLLTRACE=OFF halves the declarations but does not remove them, because HIP's
device assert() is itself a hostcall user — it routes through __ockl_fprintf to
print its message before aborting. That is why __ockl_fprintf stays at 3 with
COLLTRACE off. The asserts, not COLLTRACE, are the load-bearing source, so the
supported knob alone will not do it; NDEBUG is what does the work.

(To be precise about what was built: I tested COLLTRACE=OFF alone and
COLLTRACE=OFF + NDEBUG. I did not build NDEBUG alone — you already report that
combination working on 2.27.7, and these counts explain why it does.)

The corollary may matter more than the answer

CMAKE_BUILD_TYPE=Release does not imply NDEBUG for RCCL's device compile.

The distro package above is a Release build and still ships 5 __assert_fail and 6
declarations. If that generalises to other distro and vendor RCCL packages, then
every 2.27.7-era binary is affected out of the box on any machine without atomics —
and adding NDEBUG to the device compile would fix that whole class at the source,
independently of whatever is decided for 2.30.4's device linker.

Confirming your 2.27.7 result

Rebuilt for gfx1100 only, RCCL at tag rocm-7.2.4:

cmake -S rccl -B build-ndebug -G Ninja \
  -DCMAKE_BUILD_TYPE=Release -DCMAKE_PREFIX_PATH=/opt/rocm \
  -DCMAKE_CXX_COMPILER=/opt/rocm/bin/hipcc \
  -DGPU_TARGETS=gfx1100 -DCOLLTRACE=OFF -DBUILD_TESTS=OFF \
  -DCMAKE_CXX_FLAGS="-DNDEBUG"

Two-process ncclAllReduce, one GPU per rank, values checked rather than just
exit status:

  loaded library: build-ndebug/librccl.so.1        <- banner-gated, so the rebuild is provably what loaded
[rank 0] hipIpcGetMemHandle              OK
[rank 1] hipIpcOpenMemHandle           OK, peer read = 42.0 (want 42.0) PASS
[rank 0] ncclCommInitRank                OK
[rank 1] ncclCommInitRank                OK
[rank 0] ncclAllReduce                   OK  PASS (host[0]=3.0, 0/1048576 wrong)
[rank 1] ncclAllReduce                   OK  PASS (host[0]=3.0, 0/1048576 wrong)
==== VERDICT: RCCL 2-GPU COLLECTIVES WORK ====

Note this works even though one of the two GPUs still has no PCIe atomics —
consistent with the declaration, not any actual hostcall use, being the whole
problem.

Caveat on confidence: this is one clean end-to-end run. A follow-up stability
loop could not be completed because the chipset-attached GPU wedged partway through
for an unrelated reason (failed to load ucode RS64_MEC_P1_STACK → PSP resume failed → amdgpu_device_ip_resume failed (-22) after a runtime-PM suspend/resume
cycle, power/runtime_status = error). Those runs failed with "no ROCm-capable
device is detected", not an RCCL error. I would not present this as a soak-tested
result.

One thing that does not work, in case it saves someone a download

Downgrading the RCCL binary to the ROCm 7.1.1 build — suggested in
vllm-project/vllm#38587 — does not help here. Verified with the version banner
confirming the older library actually loaded (ROCm version : 7.1.1.0,
Librccl path pointing into the extracted tree), and it fails identically. Matches
your finding that the version correlation is real but the causation is not.

On the topology discriminator

Your dmesg | grep "PCIE atomic" check is the right one and it worked here. One
refinement for bare-metal chipset cases — it is worth identifying which bridge
drops routing, because it decides whether reslotting can help:

00:01.2   CPU -> chipset GPP bridge          AtomicOpsCap: Routing+
03:00.0   500-series chipset Switch Down     AtomicOpsCap: Routing-     <- breaks here

The CPU side advertises routing fine; the break is inside the chipset switch. Since
every chipset-fed slot hangs below that downstream port, no amount of moving the card
between chipset slots helps — only CPU-direct lanes do. (lspci honours only the
last -s, so query one device per invocation.)

Strongly seconding the error-message ask

hipErrorIllegalState at enqueue.cc cost me a day too, including a wrong turn
through HIP IPC (hipIpcGetMemHandle) that turned out to be a red herring. Something
like "hostcall unavailable: PCIe atomics not enabled on device N" would have ended it
in minutes, whatever is decided about the dependency itself.

[cadamcat]

A correction to my own report, because it changes how someone should test their platform.

I wrote that Routing- on the root port is what makes amdgpu disable atomics. That is not the bit the kernel reads. pci_enable_atomic_ops_to_root() checks 32- and 64-bit AtomicOp completer support on the root port, and requires AtomicOp routing only on switch ports strictly between the device and that root port. A root port's own routing bit describes peer-to-peer between root ports and is never consulted.

For passthrough guests it depends on how the device is handed over, which I did not know when I wrote this: QEMU's emulated pcie-root-port advertises 32bit- 64bit- for a multifunction passthrough, which is the Proxmox default and is this machine, and 32bit+ 64bit+ for a single-function one. vfio_pci_enable_rp_atomics() has drawn that line since QEMU v8.1.0. Bare metal is where the distinction bites. My host reports Routing- 32bit+ 64bit+ on all eight root ports, and the driver confirms that passes rather than me reading bits at it: before these cards were handed to vfio-pci, amdgpu bound them on the host across five boots and never printed PCIE atomic ops is not supported, while in the guest it prints for both GPUs every time. Same cards, same silicon, and the passthrough layer is the entire difference. What breaks @adderek's B550 is a third thing again, a chipset downstream port that does not route.

Reading Routing on a root port, as my diagnostic script did until today, calls healthy platforms broken. That is fixed, and I have corrected the issue body to match.

Edited 2026-08-28: the opening of the second paragraph said passthrough guests never get completer support. They do, when the device is passed as a single function -- measured on this machine on 2026-08-23, one line of VM configuration and nothing else changed: vfio-atomics.md.

[cadamcat]

Thanks @adderek — bare metal with IOMMU entirely disabled closes the one caveat I could not close from a VFIO guest. And your split machine, same host and driver, one chipset-attached GPU affected and one CPU-direct GPU healthy, is a cleaner controlled test than two identical slots would have let me run. Both are in the repository now, credited.

Your COLLTRACE=OFF counts answer the question I asked: not sufficient, and device assert() is why. The corollary matters more. CMAKE_BUILD_TYPE=Release does not carry NDEBUG into RCCL's device compile, so distro packages ship affected out of the box, and adding it there fixes that class at the source, independently of what is decided for 2.30.4's device linker. @harkgill-amd, that looks like the cheap half of the keep-or-remove decision.

Your output also caught a defect in my own probe. hipgate3.cpp read only the launch and sync return codes, so on your machine it would have reported a pass; it now checks hipGetLastError() and a device-side marker, and iterates per device. Your bridge-chain walk is now in check-platform.sh, with credit: reporting only the first hop up hides the 00:01.2 Routing+ / 03:00.0 Routing- case, the one where reslotting cannot help.

[adderek]

Glad to help. If you need any additional tests or platform details from my side, just let me know - I can run more scenarios on this machine.

Human-readable rephrasing of this issue: asserts were not disabled as expected → crash

[bkvargyas]

Confirming this on different silicon and adding three data points, including one that makes this bug the direct blocker for GPU P2P in VMs.

Setup: 2× Radeon AI PRO R9700 (gfx1201/RDNA4), Proxmox VE 9.2 / pve-qemu 11.0.3 VFIO passthrough, EPYC 7543 host, guest kernel 7.1.10 (XanMod, CONFIG_HSA_AMD_P2P=y), vLLM TP=2 (stilldeadcode/vllm-radiance:0.9.3, ROCm 7.14, RCCL 2.30.4).

1. Still present in RCCL 2.30.7 / ROCm 10.1 nightly. TheRock nightly torch 2.14.0+rocm10.1.0a20260827 (HIP 7.16) fails identically — HIP failure 'the operation cannot be performed in the present state', now at hipify/src/enqueue.cc:2255 (was 2061 in 2.30.4). Minimal repro: 2-process torch.distributed all-reduce.

2. Modern pve-qemu does advertise AtomicOp completers on direct root ports — when the host CPU supports them. On a plain hostpci (flat) topology, my guest root port shows AtomicOpsCap: Routing- 32bit+ 64bit+, the GPU gets AtomicOpsCtl: ReqEn+, no PCIE atomic ops is not supported in dmesg, and stock RCCL works. This differs from @cadamcat's X399 host (Zen1, no host-side completer support) and suggests the vfio AtomicOps auto-enable path works on Zen3+ hosts for devices directly on pcie-root-port.

3. The catch-22: enabling GPU↔GPU P2P kills atomics, which trips this bug. To get pci_p2pdma/KFD peer links in a guest, both GPUs must share an upstream bridge — i.e. an emulated switch (x3130-upstream + xio3130-downstream). But the switch ports advertise AtomicOpsCap: Routing- only (no completer/routing chain), so behind them both GPUs log PCIE atomic ops is not supported, and every RCCL collective launch fails with this exact error while torch.cuda.can_device_access_peer(0,1) is True and P2P SDMA copies run fine. So the only guest topology that allows P2P is precisely the topology this hostcall/atomics dependency breaks.

Workaround verified on gfx1201: RCCL 2.27.7 (release/rocm-rel-7.1.1.1) with project-level add_compile_definitions(NDEBUG) per @cadamcat's build script, -DGPU_TARGETS=gfx1201 → hidden_hostcall_buffer count 0 → collectives work on the atomics-less switch topology, including with KFD P2P active and a P2P one-shot all-reduce layered on top. End-to-end effect on vLLM TP=2 was large: cold prefill 2,191 → 4,122 tok/s at 7k ctx (+88%), 2,886 → 3,511 at 30k, aggregate decode +15–29% at 8 streams. One packaging note: the torch build I run binds several 2.30-era symbols eagerly (ncclDevCommCreate/Destroy, ncclPutSignal/ncclSignal/ncclWaitSignal, ncclGetLsaMultimemDevicePointer, and C++-mangled ncclCommDump), so the 2.27.7 drop-in needs small stub libs via patchelf --add-needed.

Given P2P-in-VM setups land on this by construction, +1 for fixing the unconditional hidden_hostcall_buffer emission in the 2.30.x device linker (or gating the ROCr hostcall requirement) rather than relying on rebuilds.

[cadamcat]

@bkvargyas The switch-versus-atomics catch-22 in your point 3 is the part I had not seen stated anywhere, and this box is the other horn of it: single-function passthrough, so AtomicOpsCtl: ReqEn+ and stock RCCL works, but isAllDirectP2p 0 and everything goes over SHM. You took the switch and got P2P without atomics; I took the flat topology and got atomics without P2P. Neither of us can have both in a guest, which makes the hostcall dependency unavoidable for any VM user who wants P2P rather than merely inconvenient.

Small correction while I am here: the X399 host does advertise completer support, all its root ports report AtomicOpsCap: Routing- 32bit+ 64bit+ and bare metal never emits the message. What removed it was QEMU's multifunction condition, not Zen 1, and the A/B is one line of VM config: https://github.com/cadamcat/dual-radeon-vllm/blob/main/docs/vfio-atomics.md So your EPYC result and mine agree more than they look: on a flat topology QEMU advertises completers whenever the host root port has them.

One data point for your item 1. With atomics present, stock unpatched RCCL 2.30.4 is fine here: 135 cold ncclCommInitRank today across three transport configurations and a 1-to-16 channel sweep, all collectives correct, no rebuild. So on gfx1100 the 2.30.x device-linker behaviour is only fatal when atomics are missing, which lines up with your gfx1201 result and with the fix belonging at hostcall emission rather than in RCCL's logic.

[Qbason]

+1
I got the same problem in llama.cpp while building with RCCL and running with parameter --split-mode tensor
proxmox - gpu passthrough
two rx 7900 xtx