[Bug]: Segfault (SIGSEGV) in `scene.build()` with an articulated robot on AMD `gfx942` (CDNA3 / MI300-class), works on `gfx950` (CDNA4 / MI350)

[ZJLi2013]

Bug Description

Summary

Building any scene that contains a robot with convex-decomposed collision
geometry (e.g. the bundled Franka Panda) crashes the process with a bare
SIGSEGV (exit code 139, no Python traceback) during scene.build() on an
AMD Instinct MI300X (gfx942, CDNA3) GPU.

The exact same image, code, and Genesis/Quadrants versions run fine on an AMD
Instinct MI350 (gfx950, CDNA4). The crash is therefore architecture
specific to gfx942, and is independent of box_box_detection and
use_gjk_collision (both toggles still crash at the same point).

A scene with only primitives (plane + box), with or without cameras and
rendering, builds and steps fine on the same gfx942 GPU — so general GPU kernel
codegen works. The crash only appears once a robot's convex collision geometry
is added.

This looks like the same class of LLVM ISel / codegen failure previously
reported in #2570 for the convex collision kernel on CDNA3, resurfacing on a
different code path with current versions.

Minimal reproduction

Pure Genesis, no third-party code, using the bundled Franka MJCF:

import genesis as gs

gs.init(backend=gs.gpu, logging_level="warning")
scene = gs.Scene(show_viewer=False)
scene.add_entity(gs.morphs.Plane())
scene.add_entity(gs.morphs.MJCF(file="xml/franka_emika_panda/panda.xml"))
print("ENTITIES_ADDED", flush=True)
scene.build()                      # <-- SIGSEGV here on gfx942
print("BUILD_OK", flush=True)
for _ in range(10):
    scene.step()
print("STEP_OK", flush=True)

Run inside the ROCm container:

python -u franka_baseline.py ; echo "EXIT=$?"

Observed behavior (gfx942)

Process dies with exit code 139 right after the neutral-config
self-collision filtering, with no Python traceback:

[Genesis] [WARNING] (MJCF) Approximating tendon by joint actuator for `finger_joint1`
[Genesis] [WARNING] (MJCF) Approximating tendon by joint actuator for `finger_joint2`
[Genesis] [WARNING] Neutral robot position (qpos0) exceeds joint limits.
[Genesis] [WARNING] Constraint solver time constant should be greater than 2*substep_dt.
                    timeconst is changed from `0.005` to `0.00833333`.
[Genesis] [WARNING] Filtered out geometry pairs causing self-collision for the
                    neutral configuration (qpos0): (6, 11), (7, 11), (14, 21),
                    (15, 20), (16, 23), (17, 22).
<process exits 139 — SIGSEGV, no traceback, "BUILD_OK" never printed>

STEP_OK / BUILD_OK are never reached.

Workarounds already ruled out (all still SIGSEGV at the same point, gfx942)

Attempt	Result
RigidOptions(box_box_detection=False) (the #2570 workaround)	still exit 139
RigidOptions(use_gjk_collision=True)	still exit 139
Plane + Box only (no robot)	OK (build + step)
Plane + Box + 2 cameras + cam.render()	OK (build + render)

Hardware / version matrix

Same image, same package versions; only the GPU differs:

GPU	Arch	Family	scene.build() with Franka	Source
AMD Instinct MI300X	gfx942	CDNA3	SIGSEGV (exit 139)	this report (minimal repro)
AMD Instinct MI350	gfx950	CDNA4	OK	prior full-pipeline runs on this node

Environment

Component	Version
Genesis	1.0.0
Quadrants	0.8.0
ROCm (container)	7.2.0
HIP	7.2.26015-fc0010cf6a
AMD clang / LLVM	22.0.0git (roc-7.2.0, 26014, 7b800a1)
PyTorch	2.10.0+rocm7.2.0
NumPy	2.2.6
Python	3.13
OS	Ubuntu 24.04.4 LTS
Base image	rocm/pytorch:rocm7.2_ubuntu24.04_py3.13_pytorch_release_2.10.0
Host ROCm	6.3.2-66

Questions

1. Is the convex collision narrow-phase kernel expected to codegen on gfx942
   with LLVM 22 / ROCm 7.2? Is there a known-good Genesis/Quadrants combination
   for CDNA3?
2. Is there a runtime/codegen flag to work around the gfx942 ISel crash short
   of changing GPU?

Steps to Reproduce

Minimal reproduction

Pure Genesis, no third-party code, using the bundled Franka MJCF:

import genesis as gs

gs.init(backend=gs.gpu, logging_level="warning")
scene = gs.Scene(show_viewer=False)
scene.add_entity(gs.morphs.Plane())
scene.add_entity(gs.morphs.MJCF(file="xml/franka_emika_panda/panda.xml"))
print("ENTITIES_ADDED", flush=True)
scene.build()                      # <-- SIGSEGV here on gfx942
print("BUILD_OK", flush=True)
for _ in range(10):
    scene.step()
print("STEP_OK", flush=True)

Run inside the ROCm container:

python -u franka_baseline.py ; echo "EXIT=$?"

Expected Behavior

no build error

Screenshots/Videos

No response

Relevant log output

Environment

GPU	Arch	Family	scene.build() with Franka	Source
AMD Instinct MI300X	gfx942	CDNA3	SIGSEGV (exit 139)	this report (minimal repro)
AMD Instinct MI350	gfx950	CDNA4	OK	prior full-pipeline runs on this node

Release version or Commit ID

Environment

Component	Version
Genesis	1.0.0
Quadrants	0.8.0
ROCm (container)	7.2.0
HIP	7.2.26015-fc0010cf6a
AMD clang / LLVM	22.0.0git (roc-7.2.0, 26014, 7b800a1)
PyTorch	2.10.0+rocm7.2.0
NumPy	2.2.6
Python	3.13
OS	Ubuntu 24.04.4 LTS
Base image	rocm/pytorch:rocm7.2_ubuntu24.04_py3.13_pytorch_release_2.10.0
Host ROCm	6.3.2-66

Additional Context

No response

[duburcqa]

@hughperkins

[hughperkins]

Before looking into this, just want to say, appreciate your very detailed and thorough report. Very useful. Thank you @ZJLi2013 🙌

[hughperkins]

(Note: no MI300X available in amd gpu cloud currently; will check again once GPUs are available).

[hughperkins]

Drafted a PR Genesis-Embodied-AI/quadrants#746 . Will try to build a pre-release wheel of Quadrants for you to try.

[ZJLi2013]

Drafted a PR Genesis-Embodied-AI/quadrants#746 . Will try to build a pre-release wheel of Quadrants for you to try.

hi, @hughperkins , I am actually from AMD (zhengjli@amd.com), will be glad to support MI300x in case

[hughperkins]

Woo, fantastic! 🙌

There are a number of areas where you could help enormously 🙌

I will brainstorm these ideas, and you can decide which bits you think sound potentially interesting to you.

Pretty much everything comes under the high level domain of 'performance optimization'. Genesis-world already runs on AMD hardware 🙌

Then beneath that there are a few sub-categories:

1. 'low hanging fruit' implementation improvements

• we have a portable shuffle family of operations, that implements shuffle, shuffle_down, shuffle_up
  • these all use generic shuffle
  • on AMD, native generic shuffle goes via shared memory per my understanding? (it's not actually called shared memory, but it's something conceptually similar I believe?)
  • on AMD, there are specialist operations available for shuffle down and shuffle up, which should be faster
  • it could be useful to migrate the Quadrants portable shuffle down and shuffle up to use ths specialized shuffle down and shuffle up ops potentially?
• there is an outstanding AMDGPU bug here AMDGPU modulo bug quadrants#749

2. correctness CI for CDNA

• we currently have CI for RDNA, using Amazon EC2 V520 GPUS, for Quadrants
  • and we have standardized to use WAVE64 everywhere, even on RDNA
• it would however be useful to run unit tests also on CDNA
  • as we see in this logged issue, there are some thing that fail on CDNA, even if they are passing on RDNA, using WAVE64
• so there is an opportunity for someone to setup and maintain a CDNA CI for Quadrants
  • would also be useful to have a similar correctness CI, for AMD GPU, on genesis-world

3. performance benchmarks CI for AMD GPUs

• we currently have performance benchmarks that run in genesis-world on CUDA GPUs (rtx6000 Blackwell)
• it could be useful and interesting to have performance benchmarks CI also running and configured for AMD GPUs, e.g. for CDNA
• details on the benchmarks we run
  • the benchmarks we run are at https://github.com/Genesis-Embodied-AI/genesis-world/blob/main/tests/test_rigid_benchmarks.py
  • the most important benchmark currently is: dex_hand, which represents learning a dexterous hand RL task
    • in addition, from AMD PoV, if you optimize only for dex_hand, performance on other benchmarks will likely improve monotonically with improvements on dex_hand
  • we run this benchmark in our CUDA CI using production.yml

    genesis-world/.github/workflows/production.yml

    Line 114 in 639131d

    benchmarks:

4. optimize genesis-world rigid benchmarks

Optimize the benchmarks you created in 3. above. This will likely a combination of:

• PRs onto genesis-world itself, in order to make various geometric and heuristic choices be reasonable on AMD hardware
• PRs onto Quadrants to improve the optimization of various operations on AMD (similar to the shuffle down and shuffle up operations proposals above)
• (maybe) changes to AMDGPU drivers, to optimize optimization choices and so on to work well for Genesis and Quadrants
• (maybe) changes to future AMD hardware, e.g. the proposals below to provide a fast hardware generic shuffle; and to provide conditional graph nodes

5. AMD GPU fundamental/underlying improvements

These would be long-term fixes, probably inside the hardware itself (likely not even at driver level I imagine?)

• implement a fast generic shuffle, that does not use shared memory or similar
• provide a conditional graph node
  • in order to keep everything running on the GPU, on full speed, we need to be able to run do while loops, or similar, on the gpu side, without needing to make decisions on the host side
  • in order to handle this, one way would be to provide conditional nodes in kernel graphs, such that decisions about whether to continue looping or not can be made on the GPU side

6. Personal high-level observation/opinion on WAVE32 vs WAVE64

Splitting AMD architecture between WAVE32 and WAVE64 doubles the amount of cognitive load required in order to support AMD

• in addition WAVE64 is an outlier
  • Metal, Intel, CUDA all use WAVE32 (or at least provide WAVE32, in the case of Intel)
• would be much cheaper to provide AMD support if AMD hardware standardized on WAVE32, everywhere (I assume this would need re-running floor planning for the GPU die, to take into account changing the SMPs from 64-lane to 32 or 16 lane; and then re-running verification etc; I don't claim this is a trivial amount of work, or easy)

Thoughts?

Hugh

[hughperkins]

Hi, for the issue originally flagged in this issue, what are your thoughts on how you want to proceed? Do you want me to create a pypi pre-release of Quadrants, so you can test whether it works correctly now? Do you want to look into learning how to build Quadrants yourself? Something else?

[ZJLi2013]

Hi, for the issue originally flagged in this issue, what are your thoughts on how you want to proceed? Do you want me to create a pypi pre-release of Quadrants, so you can test whether it works correctly now? Do you want to look into learning how to build Quadrants yourself? Something else?

hi, hugh, thanks a lot for the 1~6 features feedback, we are prioritying them now, will work out soon. For this bug itself, yes, I think it's better you can point me to build Quadrants myself

[hughperkins]

Sounds good. Here is the contributing.md doc for quadrants https://genesis-embodied-ai.github.io/quadrants/user_guide/contributing.html

But concretely, something like:

cd quadrants
uv venv --python '3.10'
source .venv/bin/activate
./build.py --shell
CMAKE_ARGS=-DQD_WITH_AMDGPU=ON pip install --no-build-isolation -e . -v

[ZJLi2013]

hi, @hughperkins , just ran the source build for MI300(AMD CDNA3) and also cross check on 9700(AMD RDNA4), Summary: the root cause is the Quadrants codegen bug that PR #746 addresses, and 746 is the general fix. thanks again.

Root cause. Quadrants emits llvm.amdgcn.permlane64 on CDNA (gfx942), where v_permlane64_b32 doesn't exist, and it crashes during branch relaxation (SIInstrInfo::getInstSizeInBytes). Present in both quadrants 0.8.0 and 1.0.2.

Two fixes, only one is complete:

1) Genesis-side (partial). genesis 1.1.1 rewrote the Franka self-collision kernel to use subgroup.bitonic_sort_kv_tiled (added in quadrants 1.0.2), which stops emitting the cross-half shuffle — so the Franka repro passes on released genesis 1.1.1 + quadrants 1.0.2. Version bisect on gfx942 (minimal Franka repro, HIP_VISIBLE_DEVICES=0):

Genesis \ Quadrants	0.8.0	1.0.2
1.0.0	💥 SIGSEGV 139 (this issue)	💥 SIGSEGV 139
1.1.1	⚠️ AttributeError: subgroup has no 'bitonic_sort_kv_tiled'	✅ BUILD_OK / STEP_OK

But this only covers the kernel that was rewritten. On the same released genesis 1.1.1 + quadrants 1.0.2 stack, other rigid scenes still crash on gfx942 — e.g. the go2 benchmark SIGSEGVs at build time while franka passes. So bumping Genesis is a per-kernel workaround, not a fix for the underlying bug.

2) Quadrants-side, PR 746 (complete). I built quadrants from hp/amdgpu-no-permlane64-on-cdna (commit 406a932, CMAKE_ARGS=-DQD_WITH_AMDGPU=ON) and validated it directly on gfx942:

Genesis	Quadrants	scene.build()
1.0.0	0.8.0 / 1.0.2 (stock)	💥 SIGSEGV 139
1.0.0	746 (source)	✅ BUILD_OK / STEP_OK

With 746 the whole rigid benchmark suite (dex_hand / franka / go2 / anymal / box_pyramid / g1_fall) builds and runs on gfx942. So 746 fixes the codegen bug at the root for all kernels — worth merging, since the Genesis rewrite alone leaves the bug live for any kernel using a cross-half shuffle.

RDNA (gfx1201 / R9700) — for RDNA users: not affected. RDNA has permlane64 in hardware, so this never triggers. The Franka repro passes on RDNA4 with both released genesis 1.1.1 + quadrants 1.0.2 and a source build of quadrants main (BUILD_OK / STEP_OK, exit 0). RDNA users can run stock; CDNA (gfx942 / MI300) users are the ones who need 746.

Net:

• RDNA (gfx1201, etc.): works out of the box.
• CDNA gfx942: genesis>=1.1.1 fixes the reported Franka crash, but not all kernels (go2 still crashes) — the general fix is PR 746, which I've confirmed works.

(Minor: on py3.12 quadrants prints DeprecationWarning: ast.Str is deprecated from function_def_transformer.py.)

Next step The complete fix needs both: merge #746 into Quadrants, and have Genesis pin a Quadrants version that includes it

[ZJLi2013]

Rigid Body Benchmark Baseline

Functional matrix on gfx942 (scenes from tests/test_rigid_benchmarks.py, build + a few scene.step()s at the benchmark n_envs; MI300X, host ROCm 6.3.2, rocm/pytorch:...py3.12, HIP_VISIBLE_DEVICES=0):

scene (n_envs)	released genesis 1.1.1 + quadrants 1.0.2	quadrants #746 (source)
franka (30000)	✅ OK	✅
franka_random (30000)	✅ OK	✅
anymal_random (30000)	✅ OK	✅
go2 (4096)	💥 SIGSEGV 139	✅
box_pyramid_5 (4096)	💥 SIGSEGV 139	✅
dex_hand (4096)	💥 SIGSEGV 139	✅
g1_fall (4096)	💥 SIGSEGV 139	✅

Released: 3/7 pass, 4/7 SIGSEGV. With #746: 7/7 pass.

Root cause & why the released stack still crashes. Quadrants emits llvm.amdgcn.permlane64 on CDNA (gfx942), where v_permlane64_b32 doesn't exist, crashing in branch relaxation (SIInstrInfo::getInstSizeInBytes) — present in both quadrants 0.8.0 and 1.0.2. genesis-world 1.1.1 worked around #2962 by rewriting the Franka self-collision kernel to use subgroup.bitonic_sort_kv_tiled (no more cross-half shuffle), but that's per-kernel: any other scene whose kernel still emits that shuffle hits the same crash. quadrants#746 gates permlane64 off CDNA at the compiler level, fixing all kernels at once — I built it from hp/amdgpu-no-permlane64-on-cdna (406a932, -DQD_WITH_AMDGPU=ON) and confirmed 7/7 pass.

This is an architecture-level result (gfx942 ISA), independent of the specific card, so it applies to MI300-class CDNA3 generally. RDNA (gfx1201, etc.) is unaffected — it has permlane64 in hardware (confirmed: Franka repro passes on R9700 with both stock and a source build of quadrants main).

Ask — a complete out-of-the-box fix on gfx942 needs two steps: (1) merge quadrants#746 and ship it in a Quadrants release, and (2) bump Genesis's quadrants pin to that release — every Genesis release currently pins quadrants==1.0.2 (pre-#746), so users can't get the fix via pip even once #746 lands.

(Minor: on py3.12 quadrants prints DeprecationWarning: ast.Str is deprecated from function_def_transformer.py.)

Related: quadrants#746.