[Comgr][hotswap] Decline far s_add_pc_i64 trampolines - #3305
Merged
xintin merged 1 commit intoJul 10, 2026
Merged
Conversation
xintin
marked this pull request as ready for review
July 10, 2026 06:44
xintin
force-pushed
the
amd/dev/xintin/hsv009-rccl-hotswap-fault-bisect
branch
from
July 10, 2026 06:52
070a2d0 to
3c24f63
Compare
xintin
force-pushed
the
amd/dev/xintin/hsv009-rccl-hotswap-fault-bisect
branch
from
July 10, 2026 06:54
3c24f63 to
c3e574d
Compare
chinmaydd
approved these changes
Jul 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
On gfx1250 A0, RCCL
all_reduce_perffaults 10/10 with hotswap enabled: every rank hits a GPU memory fault inncclDevKernel_Generic_4(page-not-present at ~0x10000X000).HSA_HOTSWAP_DISABLE=1passes 10/10.Root cause (bisected on hardware + rocgdb): when a B0 to A0 patch site sits beyond
s_branch's +/-128 KB reach of the appended trampoline pool (only happens in huge objects like the ~225 MB RCCL fatbin),emitToTrampolinetakes the far path, whose backwards_add_pc_i64branch-back corrupts wave state on gfx1250 A0. The forward positive-offsets_add_pc_i64, the DS-2addr expansion, and the near NOP-sled/short-s_branchpath are all correct - only the far backward long branch is at fault. The faulting kernel is unchanged; a trampolinedds_*_2addrsite in an RCCL device function (e.g.runTreeUpDown) corrupts data it reads.Fix
Decline far (
Long) trampolines inemitToTrampoline- leave the site's original instruction in place rather than emit the crashing redirect. In-place patches (s_clausetos_nop, etc.) and near short-s_branchtrampolines are unaffected. Unpatchedds_*_2addris only incorrect on A0 for payload-unaligned offsets, which is strictly safer than a hard GPU fault. Small objects (rocBLAS/hipBLASLt/etc.) never hit the far path, so they are unchanged. Follow-up (tracked separately): patch far sites via a scratch-registers_getpc_b64/s_setpc_b64branch-back, gated on an SGPR-liveness pass so it also covers non-kernel device functions.This is a gating mechanism; as a follow-up it's the getpc/setpc + SGPR-liveness work.
Testing
all_reduce_perf(float, 4x gfx1250), hotswap enabled: 10/10 PASS (was 0/10),#wrong=0, ~4.3 GB/s.hotswap-trampoline-ds-long-branch.sregression test encodes the RCCLds_2addrfar-path scenario (load + store, declined).hotswap_rewrite20/20 +hotswap_gfx_query10/10.