[SPARC64] PLT relocations might point to the wrong place when linking

[koachan]

This is another case I found when building LLVM. I found that PLT relocations might end up pointing somewhere else after linking.
For example, in this snippet:

  88:   f8 5e 00 00     ldx  [ %i0 ], %i4
  8c:   c2 5f a6 97     ldx  [ %fp + 0x697 ], %g1
  90:   fa 5d c0 01     ldx  [ %l7 + %g1 ], %i5
                        90: R_SPARC_GOTDATA_OP  TLS init function for llvm::parallel::threadIndex
  94:   02 c7 40 04     brz,pn   %i5, a4 <(anonymous namespace)::ConcurrentHashTableTest_AddStringMultiplueEntriesWithResize_Test::TestBody()::{lambda()#1}::operator()() const+0xa4>
  98:   01 00 00 00     nop 
  9c:   40 00 00 00     call  9c <(anonymous namespace)::ConcurrentHashTableTest_AddStringMultiplueEntriesWithResize_Test::TestBody()::{lambda()#1}::operator()() const+0x9c>
                        9c: R_SPARC_WPLT30      TLS init function for llvm::parallel::threadIndex
  a0:   01 00 00 00     nop 
  a4:   40 00 00 00     call  a4 <(anonymous namespace)::ConcurrentHashTableTest_AddStringMultiplueEntriesWithResize_Test::TestBody()::{lambda()#1}::operator()() const+0xa4>
                        a4: R_SPARC_TLS_GD_CALL llvm::parallel::threadIndex
  a8:   d0 5f a6 9f     ldx  [ %fp + 0x69f ], %o0
  ac:   c2 02 00 00     ld  [ %o0 ], %g1

After linking with mold, the R_SPARC_WPLT30 call points to another place, causing crashes during runtime:

  488000:       f8 5e 00 00     ldx  [ %i0 ], %i4
  488004:       c2 5f a6 97     ldx  [ %fp + 0x697 ], %g1
  488008:       01 00 00 00     nop 
  48800c:       02 c7 40 04     brz,pn   %i5, 48801c <(anonymous namespace)::ConcurrentHashTableTest_AddStringMultiplueEntriesWithResize_Test::TestBody()::{lambda()#1}::operator()() const+0xa4>
  488010:       01 00 00 00     nop 
  488014:       7f ed df fb     call  0 <SmallSetTest_Grow_Test::~SmallSetTest_Grow_Test()>
  488018:       01 00 00 00     nop 
  48801c:       90 01 c0 08     add  %g7, %o0, %o0
  488020:       d0 5f a6 9f     ldx  [ %fp + 0x69f ], %o0
  488024:       c2 02 00 00     ld  [ %o0 ], %g1

For comparison this is the same code after linking with GNU ld, with the call pointing to the symbol's entry in the PLT:

  4101e0:       f8 5e 00 00     ldx  [ %i0 ], %i4
  4101e4:       c2 5f a6 97     ldx  [ %fp + 0x697 ], %g1
  4101e8:       fa 5d c0 01     ldx  [ %l7 + %g1 ], %i5
  4101ec:       02 c7 40 04     brz,pn   %i5, 4101fc <(anonymous namespace)::ConcurrentHashTableTest_AddStringMultiplueEntriesWithResize_Test::TestBody()::{lambda()#1}::operator()() const+0xa4>
  4101f0:       01 00 00 00     nop 
  4101f4:       40 1f c2 fb     call  c00de0 <TLS init function for llvm::parallel::threadIndex@plt>
  4101f8:       01 00 00 00     nop 
  4101fc:       01 00 00 00     nop 
  410200:       d0 5f a6 9f     ldx  [ %fp + 0x69f ], %o0
  410204:       c2 02 00 00     ld  [ %o0 ], %g1

[rui314]

How did you build LLVM? On what distro? I think I need to reproduce the issue locally to debug this issue.

[koachan]

I'm personally on Gentoo, although the issue is also observable on LLVM's SPARC buildbot machine (e.g https://lab.llvm.org/staging/#/builders/82/builds/13672) which uses Debian.
Both mold and LLVM are of their respective latest main, both of them are compiled with system compiler (which is GCC 15.2.0).

LLVM is configured with:

cmake -G "Ninja" -DCMAKE_BUILD_TYPE="Release" -DCMAKE_INSTALL_PREFIX="~/llvm-build" -DLLVM_ENABLE_PROJECTS="clang" -DLLVM_ENABLE_ASSERTIONS=True -DLLVM_LINK_LLVM_DYLIB=ON -DLLVM_TARGETS_TO_BUILD="Sparc" -DLLVM_USE_LINKER=mold ../llvm

And upon running ninja check-all, some of the testcases would crash in the manner I described above.

[rui314]

Which tests failed in your environment? I just build clang on a SPARC64 host with "GCC: (Debian 15.2.0-7) 15.2.0", and only the following tests failed when I run ninja check-all.

Failed Tests (5):
  LLVM :: DebugInfo/Generic/discriminated-union.ll
  LLVM-Unit :: ADT/./ADTTests/24/110
  LLVM-Unit :: ADT/./ADTTests/25/110
  LLVM-Unit :: DWARFLinkerParallel/./DWARFLinkerParallelTests/1/3
  LLVM-Unit :: DWARFLinkerParallel/./DWARFLinkerParallelTests/2/3

[koachan]

Which tests failed in your environment? I just build clang on a SPARC64 host with "GCC: (Debian 15.2.0-7) 15.2.0", and only the following tests failed when I run ninja check-all.

Failed Tests (5):
  LLVM :: DebugInfo/Generic/discriminated-union.ll
  LLVM-Unit :: ADT/./ADTTests/24/110
  LLVM-Unit :: ADT/./ADTTests/25/110
  LLVM-Unit :: DWARFLinkerParallel/./DWARFLinkerParallelTests/1/3
  LLVM-Unit :: DWARFLinkerParallel/./DWARFLinkerParallelTests/2/3

The same as yours, yeah.
The DebugInfo/Generic/discriminated-union.ll is due to an LLVM bug, but the rest of the failures only happen when linking with mold.

[rui314]

I believe GCC 15 miscompiles llvm/unittests/ADT/ConcurrentHashtableTest.cpp. Here is disaseembly of ConcurrentHashtableTest.cpp.o.

0000000000000000 <_ZZN12_GLOBAL__N_154ConcurrentHashTableTest_AddStringMultiplueEntries_Test8TestBodyEvENKUlvE_clEv>:
   0:   9d e3 bd e0     save  %sp, -544, %sp
   4:   c0 77 a6 f7     clrx  [ %fp + 0x6f7 ]
   8:   05 00 00 00     sethi  %hi(0), %g2
                        8: R_SPARC_GOTDATA_OP_HIX22     _ZTHN4llvm8parallel11threadIndexE
   c:   84 18 a0 00     xor  %g2, 0, %g2
                        c: R_SPARC_GOTDATA_OP_LOX10     _ZTHN4llvm8parallel11threadIndexE
  10:   2f 00 00 00     sethi  %hi(0), %l7
                        10: R_SPARC_PC22        _GLOBAL_OFFSET_TABLE_-0x4
  14:   40 00 00 00     call  14 <_ZZN12_GLOBAL__N_154ConcurrentHashTableTest_AddStringMultiplueEntries_Test8TestBodyEvENKUlvE_clEv+0x14>
                        14: R_SPARC_WPLT30      __sparc_get_pc_thunk.l7
  18:   ae 05 e0 00     add  %l7, 0, %l7        ! 0 <_ZZN12_GLOBAL__N_154ConcurrentHashTableTest_AddStringMultiplueEntries_Test8TestBodyEvENKUlvE_clEv>
                        18: R_SPARC_PC10        _GLOBAL_OFFSET_TABLE_+0x4
  1c:   23 00 00 00     sethi  %hi(0), %l1
                        1c: R_SPARC_GOTDATA_OP_HIX22    _ZN4llvm8parallel8strategyE
  20:   2b 00 00 00     sethi  %hi(0), %l5
                        20: R_SPARC_TLS_GD_HI22 _ZN4llvm8parallel11threadIndexE
  24:   a2 1c 60 00     xor  %l1, 0, %l1
                        24: R_SPARC_GOTDATA_OP_LOX10    _ZN4llvm8parallel8strategyE
  28:   aa 05 60 00     add  %l5, 0, %l5
                        28: R_SPARC_TLS_GD_LO10 _ZN4llvm8parallel11threadIndexE
  2c:   03 00 00 00     sethi  %hi(0), %g1
                        2c: R_SPARC_GOTDATA_OP_HIX22    _ZTVN4llvm7support6detail23provider_format_adapterIRmEE
  30:   33 00 00 00     sethi  %hi(0), %i1
                        30: R_SPARC_GOTDATA_OP_HIX22    _ZTVN4llvm18raw_string_ostreamE
  34:   82 18 60 00     xor  %g1, 0, %g1
                        34: R_SPARC_GOTDATA_OP_LOX10    _ZTVN4llvm7support6detail23provider_format_adapterIRmEE
  38:   b2 1e 60 00     xor  %i1, 0, %i1
                        38: R_SPARC_GOTDATA_OP_LOX10    _ZTVN4llvm18raw_string_ostreamE
  3c:   c2 5d c0 01     ldx  [ %l7 + %g1 ], %g1
                        3c: R_SPARC_GOTDATA_OP  _ZTVN4llvm7support6detail23provider_format_adapterIRmEE
  40:   f2 5d c0 19     ldx  [ %l7 + %i1 ], %i1
                        40: R_SPARC_GOTDATA_OP  _ZTVN4llvm18raw_string_ostreamE
  44:   c4 77 a6 c7     stx  %g2, [ %fp + 0x6c7 ]
  48:   82 00 60 10     add  %g1, 0x10, %g1
  4c:   84 05 c0 15     add  %l7, %l5, %g2
                        4c: R_SPARC_TLS_GD_ADD  _ZN4llvm8parallel11threadIndexE
  50:   e2 5d c0 11     ldx  [ %l7 + %l1 ], %l1
                        50: R_SPARC_GOTDATA_OP  _ZN4llvm8parallel8strategyE
  54:   c4 77 a6 cf     stx  %g2, [ %fp + 0x6cf ]
  58:   c2 77 a6 d7     stx  %g1, [ %fp + 0x6d7 ]
  5c:   b2 06 60 10     add  %i1, 0x10, %i1
  60:   c2 04 40 00     ld  [ %l1 ], %g1
  64:   80 a0 60 01     cmp  %g1, 1
  68:   02 40 01 ba     be,pn   %icc, 750 <_ZZN12_GLOBAL__N_154ConcurrentHashTableTest_AddStringMultiplueEntries_Test8TestBodyEvENKUlvE_clEv+0x750>
  6c:   f8 5e 00 00     ldx  [ %i0 ], %i4
  70:   c2 5f a6 c7     ldx  [ %fp + 0x6c7 ], %g1
  74:   fa 5d c0 01     ldx  [ %l7 + %g1 ], %i5
                        74: R_SPARC_GOTDATA_OP  _ZTHN4llvm8parallel11threadIndexE
  78:   02 c7 40 04     brz,pn   %i5, 88 <_ZZN12_GLOBAL__N_154ConcurrentHashTableTest_AddStringMultiplueEntries_Test8TestBodyEvENKUlvE_clEv+0x88>
  7c:   01 00 00 00     nop
  80:   40 00 00 00     call  80 <_ZZN12_GLOBAL__N_154ConcurrentHashTableTest_AddStringMultiplueEntries_Test8TestBodyEvENKUlvE_clEv+0x80>
                        80: R_SPARC_WPLT30      _ZTHN4llvm8parallel11threadIndexE
  84:   01 00 00 00     nop
  88:   40 00 00 00     call  88 <_ZZN12_GLOBAL__N_154ConcurrentHashTableTest_AddStringMultiplueEntries_Test8TestBodyEvENKUlvE_clEv+0x88>
                        88: R_SPARC_TLS_GD_CALL _ZN4llvm8parallel11threadIndexE
  8c:   d0 5f a6 cf     ldx  [ %fp + 0x6cf ], %o0
  90:   c2 02 00 00     ld  [ %o0 ], %g1
  94:   80 a0 7f ff     cmp  %g1, -1

Look at the following lines:

   8:   05 00 00 00     sethi  %hi(0), %g2
                        8: R_SPARC_GOTDATA_OP_HIX22     _ZTHN4llvm8parallel11threadIndexE
   c:   84 18 a0 00     xor  %g2, 0, %g2
                        c: R_SPARC_GOTDATA_OP_LOX10     _ZTHN4llvm8parallel11threadIndexE

  74:   fa 5d c0 01     ldx  [ %l7 + %g1 ], %i5
                        74: R_SPARC_GOTDATA_OP  _ZTHN4llvm8parallel11threadIndexE

These relocations are always used as a triple, and the last GOTDATA_OP must refer to two registers: a register containing the address of the .got section, and another set by GOTDATA_OP_HIX22 and GOTDATA_OP_LOX10. However, in the above case, although %l7 refers to .got, %g1 refers to something else.

GNU ld happens to link this program without causing a segmentation fault because, for some reason, the linker doesn’t eliminate the GOT slot for _ZTHN4llvm8parallel11threadIndexE. In this case, that symbol is weak and undefined, so its value is 0. Therefore, it’s safe to eliminate its GOT slot. However, GNU ld seems to create a GOT slot for that symbol anyway, which happens to hide the miscompilation issue. I believe what mold does is correct — since the symbol’s value is known at link time, we can optimize away the GOT slot altogether, which is precisely why the GOTDATA_OP_* relocations were ratified for in the first place.

[koachan]

Huh, yeah, interesting. That does look like the wrong register being used, yes.

Also, if I understand it right then the R_SPARC_WPLT30 call at 0x80 shouldn't exist in the first place either?

80:   40 00 00 00     call  80 <_ZZN12_GLOBAL__N_154ConcurrentHashTableTest_AddStringMultiplueEntries_Test8TestBodyEvENKUlvE_clEv+0x80>
                        80: R_SPARC_WPLT30      _ZTHN4llvm8parallel11threadIndexE

Because if the slot for it is elminated then the call will end up pointing to nowhere?

If so then I'll try making a report to the GCC folks.

[rui314]

_ZTHN4llvm8parallel11threadIndexE is an undefined weak symbol, so the symbol is resolved to address 0 at link-time. A weak symbol is typically used in code like the following:

if (some_weak_func)
  some_weak_func();

What you are seeing with the WPLT30 relocation is the function call part, while the address materialized by the GOTDATA relocations corresponds to the condition in the if statement. Since the condition is false when the symbol is weak undefined, the call instruction referenced by the WPLT30 relocation is never executed at runtime. Because there’s no valid value to assign to the WPLT30 relocation, all linkers set some placeholder value. That value may differ between linkers, but since the call is never executed, it doesn’t matter.

[koachan]

Thanks so much for the explanation!
I'll close this issue and write a report for the GCC folks, then.

[rui314]

Please share a URL to your bug report here for tracking purposes.

By the way, are you working on SPARC64 as a hobby or for business? Just curious.

[koachan]

Please share a URL to your bug report here for tracking purposes.

Sure! The GCC bug report is at https://gcc.gnu.org/bugzilla/show_bug.cgi?id=122538.

By the way, are you working on SPARC64 as a hobby or for business? Just curious.

Ah, this is just a hobby for me.

[koachan]

Look at the following lines:

   8:   05 00 00 00     sethi  %hi(0), %g2
                        8: R_SPARC_GOTDATA_OP_HIX22     _ZTHN4llvm8parallel11threadIndexE
   c:   84 18 a0 00     xor  %g2, 0, %g2
                        c: R_SPARC_GOTDATA_OP_LOX10     _ZTHN4llvm8parallel11threadIndexE

  74:   fa 5d c0 01     ldx  [ %l7 + %g1 ], %i5
                        74: R_SPARC_GOTDATA_OP  _ZTHN4llvm8parallel11threadIndexE

Sorry, I just noticed it today, but it seems like between the xor and the ldx there's a spill happening, so the full sequence goes like this:

   8:   05 00 00 00     sethi  %hi(0), %g2
                        8: R_SPARC_GOTDATA_OP_HIX22     _ZTHN4llvm8parallel11threadIndexE
   c:   84 18 a0 00     xor  %g2, 0, %g2
                        c: R_SPARC_GOTDATA_OP_LOX10     _ZTHN4llvm8parallel11threadIndexE
  44:   c4 77 a6 c7     stx  %g2, [ %fp + 0x6c7 ]
  70:   c2 5f a6 c7     ldx  [ %fp + 0x6c7 ], %g1
  74:   fa 5d c0 01     ldx  [ %l7 + %g1 ], %i5
                        74: R_SPARC_GOTDATA_OP  _ZTHN4llvm8parallel11threadIndexE
  78:   02 c7 40 04     brz,pn   %i5, 88 <_ZZN12_GLOBAL__N_154ConcurrentHashTableTest_AddStringMultiplueEntries_Test8TestBodyEvENKUlvE_clEv+0x88>
  7c:   01 00 00 00     nop

So as far as the values are concerned, the code should be correct, I think?
But as you can see in my first post, the ldx that feeds the brz gets turned into a nop.

So I think this is an issue on the mold side, not GCC?

[koachan]

Just checked over at the GCC report, and they seem to also think so:
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=122538#c1