LTO build of rpcs3 fails on Windows

[oltolm]

Hello I use clang 19.1.7 from MSYS2 clang64. Recently https://github.com/RPCS3/rpcs3 has enabled LTO. Now the release build is failing with

[build] ld.lld: error: undefined symbol: thread-local initialization routine for perf_stat<19226358023673171ull>::g_tls_perf_stat
[build] >>> referenced by C:/src/rpcs3/rpcs3/Emu/CPU/CPUThread.cpp
[build] >>>               librpcs3_emu.a(CPUThread.cpp.obj)
[build] >>> referenced by C:/src/rpcs3/rpcs3/Emu/Memory/vm.cpp
[build] >>>               librpcs3_emu.a(vm.cpp.obj)
[build] 

I reduced it

CMakeListsts.txt

cmake_minimum_required(VERSION 3.21)
project(lto VERSION 0.0.1 LANGUAGES C CXX)

add_executable(myexe src/myexe.cpp)
target_compile_features(myexe PRIVATE cxx_std_20)
target_link_options(myexe PRIVATE -municode)

set_property(TARGET myexe PROPERTY INTERPROCEDURAL_OPTIMIZATION TRUE)

myexe.cpp

#include "perf_meter.hpp"

int wmain(int argc, wchar_t** argv)
{
    perf_meter<"DMA"_u32> perf_;
    return 0;
}

perf_meter.hpp

template <auto ShortName> class perf_stat final {
    static inline thread_local struct perf_stat_local {
        // Local non-atomic values for increments
        unsigned m_log[66]{};

        perf_stat_local() noexcept {}

        ~perf_stat_local() {}

    } g_tls_perf_stat;

  public:
    static void push(unsigned start_time) noexcept { (void)g_tls_perf_stat; }
};

template <auto ShortName, auto... SubEvents> class perf_meter {
  public:
    ~perf_meter() { perf_stat<ShortName>::push(0); }
};

constexpr unsigned operator""_u32(const char* s, unsigned long long /*length*/)
{
    return 0;
}

[llvmbot]

@llvm/issue-subscribers-lld-coff

Author: oltolm (oltolm)

Hello I use clang 19.1.7 from MSYS2 clang64. Recently https://github.com/RPCS3/rpcs3 has enabled LTO. Now the release build is failing with

[build] ld.lld: error: undefined symbol: thread-local initialization routine for perf_stat<19226358023673171ull>::g_tls_perf_stat
[build] >>> referenced by C:/src/rpcs3/rpcs3/Emu/CPU/CPUThread.cpp
[build] >>>               librpcs3_emu.a(CPUThread.cpp.obj)
[build] >>> referenced by C:/src/rpcs3/rpcs3/Emu/Memory/vm.cpp
[build] >>>               librpcs3_emu.a(vm.cpp.obj)
[build] 

I reduced it

CMakeListsts.txt

cmake_minimum_required(VERSION 3.21)
project(lto VERSION 0.0.1 LANGUAGES C CXX)

add_executable(myexe src/myexe.cpp)
target_compile_features(myexe PRIVATE cxx_std_20)
target_link_options(myexe PRIVATE -municode)

set_property(TARGET myexe PROPERTY INTERPROCEDURAL_OPTIMIZATION TRUE)

myexe.cpp

#include "perf_meter.hpp"

int wmain(int argc, wchar_t** argv)
{
    perf_meter<"DMA"_u32> perf_;
    return 0;
}

perf_meter.hpp

template <auto ShortName> class perf_stat final {
    static inline thread_local struct perf_stat_local {
        // Local non-atomic values for increments
        unsigned m_log[66]{};

        perf_stat_local() noexcept {}

        ~perf_stat_local() {}

    } g_tls_perf_stat;

  public:
    static void push(unsigned start_time) noexcept { (void)g_tls_perf_stat; }
};

template <auto ShortName, auto... SubEvents> class perf_meter {
  public:
    ~perf_meter() { perf_stat<ShortName>::push(0); }
};

constexpr unsigned operator""_u32(const char* s, unsigned long long /*length*/)
{
    return 0;
}

[mstorsjo]

For reference; this is a crosspost of an issue originally reported at mstorsjo/llvm-mingw#483, where I did some initial analysis of the issue. Reposting the analysis here for further discussion...

The issue can be reproduced with a quite minimal test input like this:

struct tlsdata {
    int val;
    ~tlsdata() {}
};
inline thread_local struct tlsdata tlsvar;

int main(int argc, char** argv) {
    (void)tlsvar;
    return 0;
}

It can be compiled with one single command without needing cmake, just x86_64-w64-mingw32-clang++ myexe.cpp -flto (with e.g. http://github.com/mstorsjo/llvm-mingw/).

The inline on the tlsvar definition is essential - without it, this works fine.

[mstorsjo]

When doing LTO linking, we load the LTO objects which essentially are LLVM IR. Based on the symbols in the IR, we populate the LLD symbol table (while we don't actually have COFF object files for the symbols yet) and do linking just like regular, then after we replace the fake symbols with the actual ones output from the LTO compilation. (This step is very tricky for cases when the compilation output doesn't contain the exact same symbols as the LLVM IR did, e.g. if the compiler adds references to symbols that weren't visible on the LLVM IR level.)

In this case, we end up here in the linker: https://github.com/llvm/llvm-project/blob/llvmorg-20.1.0/lld/COFF/InputFiles.cpp#L1338-L1343 We try to set up a weak alias towards the target symbol.

However in this case, we have the following situation in the IR from the compiled object (x86_64-w64-mingw32-clang++ myexe.cpp -flto -S -o -):

@_ZTH6tlsvar = linkonce_odr dso_local alias void (), ptr @__tls_init
define internal void @__tls_init() #4 {
...
}

We have _ZTH6tlsvar as a weak alias, pointing at __tls_init. However the unusual thing is that __tls_init isn't an external symbol. So when we try to look up __tls_init in the symbol table, we find nothing. And thus, as _ZTH6tlsvar is a weak alias, with the function it points at unresolvable, we also end up with having _ZTH6tlsvar essentially unresolved.

When such things are compiled to a regular object file before linking, the LLVM MC layer generates a regular extern symbol for the target of the weak alias, named e.g. .weak._ZTH6tlsvar.default.main in this case.

I'm not entirely sure what the right fix is here though. Perhaps the code in https://github.com/llvm/llvm-project/blob/llvmorg-20.1.0/lld/COFF/InputFiles.cpp#L1338-L1343 would need to look at the target symbol, and see that if it isn't extern, we'd need to treat the alias differently, somehow.