Missing Symbols in librealm.a

[lightsighter]

If I ask cmake to build librealm.a then it fails to statically link in the symbols for cpptrace which leads to linker errors. librealm.a should be self-contained and include the symbols for all its dependences.

nm librealm.a | grep cpptrace
0000000000000fbc T __ZN8cpptrace10stacktraceD1Ev
0000000000001f60 T __ZN8cpptrace10stacktraceD2Ev
0000000000002404 T __ZN8cpptrace16stacktrace_frameD1Ev
0000000000002430 T __ZN8cpptrace16stacktrace_frameD2Ev
                 U __ZN8cpptrace18generate_raw_traceEmm
0000000000000e58 T __ZN8cpptrace9raw_traceC1Ev
0000000000001ed4 T __ZN8cpptrace9raw_traceC2Ev
0000000000000bd8 T __ZN8cpptrace9raw_traceD1Ev
0000000000001ea8 T __ZN8cpptrace9raw_traceD2Ev
                 U __ZNK8cpptrace16stacktrace_frame9to_stringEb
                 U __ZNK8cpptrace9raw_trace7resolveEv

Repo on MacOS with Apple clang version 15.0.0 (clang-1500.0.40.1) and

cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_STANDARD=17

Trying to link anything against librealm.a will lead to:

ld: Undefined symbols:
  cpptrace::generate_raw_trace(unsigned long, unsigned long), referenced from:
      Realm::Backtrace::capture_backtrace(int, int) in librealm.a[14](faults.cc.o)
  cpptrace::stacktrace_frame::to_string(bool) const, referenced from:
      Realm::Backtrace::print_symbols(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>>&) const in librealm.a[14](faults.cc.o)
  cpptrace::raw_trace::resolve() const, referenced from:
      Realm::Backtrace::print_symbols(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>>&) const in librealm.a[14](faults.cc.o)

[muraj]

In order to do that, you have to un-archive libcpptrace.a, grab all the object files and add them to librealm.a. That's not really reasonable to do and definitely not typical of most libraries. If you want a self-contained library, then you should build a .so with -DBUILD_SHARED_LIBS=ON cmake variable. For static libraries, you need to add all the dependencies explicitly in your linker line. You don't have to do that for private header-only libraries.

[elliottslaughter]

un-archive libcpptrace.a, grab all the object files and add them to librealm.a

For what it's worth, this is exactly what Terra does, precisely because packaging the .a files isn't really reasonable either.

[muraj]

I really don't think it's reasonable to do that with all our dependencies that are built statically. It's also a bit of a pain in the butt to do in CMake and is definitely not portable to all our supported platforms.

[lightsighter]

@muraj This is exactly the kind of dependence management thing you told me that cmake would automatically do for us when you were trying to convince me that we should have a cmake-only build for Realm so I think you need to make sure that it does exactly that. At a minimum you need to make cmake put the cpptrace.a library in the lib directory in an install because right now the install only puts a librealm.a in there that is clearly missing symbols which is unusable. The cmake build needs to work stand-alone in addition to being called by cmake from other projects.

[elliottslaughter]

If you don't want to bundle the object files, I agree that it's the responsibility of Realm's CMake build to install anything that it builds internally, and include those in the link line provided to pkg-config. Whether it's a .so or .a actually doesn't matter, what matters is that Realm internally downloaded and built some dependency. Otherwise it's literally impossible to use the resulting installation.

[muraj]

@lightsighter Yes, and I didn't lie to you. We already have someone working to install the static library dependencies (like libcpptrace.a) for librealm.a and as @elliottslaughter says, the pkg-config file will be updated to add that as well. I just couldn't wait until that work was completed. Using -DBUILD_SHARED_LIBS=ON and creating a librealm.so will handle the dependencies for you, that part is completed. Alternatively, if you need static libraries with the current build system, you can disable those dependencies (e.g. with -DREALM_ENABLE_CPPTRACE=OFF), build and install cpptrace in your system so cmake will find it as an external dependency, or use something like cmake's add_subdirectory / or CPM.cmake to build realm within a cmake project producing your executable and it'll make sure all the right linker flags are added (this is what I do in legion's cmake).

I am leaving this issue open to handle the static library packaging part, we do plan to implement this for static library builds, but until then, the alternatives are listed above. My earlier comments were directed specifically at bundling librealm.a with all the dependencies inside of it -- I do not believe that is a reasonable thing to do. Installing downloaded dependencies for static library builds along with librealm.a seems more reasonable.

[lightsighter]

Whether it's a .so or .a actually doesn't matter, what matters is that Realm internally downloaded and built some dependency. Otherwise it's literally impossible to use the resulting installation.

Yes, agreed, changing the status to a bug. librealm.a in the current form is unusable (and so is librealm.dylib, see below).

Yes, and I didn't lie to you.

Well you did. You said cmake would automatically handle all dependency management for the build for us and clearly it doesn't if you're saying now that extra work has to be done to make it happen. I was very specific when I asked you about this before we started. The exact phrasing that I used was: "will our cmake build handle all aspects of dependence management?" and you responded that it would without carving out any exceptions for circumstances like this. I'm just holding you to that contract that we established before we started.

I just couldn't wait until that work was completed

I think we need to slow down or stop the current migration if it's going to produce unusable builds like this. Everything should be tested and working before we move things over. We can't take any short cuts because it breaks people's workflows. Realm in it's current form is unusable by me and that blocks any work that I am doing. I'm sure I'm not the only one.

Using -DBUILD_SHARED_LIBS=ON and creating a librealm.so will handle the dependencies for you, that part is completed.

Tried that. No libcpptrace.dylib was installed in the lib directory so everything is still broken and fails to link.

Alternatively, if you need static libraries with the current build system, you can disable those dependencies (e.g. with -DREALM_ENABLE_CPPTRACE=OFF

If we can do that why do we need the dependence on the cpptrace library in the first place?

[muraj]

I was very specific when I asked you about this before we started. The exact phrasing that I used was: "will our cmake build handle all aspects of dependence management?" and you responded that it would without carving out any exceptions for circumstances like this. I'm just holding you to that contract that we established before we started.

And I hold myself to that same contract. As I stated, we do plan to implement that support in all cases, and I mentioned the lack of this support for static libraries many times previously and no one raised any red flags at the time. I always followed it up with "we will work on it". And we are, we had someone working on it and implementing a solution for static libraries, but it never got merged in. I'll make it my first priority once the github repo is working and people are unblocked from using it so we can enable parallel work to continue.

Tried that. No libcpptrace.dylib was installed in the lib directory so everything is still broken and fails to link.

Yes, there is no libcpptrace.dylib because we still build cpptrace as a static library regardless, which will get packaged into the librealm.dylib -- that's just how dynamic libraries work in general: they, like executables, compile in all static libraries making them self-contained. So it should build and link just fine without any knowledge of libcpptrace.dylib or any of our other downloaded dependencies (or external shared dependencies either, assuming they are available at runtime). And, in practice, it does work, as you can see in your own CI on gitlab:

• https://gitlab.com/StanfordLegion/legion/-/jobs/10707208750
• https://gitlab.com/StanfordLegion/legion/-/jobs/10707208759
• https://gitlab.com/StanfordLegion/legion/-/jobs/10707208770

If we can do that why do we need the dependence on the cpptrace library in the first place?

cpptrace enables a feature. When I asked about automatically enabling features, you specifically stated that you wanted to default enable features and have us automatically download the dependencies needed for them if and when possible. cpptrace is one of those features. You can specifically disable the dependency, but at the cost of losing the the feature it enables -- pretty backtraces. That's why we need the dependency in the first place.

[elliottslaughter]

I've been discussing this with @muraj. It turns out that cpptrace is actually disabled by default in the old Legion build system:

https://gitlab.com/StanfordLegion/legion/-/blob/df1ba424131e70dcbf511b7628828293bd78556e/CMakeLists.txt#L746 (using grandrefactor mainly because I know it hasn't merged master yet)

Therefore it seems the root cause of this issue is the combination of:

1. Enabling automatic downloads for dependencies
2. The difficulty of installing those dependencies (and all their transitive dependencies, etc.), which would be required for static linking (edit: see footnote ¹)

Given that this is a new feature and not something previously supported by the Legion build system, I suggest we put automatic downloads behind a flag. My impression is that the static build will "just work" as long as automatic downloads are not enabled, and we can work towards re-enabling downloads as time permits.

(Optionally we could enable automatic downloads for shared builds, but it seems a bit screwy to enable it based on what kind of build you are doing.)

Footnotes

1. I previously wrote here that it would break in a shared build too. @muraj corrected me even in a shared Realm build, dependencies are static. So the static libcpptrace.a gets linked into the dynamic librealm.so. This means that, indeed, shared builds work while static builds do not. However I would note an additional caveat that if the user ever wants to separately link any of the transitive dependencies (e.g., zstd), that would break (due to having multiple versions linked). It works in this hacky way because Realm is the exclusive user of these dependencies. ↩

[lightsighter]

I agree that we should set the defaults back to whatever they were before. If that fixes things all the better. The defaults shouldn't be changed until they are robust in all circumstances.