An option to force the usage of an external symbolizer?

[loskutov]

As far as I can tell from the code, currently libbacktrace-based symbolizer always has a priority over external ones:

// sanitizer_symbolizer_posix_libcdep.cpp
  if (SymbolizerTool *tool = LibbacktraceSymbolizer::get(allocator)) {
    VReport(2, "Using libbacktrace symbolizer.\n");
    list->push_back(tool);
    return;
  }

  if (SymbolizerTool *tool = ChooseExternalSymbolizer(allocator)) {
    list->push_back(tool);
  }

However, in some cases addr2line is able to symbolize the addresses libbacktrace is unable to. It would be useful to have an option to force addr2line usage even if libbacktrace is available.

[bengtj]

I had a session yesterday trying to get line numbers in backtraces from tsan builds using split-dwarf/dwo without success. Eventually I realized that it was silently ignoring our $TSAN_SYMBOLIZER_PATH and external_symbolizer_path in $TSAN_OPTIONS, after setting verbosity=100 I got:
==24470==Using libbacktrace symbolizer.

As far as I can tell there is no support for reading dwo files in libbacktrace so we really need to use llvm-symbolizer with $LLVM_SYMBOLIZER_OPTS and --debug-file-directory= to tell it where to find those dwo files relative to the compilation dir.

I'm of course hopeful that it is already possible to make it respect external_symbolizer_path somehow. Note that we are using sanitizers through gcc and I've yet to check the source code but then I found this issue indicating "here be dragons".

[bengtj]

We've currently worked around this issue with a script that will post process sanitizer backlogs and replace with actual source code location using ldd/addr2line. Note that it is possible to use addr2line even though I think it does not support dwo/split-dwarf so libbacktrace should probably be able to print something better than .

However, this ticket needs some attention to fix so that it is possible to override using extrernal_symbolizer again even though libbacktrace is in use. Still hoping for a runtime solution that I failed to find but we're also contemplating recompling gcc/clang and setting SANITIZER_LIBBACKTRACE to 0.

Also note that the documentation seems out-of-sync:
https://clang.llvm.org/docs/AddressSanitizer.html#symbolizing-the-reports

[jiridanek]

I've hit the same issue. On Fedora 34, when symbolizing the Python shared library, only addr2line is able to give me the function names I need:

# LD_PRELOAD=/usr/lib64/libasan.so.6.0.0 python3 -c 'print(42)'
42

=================================================================
==325==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 112496 byte(s) in 97 object(s) allocated from:
    #0 0x7f689b50bd1f in __interceptor_malloc (/usr/lib64/libasan.so.6.0.0+0xaed1f)
    #1 0x7f689b1febae in PyObject_Malloc (/lib64/libpython3.9.so.1.0+0xf8bae)

Direct leak of 1 byte(s) in 1 object(s) allocated from:
    #0 0x7f689b50bd1f in __interceptor_malloc (/usr/lib64/libasan.so.6.0.0+0xaed1f)
    #1 0x7f689b204d1a in PyMem_Malloc (/lib64/libpython3.9.so.1.0+0xfed1a)

Indirect leak of 25922 byte(s) in 26 object(s) allocated from:
    #0 0x7f689b50bd1f in __interceptor_malloc (/usr/lib64/libasan.so.6.0.0+0xaed1f)
    #1 0x7f689b1febae in PyObject_Malloc (/lib64/libpython3.9.so.1.0+0xf8bae)

Indirect leak of 536 byte(s) in 1 object(s) allocated from:
    #0 0x7f689b50bd1f in __interceptor_malloc (/usr/lib64/libasan.so.6.0.0+0xaed1f)
    #1 0x7f689b286495  (/lib64/libpython3.9.so.1.0+0x180495)

SUMMARY: AddressSanitizer: 138955 byte(s) leaked in 125 allocation(s).

Now, if I want to symbolize /lib64/libpython3.9.so.1.0+0x180495, I can do that with addr2line

# addr2line --exe /lib64/libpython3.9.so.1.0 -f 0x103a5b
list_resize.part.0
/usr/src/debug/python3.9-3.9.5-1.fc34.x86_64/Objects/listobject.c:72

Using llvm-symbolizer, I cannot get at the function name, however. This is bad because I want to use the function name in my suppressions

# llvm-symbolizer --obj /lib64/libpython3.9.so.1.0 --functions 0x103a5b
??
/usr/src/debug/python3.9-3.9.5-1.fc34.x86_64/Objects/listobject.c:72:26

@bengtj thanks for pointing out verbosity=100, that helped with debugging!

[OlivierNicole]

I second the report that it seems impossible to use an external symbolizer. When libbacktrace does not work well on your executables for some reason (e.g. no usual C-like stacks), it makes it hard to give the user useful backtraces.