libcxx msan tests fail with llvm unwinder

[vitalybuka]

We work around that as:
https://github.com/llvm/llvm-zorg/blob/95b6239a1e159342caf514cd7b3ab0eaf7996251/zorg/buildbot/builders/sanitizers/buildbot_functions.sh#L303

Would be nice to switch to llvm unwinder.

@petrhosek @ldionne I know you run check-cxx check-cxxabi with sanitizers. Which unwinder do you use?

[DanielKristofKiss]

@vitalybuka is there some pointer for the fail? Thanks!

[vitalybuka]

Msan fails all over the build. Actually, I am not sure if other sanitizers have problem, asan seems fine.

To reproduce https://github.com/google/sanitizers/wiki/SanitizerBotReproduceBuild
and use zorg/zorg/buildbot/builders/sanitizers/buildbot_bootstrap_msan.sh
Or copy cmake/ninja commands from https://lab.llvm.org/buildbot/#/builders/sanitizer-x86_64-linux-bootstrap-msan

If we instrument libunwind, then data which gets into unwinder is not correctly annotated by msan. Probably bad exception support by msan.

Remove: -DLIBCXXABI_USE_LLVM_UNWINDER=OFF

********************
Failed Tests (46):
  llvm-libc++abi-shared.cfg.in :: backtrace_test.pass.cpp
  llvm-libc++abi-shared.cfg.in :: catch_array_01.pass.cpp
  llvm-libc++abi-shared.cfg.in :: catch_array_02.pass.cpp
  llvm-libc++abi-shared.cfg.in :: catch_class_01.pass.cpp
  llvm-libc++abi-shared.cfg.in :: catch_class_02.pass.cpp
  llvm-libc++abi-shared.cfg.in :: catch_class_03.pass.cpp
  llvm-libc++abi-shared.cfg.in :: catch_class_04.pass.cpp
  llvm-libc++abi-shared.cfg.in :: catch_const_pointer_nullptr.pass.cpp
  llvm-libc++abi-shared.cfg.in :: catch_function_01.pass.cpp
  llvm-libc++abi-shared.cfg.in :: catch_function_02.pass.cpp
  llvm-libc++abi-shared.cfg.in :: catch_function_03.pass.cpp
  llvm-libc++abi-shared.cfg.in :: catch_in_noexcept.pass.cpp
  llvm-libc++abi-shared.cfg.in :: catch_member_data_pointer_01.pass.cpp
  llvm-libc++abi-shared.cfg.in :: catch_member_function_pointer_01.pass.cpp
  llvm-libc++abi-shared.cfg.in :: catch_member_function_pointer_02.pass.cpp
  llvm-libc++abi-shared.cfg.in :: catch_member_pointer_nullptr.pass.cpp
  llvm-libc++abi-shared.cfg.in :: catch_multi_level_pointer.pass.cpp
  llvm-libc++abi-shared.cfg.in :: catch_null_pointer_to_object_pr64953.pass.cpp
  llvm-libc++abi-shared.cfg.in :: catch_pointer_nullptr.pass.cpp
  llvm-libc++abi-shared.cfg.in :: catch_pointer_reference.pass.cpp
  llvm-libc++abi-shared.cfg.in :: catch_ptr.pass.cpp
  llvm-libc++abi-shared.cfg.in :: catch_ptr_02.pass.cpp
  llvm-libc++abi-shared.cfg.in :: catch_reference_nullptr.pass.cpp
  llvm-libc++abi-shared.cfg.in :: cxa_bad_cast.pass.cpp
  llvm-libc++abi-shared.cfg.in :: cxa_bad_typeid.pass.cpp
  llvm-libc++abi-shared.cfg.in :: cxa_call_terminate.pass.cpp
  llvm-libc++abi-shared.cfg.in :: cxa_vec_new_overflow_PR41395.pass.cpp
  llvm-libc++abi-shared.cfg.in :: exception_object_alignment.2.pass.cpp
  llvm-libc++abi-shared.cfg.in :: forced_unwind1.pass.cpp
  llvm-libc++abi-shared.cfg.in :: forced_unwind2.pass.cpp
  llvm-libc++abi-shared.cfg.in :: forced_unwind3.pass.cpp
  llvm-libc++abi-shared.cfg.in :: guard_threaded_test.pass.cpp
  llvm-libc++abi-shared.cfg.in :: incomplete_type.sh.cpp
  llvm-libc++abi-shared.cfg.in :: inherited_exception.pass.cpp
  llvm-libc++abi-shared.cfg.in :: native/x86_64/lpstart-zero.pass.sh.s
  llvm-libc++abi-shared.cfg.in :: test_aux_runtime.pass.cpp
  llvm-libc++abi-shared.cfg.in :: test_aux_runtime_op_array_new.pass.cpp
  llvm-libc++abi-shared.cfg.in :: test_exception_address_alignment.pass.cpp
  llvm-libc++abi-shared.cfg.in :: test_guard.pass.cpp
  llvm-libc++abi-shared.cfg.in :: test_vector1.pass.cpp
  llvm-libc++abi-shared.cfg.in :: test_vector2.pass.cpp
  llvm-libc++abi-shared.cfg.in :: test_vector3.pass.cpp
  llvm-libc++abi-shared.cfg.in :: uncaught_exception.pass.cpp
  llvm-libc++abi-shared.cfg.in :: uncaught_exceptions.pass.cpp
  llvm-libc++abi-shared.cfg.in :: unwind_01.pass.cpp
  llvm-libc++abi-shared.cfg.in :: unwind_06.pass.cpp

/usr/bin/python3 /tmp/bot/llvm-project/libcxx/utils/run.py --execdir /tmp/bot/libcxx_build_msan/libcxxabi/test/Output/catch_multi_level_pointer.pass.cpp.dir --  /tmp/bot/libcxx_build_msan/libcxxabi/test/Output/catch_multi_level_pointer.pass.cpp.dir/t.tmp.exe
# executed command: /usr/bin/python3 /tmp/bot/llvm-project/libcxx/utils/run.py --execdir /tmp/bot/libcxx_build_msan/libcxxabi/test/Output/catch_multi_level_pointer.pass.cpp.dir -- /tmp/bot/libcxx_build_msan/libcxxabi/test/Output/catch_multi_level_pointer.pass.cpp.dir/t.tmp.exe
# .---command stderr------------
# | MemorySanitizer:DEADLYSIGNAL
# | ==3144582==ERROR: MemorySanitizer: stack-overflow on address 0x7fff69989f30 (pc 0x563a9967a753 bp 0x7fff6998a760 sp 0x7fff69989f30 T3144582)
# | MemorySanitizer:DEADLYSIGNAL
# | MemorySanitizer: nested bug in the same thread, aborting.
# `-----------------------------
# error: command failed with exit status: 1

# nested bug because it uses unwinder on report from unwinder, here is work around:

MSAN_OPTIONS=fast_unwind_on_fatal=1 /tmp/bot/libcxx_build_msan/libcxxabi/test/Output/catch_multi_level_pointer.pass.cpp.dir/t.tmp.exe
==3146975==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x7f299486f450 in libunwind::UnwindCursor<libunwind::LocalAddressSpace, libunwind::Registers_x86_64>::getReg(int) /tmp/bot/llvm-project/libunwind/src/UnwindCursor.hpp:1353:3
    #1 0x7f299486f96e in libunwind::UnwindCursor<libunwind::LocalAddressSpace, libunwind::Registers_x86_64>::setInfoBasedOnIPRegister(bool) /tmp/bot/llvm-project/libunwind/src/UnwindCursor.hpp:2561:41
    #2 0x7f299486c826 in unw_init_local /tmp/bot/llvm-project/libunwind/src/libunwind.cpp:91:7
    #3 0x7f2994877f23 in unwind_phase1 /tmp/bot/llvm-project/libunwind/src/UnwindLevel1.c:92:3
    #4 0x7f2994877f23 in _Unwind_RaiseException /tmp/bot/llvm-project/libunwind/src/UnwindLevel1.c:459:32
    #5 0x7f2994904244 in __cxa_throw /tmp/bot/llvm-project/libcxxabi/src/cxa_exception.cpp:295:5
    #6 0x55fffdfec240 in void catch_pointer_test<int*, int*>() /tmp/bot/llvm-project/libcxxabi/test/catch_multi_level_pointer.pass.cpp:61:5
    #7 0x55fffdfebc80 in void generate_tests_imp<int, int, 3, true>::run_catch_tests<int*>() /tmp/bot/llvm-project/libcxxabi/test/catch_multi_level_pointer.pass.cpp:115:7
    #8 0x55fffdfeb62f in generate_tests_imp<int, int, 3, true>::operator()() /tmp/bot/llvm-project/libcxxabi/test/catch_multi_level_pointer.pass.cpp:102:7
    #9 0x55fffdfeb482 in main /tmp/bot/llvm-project/libcxxabi/test/catch_multi_level_pointer.pass.cpp:139:3
    #10 0x7f2994563c89 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16

SUMMARY: MemorySanitizer: use-of-uninitialized-value /tmp/bot/llvm-project/libunwind/src/UnwindCursor.hpp:1353:3 in libunwind::UnwindCursor<libunwind::LocalAddressSpace, libunwind::Registers_x86_64>::getReg(int)

I tried to annotate this one, but then other places break

If we disable instrumentation of libunwind, then cxx tests fail on test side, because data from unwinder is not annotated.

So instead we can try to replace -DLIBCXXABI_USE_LLVM_UNWINDER=OFF with -DLIBUNWIND_ADDITIONAL_COMPILE_FLAGS=-fno-sanitize=memory we see just 2:

Failed Tests (2):
  llvm-libc++abi-shared.cfg.in :: cxa_call_terminate.pass.cpp
  llvm-libc++abi-shared.cfg.in :: test_demangle.pass.cpp

/tmp/bot/libcxx_build_msan/libcxxabi/test/Output/cxa_call_terminate.pass.cpp.dir/t.tmp.exe
==3251834==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x5604c7b8f791 in std::operator==[abi:ne200000](std::exception_ptr const&, std::exception_ptr const&) /tmp/bot/libcxx_build_msan/libcxxabi/test-suite-install/include/c++/v1/__exception/exception_ptr.h:82:5
    #1 0x5604c7b8f51f in std::operator!=[abi:ne200000](std::exception_ptr const&, std::exception_ptr const&) /tmp/bot/libcxx_build_msan/libcxxabi/test-suite-install/include/c++/v1/__exception/exception_ptr.h:86:18
    #2 0x5604c7b8f2b7 in terminate_handler() /tmp/bot/llvm-project/libcxxabi/test/cxa_call_terminate.pass.cpp:28:3
    #3 0x7f0605928125 in std::__terminate(void (*)()) /tmp/bot/llvm-project/libcxxabi/src/cxa_handlers.cpp:59:9
    #4 0x7f06059280c7 in std::terminate() /tmp/bot/llvm-project/libcxxabi/src/cxa_handlers.cpp:88:17
    #5 0x5604c7b8f42b in __clang_call_terminate cxa_call_terminate.pass.cpp
    #6 0x5604c7b8f236 in func() /tmp/bot/llvm-project/libcxxabi/test/cxa_call_terminate.pass.cpp:23:3
    #7 0x5604c7b8f3da in main /tmp/bot/llvm-project/libcxxabi/test/cxa_call_terminate.pass.cpp:34:3
    #8 0x7f06055d8c89 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
    #9 0x7f06055d8d44 in __libc_start_main csu/../csu/libc-start.c:360:3
    #10 0x5604c7af3620 in _start (/tmp/bot/libcxx_build_msan/libcxxabi/test/Output/cxa_call_terminate.pass.cpp.dir/t.tmp.exe+0x6e620)

I guess we need to make a decision what is the target we want to achieve:

1. Instrument unwinder. Possible but tricky, msan use it for reports.
2. Do not instrument unwinder, than we need to __msan_unpoison or similar for unwinder outputs, so instrumented unwinder users don't break.