Many libcxx/test/benchmarks tests FAIL

[rorth]

When running a LLVM 20.1.0 rc1 reltest on Linux/x86_64, many (49 out of 75) new libcxx/test/benchmarks tests FAIL. It seems the failure mode is always the same:

# .---command stderr------------
# | 2025-02-02T09:25:53+01:00
# | Running /var/llvm/reltest/llvm-20.1.0/rc1/Phase3/Release/llvmCore-20.1.0-rc1.obj/runtimes/runtimes-bins/libcxx/test/benchmarks/Output/atomic_wait_1_waiter_1_notifier.bench.cpp.dir/t.tmp.exe
# | Run on (64 X 2350 MHz CPU s)
# | CPU Caches:
# |   L1 Data 32 KiB (x32)
# |   L1 Instruction 32 KiB (x32)
# |   L2 Unified 512 KiB (x32)
# |   L3 Unified 16384 KiB (x8)
# | Load Average: 174.90, 123.26, 82.05
# | ***WARNING*** CPU scaling is enabled, the benchmark real time measurements may be noisy and will incur extra overhead.
# | libc++abi: terminating due to uncaught exception of type std::runtime_error: failed to create thread
# `-----------------------------
# error: command failed with exit status: 250

The system is running Ubuntu 20.04 and has a single AMD EPYC 7452 32-Core cpu.

Unfortunately, the error message (from libcxx/test/benchmarks/atomic_wait_helper.h (HighPrioTask::HighPrioTask)) isn't particularly helpful: it should include the error returned from pthread_create.

When running one of the failing tests under strace, I find

139055 sched_get_priority_max(SCHED_FIFO) = 99
1139055 sched_get_priority_min(SCHED_FIFO) = 1
1139055 sched_get_priority_max(SCHED_FIFO) = 99
1139055 clone(child_stack=0x7f706411dfb0, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tid=[1139741], tls=0x7f706411e700, child_tidptr=0x7f706411e9d0) = 1139741
1139055 sched_setscheduler(1139741, SCHED_FIFO, [99] <unfinished ...>
1139741 set_robust_list(0x7f706411e9e0, 24 <unfinished ...>
1139055 <... sched_setscheduler resumed>) = -1 EPERM (Operation not permitted)

It seems this code was introduced only recently in

commit 699f19605579f25083152a9ad21e14c2751d5d66
Author: Hui <hui.xie1990@gmail.com>
Date:   Sat Jan 18 14:50:53 2025 +0000

    [libc++] remove yield from atomic::wait (#120012)

@huixie90 : this introduces an enormous amount of noise and should be fixed before the LLVM 20.1.0 release.

[ldionne]

@rorth You mention that 49 out of 75 benchmarks are failing, so I assume that non-atomic_wait related benchmarks are failing too? If so, this has nothing to do with 699f196, but might have to do with the fact that we're now running the benchmarks more often than we used to (i.e. you previously were simply not running the benchmarks as part of release testing).

Can you show me how you're running the release tests? Do you run something like llvm-lit <monorepo-root> and assume that to run the tests for Clang, LLVM, the runtimes, and everything else?

[rorth]

I'm just doing the regular release testing using lvm/utils/release/test-release.sh with -j<ncores> (64 for the system in question).

Here's the full list of failing tests from the 20.1.0 rc1 build:

  llvm-libc++-shared.cfg.in :: benchmarks/algorithms/algorithms.partition_point.bench.cpp
  llvm-libc++-shared.cfg.in :: benchmarks/algorithms/fill.bench.cpp
  llvm-libc++-shared.cfg.in :: benchmarks/algorithms/find.bench.cpp
  llvm-libc++-shared.cfg.in :: benchmarks/algorithms/lexicographical_compare.bench.cpp
  llvm-libc++-shared.cfg.in :: benchmarks/algorithms/lexicographical_compare_three_way.bench.cpp
  llvm-libc++-shared.cfg.in :: benchmarks/algorithms/lower_bound.bench.cpp
  llvm-libc++-shared.cfg.in :: benchmarks/algorithms/make_heap.bench.cpp
  llvm-libc++-shared.cfg.in :: benchmarks/algorithms/make_heap_then_sort_heap.bench.cpp
  llvm-libc++-shared.cfg.in :: benchmarks/algorithms/min.bench.cpp
  llvm-libc++-shared.cfg.in :: benchmarks/algorithms/min_max_element.bench.cpp
  llvm-libc++-shared.cfg.in :: benchmarks/algorithms/minmax.bench.cpp
  llvm-libc++-shared.cfg.in :: benchmarks/algorithms/mismatch.bench.cpp
  llvm-libc++-shared.cfg.in :: benchmarks/algorithms/pop_heap.bench.cpp
  llvm-libc++-shared.cfg.in :: benchmarks/algorithms/pstl.stable_sort.bench.cpp
  llvm-libc++-shared.cfg.in :: benchmarks/algorithms/push_heap.bench.cpp
  llvm-libc++-shared.cfg.in :: benchmarks/algorithms/ranges_make_heap.bench.cpp
  llvm-libc++-shared.cfg.in :: benchmarks/algorithms/ranges_make_heap_then_sort_heap.bench.cpp
  llvm-libc++-shared.cfg.in :: benchmarks/algorithms/ranges_pop_heap.bench.cpp
  llvm-libc++-shared.cfg.in :: benchmarks/algorithms/ranges_push_heap.bench.cpp
  llvm-libc++-shared.cfg.in :: benchmarks/algorithms/ranges_sort.bench.cpp
  llvm-libc++-shared.cfg.in :: benchmarks/algorithms/ranges_sort_heap.bench.cpp
  llvm-libc++-shared.cfg.in :: benchmarks/algorithms/ranges_stable_sort.bench.cpp
  llvm-libc++-shared.cfg.in :: benchmarks/algorithms/set_intersection.bench.cpp
  llvm-libc++-shared.cfg.in :: benchmarks/algorithms/sort.bench.cpp
  llvm-libc++-shared.cfg.in :: benchmarks/algorithms/sort_heap.bench.cpp
  llvm-libc++-shared.cfg.in :: benchmarks/algorithms/stable_sort.bench.cpp
  llvm-libc++-shared.cfg.in :: benchmarks/atomic_wait_N_waiter_N_notifier.bench.cpp
  llvm-libc++-shared.cfg.in :: benchmarks/atomic_wait_multi_waiter_1_notifier.bench.cpp
  llvm-libc++-shared.cfg.in :: benchmarks/atomic_wait_vs_mutex_lock.bench.cpp
  llvm-libc++-shared.cfg.in :: benchmarks/containers/deque.bench.cpp
  llvm-libc++-shared.cfg.in :: benchmarks/containers/deque_iterator.bench.cpp
  llvm-libc++-shared.cfg.in :: benchmarks/containers/map.bench.cpp
  llvm-libc++-shared.cfg.in :: benchmarks/containers/ordered_set.bench.cpp
  llvm-libc++-shared.cfg.in :: benchmarks/containers/string.bench.cpp
  llvm-libc++-shared.cfg.in :: benchmarks/containers/unordered_set_operations.bench.cpp
  llvm-libc++-shared.cfg.in :: benchmarks/containers/vector_operations.bench.cpp
  llvm-libc++-shared.cfg.in :: benchmarks/format/format_to.bench.cpp
  llvm-libc++-shared.cfg.in :: benchmarks/format/format_to_n.bench.cpp
  llvm-libc++-shared.cfg.in :: benchmarks/format/formatter_float.bench.cpp
  llvm-libc++-shared.cfg.in :: benchmarks/format/formatter_int.bench.cpp
  llvm-libc++-shared.cfg.in :: benchmarks/format/write_string_comparison.bench.cpp
  llvm-libc++-shared.cfg.in :: benchmarks/function.bench.cpp
  llvm-libc++-shared.cfg.in :: benchmarks/libcxxabi/dynamic_cast.bench.cpp
  llvm-libc++-shared.cfg.in :: benchmarks/stop_token.bench.cpp
  llvm-libc++-shared.cfg.in :: benchmarks/to_chars.bench.cpp
  llvm-libc++-shared.cfg.in :: benchmarks/variant_visit_1.bench.cpp
  llvm-libc++-shared.cfg.in :: benchmarks/variant_visit_2.bench.cpp
  llvm-libc++-shared.cfg.in :: benchmarks/variant_visit_3.bench.cpp

For comparison's sake, I've also tried a 2-stage main build with ninja check-all on the same system: the set of failures is way smaller

  llvm-libc++-shared.cfg.in :: benchmarks/atomic_wait_1_waiter_1_notifier.bench.cpp
  llvm-libc++-shared.cfg.in :: benchmarks/atomic_wait_N_waiter_N_notifier.bench.cpp
  llvm-libc++-shared.cfg.in :: benchmarks/atomic_wait_multi_waiter_1_notifier.bench.cpp

but the failure mode is the same. Doing so, I noticed that a handful of the libc++ benchmark tests took between 25 and 40 minutes to complete, which seems quite excessive to me.

[huixie90]

@rorth i think the tests like benchmarks/algorithms/minmax.bench.cpp definitely have different failure mode than benchmarks/atomic_wait_1_waiter_1_notifier.bench.cpp

I wonder why they fail in llvm CI but not in Libc++ CI

[ldionne]

The difference is subtle. When you run lit libcxx/test, it runs the benchmarks "for real". When you run ninja check-cxx, we run lit libcxx/test --param enable_benchmarks=dry_run which only compiles the benchmarks, but doesn't run them.

So the underlying issue here seems to be that ninja check-all (used by the release testing IIUC) runs lit libcxx/test instead of running ninja check-cxx. I'm still searching where exactly that happens.

Another viewpoint would be that lit libcxx/test should not run the benchmarks (or should only compile them), but I don't think that makes sense, since we definitely want e.g. lit libcxx/test/benchmarks/containers/vector.bench.cpp to run the benchmark for real.

CC @tstellar

[rorth]

One thing I just noticed: isn't enable_benchmarks supposed to be substituted at the top of libcxx/test/configs/cmake-bridge.cfg.in in @SERIALIZED_LIT_PARAMS@? However, when I look at the top of the generated file, there's nothing: in my case it starts with

## Autogenerated by libcxx configuration.
# Do not edit!

# Lit parameters serialized here for llvm-lit to pick them up
config.compiler = "/var/llvm/local-x86_64-release-stage2-A-flang/tools/clang/sta
ge2-bins/./bin/clang++"
config.hardening_mode = "extensive"
config.target_triple = "x86_64-unknown-linux-gnu"


#
# This file performs the bridge between the CMake configuration and the Lit

There's something amiss here.

However, in .../runtimes/runtimes-bin/CMakeCache.txt there's LIBCXX_INCLUDE_BENCHMARKS:BOOL=ON. In that case, libcxx/test/CMakeLists.txt doesn't substitute the enable_benchmarks value at all. And unless I'm mistaken the default value of enable_benchmarks in libcxx/utils/libcxx/test/params.py is run, exactly as I observe.

[ldionne]

One thing I just noticed: isn't enable_benchmarks supposed to be substituted at the top of libcxx/test/configs/cmake-bridge.cfg.in in @SERIALIZED_LIT_PARAMS@?

I don't think that's supposed to be the case, why do you think it should be the case? This is how enable_benchmarks=dry-run makes it into the check-cxx target but not others.

To buy ourselves some time to resolve this issue properly, I think we could at least temporarily disable the benchmarks on the release branch.