stall_detector: no backtrace if exception #2714
Conversation
If a an exception is in progress when a reactor stall is detected, omit taking the backtrace, as this can crash as detailed in scylladb#2697. Add a test which reproduces the issue, and which crashes (sometimes) before this fix and which runs cleanly afterwards. Fixes scylladb#2697.
|
For the record. There was an attempt to still collect the backtrace and don't crash if it fails (#2420) |
Thanks for the pointer @xemul, I hadn't seen that issue. I had done an issue search I think but did not search PRs. So this PR at least provides a reproducer which may help in evaluating the other PR as well. |
|
I also considered the longjmp approach, but went with this as I think it is simpler and overall safer. The downside is missing stalls in exceptions: however, IME that is very uncommon. I can't recall any stall which was "always" in an exception (this would probably crash pretty frequently if so), and the ones that are occasionally in an exception are usually there very rarely and the non-exception reports point just fine to the underlying location. Exceptions are slow so we also tend to remove them from hot paths. |
|
Maybe test @xemul fix with @travisdowns reproducer? Stalls in exception storms will land in exceptions. Good code should be de-exceptionalize such areas, but that's only one once experienced. |
If a an exception is in progress when a reactor stall is detected, omit taking the backtrace, as this can crash as detailed in #2697.
Add a test which reproduces the issue, and which crashes (sometimes) before this fix and which runs cleanly afterwards.
Fixes #2697.
This is the same fix we have applied in our seastar branch for Redpanda, in order to fix a similar crash in our not-upstreamed CPU profiler which relies on the same mechanism as the stall notifier. We have not seen any issues with that approach and the CPU profiler, when enabled, will take backtraces at a much higher frequency than the stall notifier in general (every 100ms, by default).
Of course, the question is if std::uncaught_exceptions is itself "signal safe" and you won't find any guarantee about it in the C++ standard (which largely ignores the existence of signals). I did check the implementation of this method in libc++ and libstdc++ and it looks "OK" to me: they both read a single field from an exceptions globals area, which looks to be OK to me.