SIGILL Crash due to _u.st == state::future assertion failure in seastar::future::set (Reactor/AIO related) on program exit

[ujoyli]

Description:

Our application, which uses Seastar, is experiencing a hard crash (SIGILL) within the Seastar core library. The crash is triggered by an assertion failure indicating a state inconsistency within a seastar::future object, specifically related to the reactor's AIO event processing.
The crash appears to happen deep within the reactor loop while handling file descriptor completions, suggesting a potential race condition or an invalid state transition in seastar::future when used in this specific context.

Stack Trace:

[Thread debugging using libthread_db enabled]                                                                                                                                                                                                    
Using host libthread_db library "/lib64/libthread_db.so.1".                                                                                                                                                                                      
Core was generated by `/home/lishuai/tstore/build/src/tfs_server/tfs_server --network-stack=posix --po'.                                                                                                                                         
Program terminated with signal SIGILL, Illegal instruction.                                                                                                                                                                                      
#0  seastar::internal::assert_fail (msg=msg@entry=0x1149786 "_u.st == state::future", file=file@entry=0x1458d70 "/tstore/tstore_dependencies/src/seastar-seastar-25.05.0/include/seastar/core/future.hh", line=line@entry=612,                   
    func=func@entry=0x114a390 "void seastar::future_state<U>::set(A&& ...) [with A = {}; T = seastar::internal::monostate]") at /tstore/tstore_dependencies/src/seastar-seastar-25.05.0/src/util/log.cc:128                                      
128     /tstore/tstore_dependencies/src/seastar-seastar-25.05.0/src/util/log.cc: Directory not empty.                                                                                                                                            
[Current thread is 1 (Thread 0x7ffff65ac900 (LWP 427351))]                                                                                                                                                                                       
Missing separate debuginfos, use: dnf debuginfo-install boost-program-options-1.75.0-8.el9.x86_64 c-ares-1.17.1-5.el9.x86_64 fmt-8.1.1-5.el9.x86_64 gmp-6.2.0-10.el9.x86_64 gnutls-3.7.6-18.el9_1.x86_64 hwloc-libs-2.4.1-5.el9.x86_64 libatomic-
11.3.1-2.1.el9.alma.x86_64 libgcc-11.3.1-2.1.el9.alma.x86_64 libibverbs-41.0-3.el9.x86_64 libidn2-2.3.0-7.el9.el9_1.x86_64 librdmacm-41.0-3.el9.x86_64 libtasn1-4.16.0-8.el9_1.x86_64 libunistring-0.9.10-15.el9.x86_64 lksctp-tools-1.0.19-2.el9.x86_6
4 lz4-libs-1.9.3-5.el9.x86_64 nettle-3.8-3.el9_0.x86_64 openssl-libs-3.0.1-47.el9_1.x86_64 p11-kit-0.24.1-2.el9.x86_64 protobuf-3.14.0-13.el9.x86_64 protobuf-compiler-3.14.0-13.el9.x86_64 snappy-1.1.8-8.el9.x86_64 yaml-cpp-0.6.3-5.el9.x86_64
 zlib-1.2.11-35.el9_1.x86_64                                                                                                                                                                                                                     
(gdb) bt                                                                                                                                                                                                                                         
#0  seastar::internal::assert_fail (msg=msg@entry=0x1149786 "_u.st == state::future", file=file@entry=0x1458d70 "/tstore/tstore_dependencies/src/seastar-seastar-25.05.0/include/seastar/core/future.hh", line=612,                   
    func=func@entry=0x114a390 "void seastar::future_state<U>::set(A&& ...) [with A = {}; T = seastar::internal::monostate]") at /tstore/tstore_dependencies/src/seastar-seastar-25.05.0/src/util/log.cc:128                                      
#1  0x00000000010f9335 in seastar::future_state<seastar::internal::monostate>::set<>() (this=<optimized out>) at /tstore/tstore_dependencies/src/seastar-seastar-25.05.0/include/seastar/core/future.hh:612                                      
#2  seastar::future_state<seastar::internal::monostate>::set<>() (this=<optimized out>) at /tstore/tstore_dependencies/src/seastar-seastar-25.05.0/include/seastar/core/future.hh:611                                                            
#3  seastar::internal::promise_base_with_type<void>::set_value<>() (this=<optimized out>) at /tstore/tstore_dependencies/src/seastar-seastar-25.05.0/include/seastar/core/future.hh:882                                                          
#4  seastar::promise<void>::set_value<>() (this=<optimized out>) at /tstore/tstore_dependencies/src/seastar-seastar-25.05.0/include/seastar/core/future.hh:970                                                                                   
#5  seastar::pollable_fd_state_completion::complete_with (this=<optimized out>, res=<optimized out>) at /tstore/tstore_dependencies/src/seastar-seastar-25.05.0/src/core/reactor_backend.cc:71                                                   
#6  0x00000000010f5ba1 in seastar::reactor_backend_aio::await_events (this=this@entry=0x600000140000, timeout=timeout@entry=0, active_sigmask=active_sigmask@entry=0x0)                                                                          
    at /tstore/tstore_dependencies/src/seastar-seastar-25.05.0/src/core/reactor_backend.cc:508                                                                                                                                                   
#7  0x00000000010f5d42 in seastar::reactor_backend_aio::reap_kernel_completions (this=0x600000140000) at /tstore/tstore_dependencies/src/seastar-seastar-25.05.0/src/core/reactor_backend.cc:541                                                 
#8  0x0000000000fb9d35 in seastar::reactor::poll_once (this=<optimized out>) at /tstore/tstore_dependencies/src/seastar-seastar-25.05.0/src/core/reactor.cc:3350                                                                                 
#9  operator() (__closure=0x7fffffffb9b0) at /tstore/tstore_dependencies/src/seastar-seastar-25.05.0/src/core/reactor.cc:3249                                                                                                                    
#10 std::__invoke_impl<bool, seastar::reactor::do_run()::<lambda()>&> (__f=...) at /opt/rh/gcc-toolset-12/root/usr/include/c++/12/bits/invoke.h:61
#11 std::__invoke_r<bool, seastar::reactor::do_run()::<lambda()>&> (__fn=...) at /opt/rh/gcc-toolset-12/root/usr/include/c++/12/bits/invoke.h:114
#12 std::_Function_handler<bool(), seastar::reactor::do_run()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /opt/rh/gcc-toolset-12/root/usr/include/c++/12/bits/std_function.h:290
#13 0x0000000000fed130 in std::function<bool ()>::operator()() const (this=0x7fffffffb9b0) at /opt/rh/gcc-toolset-12/root/usr/include/c++/12/bits/std_function.h:591
#14 seastar::reactor::do_run (this=0x60000007e000) at /tstore/tstore_dependencies/src/seastar-seastar-25.05.0/src/core/reactor.cc:3276
#15 0x0000000000fedb09 in seastar::reactor::run (this=<optimized out>) at /tstore/tstore_dependencies/src/seastar-seastar-25.05.0/src/core/reactor.cc:3143
#16 0x0000000000f73de7 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) (this=0x7fffffffbf00, ac=<optimized out>, av=0x7fffffffdf08, func=...)
    at /tstore/tstore_dependencies/src/seastar-seastar-25.05.0/include/seastar/core/reactor.hh:711
#17 0x0000000000f747be in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) (this=0x7fffffffbf00, ac=21, av=0x7fffffffdf08, func=...)
    at /tstore/tstore_dependencies/src/seastar-seastar-25.05.0/src/core/app-template.cc:167
#18 0x0000000000576643 in main (argc=21, argv=0x7fffffffdf08) at /tstore/src/tfs_server/tfs_main.cpp:190

Environment:

Seastar Version: 25.05.0
Operating System: AlmaLinux 9 (based on the dnf debuginfo-install output)
Compiler: GCC 12 (from gcc-toolset-12)
Network Stack used: --network-stack=posix
Backend used by Reactor: AIO (seastar::reactor_backend_aio::await_events)

[xemul]

What's the git commit id the seastar is built at?

[xemul]

BTW, it's pollable_fd_state_completion, so it's not disk IO, but (likely) networking.
Also, sigill comes from some assertion, which used to generate SIGILL (#2832), but nowadays calls std::terminate (not that it differs much for this case, but still)

[ujoyli]

commit 8df8212 (HEAD, tag: seastar-25.05.0, origin/seastar-25.05-branch)
Author: Calle Wilund calle@scylladb.com
Date: Mon May 26 09:12:12 2025 +0000

I think the issue may be related to the way I exit the program. I register a poller, and when shutting down (after receiving the SIGTERM signal), I call unregister_poller, and then set the corresponding poller pointer to null.

[nyh]

As @xemul noted, don't worry about why you get a bizarre SIGILL during an assertion failure - that was fixed in 86c4893 to be a SIGABRT. Which of course doesn't explain why the assertion failed in the first place. (By the way, you're using a 6 month old Seastar, you might want to try a newer snapshot just to see maybe it's a bug that was already fixed).

I see from your comment that this crash happens while exiting the program. This suggests maybe you can create a small reproducer which includes only the exiting part, and not your main application logic, and if so it might be easier to explain what the problem is - or find the bug.

Crashes during exit many times are caused by use-after-free bugs - during shutdown your code destroys some object which background work is still using. The easiest way to debug such code is to rebuild your code in debug build mode (which includes the "sanitizers") and see if it reports use-after-free and similar bugs - it's easier to debug these reports than some random assertion failure caused by the use-after-free.