bad_alloc on AIO teardown

[xemul]

Sometimes (once a year or so) reactor crashes re-allocating a vector of iocbs for retry

[2024-02-16T09:17:41.425Z] ERROR 2024-02-16 10:19:34,274 [shard 0:main] seastar - aio retry failed: boost::container::bad_alloc (boost::container::bad_alloc thrown). Aborting.

The code is trying to push an iocb pointer to _free_iocbs even it already had it at the top. and at that moment, the underlying static_vector was already full. BTW, the iocb pointer points to the last slot in the _iocb_pool.

(gdb) p seastar::local_engine                                                                                          
$1 = (seastar::reactor *) 0x6010000be000  
(gdb) p seastar::local_engine->_backend                                                                                
$2 = std::unique_ptr<seastar::reactor_backend> = {                                                                                                                                                                                            
  get() = 0x601000140000                                                                                               
}   
(gdb) p iocbs[0]                                                                                                                                                                                                                              
$8 = (seastar::internal::linux_abi::iocb *) 0x601000152040      
(gdb) p/x &(((seastar::reactor_backend_aio*)(0x601000140000)))->_storage_context._iocb_pool._iocb_pool._M_elems[0]
$16 = 0x601000142080               
(gdb) p/x &(((seastar::reactor_backend_aio*)(0x601000140000)))->_storage_context._iocb_pool._iocb_pool._M_elems[1024]
$17 = 0x601000152080 
(gdb) p/x sizeof (((seastar::reactor_backend_aio*)(0x601000140000)))->_storage_context._iocb_pool._iocb_pool._M_elems[0]
$19 = 0x40 
(gdb) python-interactive                                                                                                                                                                                                                      
>>> free_cbs = static_vector(gdb.parse_and_eval("(((seastar::reactor_backend_aio*)(0x601000140000)))->_storage_context._iocb_pool._free_iocbs.c.m_holder"))
>>> print(len(free_cbs))
1024
>>> for i, e in enumerate(free_cbs):
...     if str(e) == '0x601000152040':  
...         print(i)
1023
>>> iocbs_0 = gdb.parse_and_eval('*iocbs[0]')
>>> print(iocbs_0)
{
  aio_data = 105621836876800,
  aio_key = 0,
  aio_rw_flags = 0,
  aio_lio_opcode = seastar::internal::linux_abi::iocb_cmd::PREAD,
  aio_reqprio = 0,
  aio_fildes = 14,
  aio_buf = 105621837724672,
  aio_nbytes = 512,
  aio_offset = 0,
  aio_reserved2 = 0,
  aio_flags = 1,
  aio_resfd = 12
}

No reliable reproducer