Race condition in ctx_t::send_command() causes UAF during concurrent socket destruction

[surprized]

This is a race condition in ctx_t::send_command() where it dereferences _slots[tid_] while destroy_socket() concurrently nulls and frees that slot under _slot_sync which results in a UAF (caught by ASan), or a SIGSEGV in signaler_t::send() on non-instrumented builds of libzmq users.

Steps to trigger:
While I cannot provide a reproducible PoC (the issue was caught using an internal build on proprietary software), this can be triggered by rapidly creating and destroying sockets across threads while IO threads route term-ack commands under heavy load.

Root Cause & Technical Details

The race seems observable from any code path that calls ctx_t::send_command() without already holding _slot_sync, most critically the internal termination acknowledgement protocol running entirely on IO threads:

IO thread:

check_term_acks()
  send_term_ack(_owner)
    object_t::send_command()
      ctx_t::send_command(_slots[tid_]->send())  <-- no lock held

Reaper thread (concurrently):

socket_base_t::check_destroy()
  destroy_socket()                <-- _slots[tid] = NULL, mailbox freed
  own_t::process_destroy()        <-- delete this

ASan Report

<process name>.1594
=================================================================
==1594==ERROR: AddressSanitizer: heap-use-after-free on address 0xffff9720f828 at pc 0xffff9e89ff34 bp 0xffff78ce9e00 sp 0xffff78ce9e18
READ of size 4 at 0xffff9720f828 thread T15 (ZMQbg/IO/0)
    #0 0xffff9e89ff30 in zmq::signaler_t::send() src/signaler.cpp:195
    #1 0xffff9e7d5860 in zmq::mailbox_t::send(zmq::command_t const&) src/mailbox.cpp:73
    #2 0xffff9e7443d0 in zmq::ctx_t::send_command(unsigned int, zmq::command_t const&) src/ctx.cpp:491
    #3 0xffff9e80bf28 in zmq::object_t::send_command(zmq::command_t&) src/object.cpp:551
    #4 0xffff9e80d860 in zmq::object_t::send_term_ack(zmq::own_t*) src/object.cpp:396
    #5 0xffff9e81c630 in zmq::own_t::check_term_acks() src/own.cpp:202
    #6 0xffff9e81d694 in zmq::own_t::process_term(int) src/own.cpp:170
    #7 0xffff9e894e20 in zmq::session_base_t::pipe_terminated(zmq::pipe_t*) src/session_base.cpp:283
    #8 0xffff9e8281f0 in zmq::pipe_t::process_pipe_term_ack() src/pipe.cpp:372
    #9 0xffff9e80a0d4 in zmq::object_t::process_command(zmq::command_t&) src/object.cpp:140
    #10 0xffff9e7c1eb8 in zmq::io_thread_t::in_event() src/io_thread.cpp:101
    #11 0xffff9e7b5424 in zmq::epoll_t::loop() src/epoll.cpp:253
    #12 0xffff9e846420 in zmq::worker_poller_base_t::worker_routine(void*) src/poller_base.cpp:139
    #13 0xffff9e9214f8 in thread_routine src/thread.cpp:225
    #14 0xffffa090984c  (/lib/libpthread.so.0+0x884c)
    #15 0xffff9fa9ef68  (/lib/libc.so.6+0xcdf68)

0xffff9720f828 is located 104 bytes inside of 176-byte region [0xffff9720f7c0,0xffff9720f870)
freed by thread T14 (ZMQbg/Reaper) here:
    #0 0xffffa31e67b8 in operator delete(void*, unsigned long) (/usr/lib/libasan.so.5+0xe77b8)
    #1 0xffff9e7d5518 in zmq::mailbox_t::~mailbox_t() src/mailbox.cpp:56
    #2 0xffff9e8b3168 in zmq::socket_base_t::~socket_base_t() src/socket_base.cpp:282
    #3 0xffff9e79c3ec in zmq::dealer_t::~dealer_t() src/dealer.cpp:43
    #4 0xffff9e86e30c in zmq::req_t::~req_t() src/req.cpp:51
    #5 0xffff9e86e454 in zmq::req_t::~req_t() src/req.cpp:53
    #6 0xffff9e81ad54 in zmq::own_t::process_destroy() src/own.cpp:211
    #7 0xffff9e8bfcd0 in zmq::socket_base_t::check_destroy() src/socket_base.cpp:1591
    #8 0xffff9e8c16bc in zmq::socket_base_t::in_event() src/socket_base.cpp:1564
    #9 0xffff9e7b5424 in zmq::epoll_t::loop() src/epoll.cpp:253
    #10 0xffff9e846420 in zmq::worker_poller_base_t::worker_routine(void*) src/poller_base.cpp:139
    #11 0xffff9e9214f8 in thread_routine src/thread.cpp:225
    #12 0xffffa090984c  (/lib/libpthread.so.0+0x884c)
    #13 0xffff9fa9ef68  (/lib/libc.so.6+0xcdf68)

previously allocated by thread T0 here:
    #0 0xffffa31e58a0 in operator new(unsigned long, std::nothrow_t const&) (/usr/lib/libasan.so.5+0xe68a0)
    #1 0xffff9e8b2354 in zmq::socket_base_t::socket_base_t(zmq::ctx_t*, unsigned int, int, bool) src/socket_base.cpp:256
    #2 0xffff9e79d018 in zmq::dealer_t::dealer_t(zmq::ctx_t*, unsigned int, int) src/dealer.cpp:38
    #3 0xffff9e86e7cc in zmq::req_t::req_t(zmq::ctx_t*, unsigned int, int) src/req.cpp:46
    #4 0xffff9e8a263c in zmq::socket_base_t::create(int, zmq::ctx_t*, unsigned int, int) src/socket_base.cpp:164
    #5 0xffff9e748e50 in zmq::ctx_t::create_socket(int) src/ctx.cpp:384
    #6 0xffff9e96f144 in zmq_socket src/zmq.cpp:242
    <redacted trace>

Thread T15 (ZMQbg/IO/0) created by T13 here:
    #0 0xffffa313af28 in __interceptor_pthread_create (/usr/lib/libasan.so.5+0x3bf28)
    #1 0xffff9e920454 in zmq::thread_t::start(void (*)(void*), void*, char const*) src/thread.cpp:236
    #2 0xffff9e742238 in zmq::thread_ctx_t::start_thread(zmq::thread_t&, void (*)(void*), void*, char const*) const src/ctx.cpp:437
    #3 0xffff9e8468e4 in zmq::worker_poller_base_t::start(char const*) src/poller_base.cpp:127
    #4 0xffff9e7bfefc in zmq::io_thread_t::start() src/io_thread.cpp:67
    #5 0xffff9e747e10 in zmq::ctx_t::start() src/ctx.cpp:333
    #6 0xffff9e748efc in zmq::ctx_t::create_socket(int) src/ctx.cpp:360
    #7 0xffff9e96f144 in zmq_socket src/zmq.cpp:242
    <redacted trace>

Thread T13 created by T6 here:
    <redacted trace>

Thread T6 created by T0 here:
    <redacted trace>

Thread T14 (ZMQbg/Reaper) created by T13 here:
    #0 0xffffa313af28 in __interceptor_pthread_create (/usr/lib/libasan.so.5+0x3bf28)
    #1 0xffff9e920454 in zmq::thread_t::start(void (*)(void*), void*, char const*) src/thread.cpp:236
    #2 0xffff9e742238 in zmq::thread_ctx_t::start_thread(zmq::thread_t&, void (*)(void*), void*, char const*) const src/ctx.cpp:437
    #3 0xffff9e8468e4 in zmq::worker_poller_base_t::start(char const*) src/poller_base.cpp:127
    #4 0xffff9e86a6dc in zmq::reaper_t::start() src/reaper.cpp:74
    #5 0xffff9e747980 in zmq::ctx_t::start() src/ctx.cpp:315
    #6 0xffff9e748efc in zmq::ctx_t::create_socket(int) src/ctx.cpp:360
    #7 0xffff9e96f144 in zmq_socket src/zmq.cpp:242
    <redacted trace>

SUMMARY: AddressSanitizer: heap-use-after-free src/signaler.cpp:195 in zmq::signaler_t::send()
Shadow bytes around the buggy address:
  0x200ff2e41eb0: fd fd fd fd fa fa fa fa fa fa fa fa fd fd fd fd
  0x200ff2e41ec0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x200ff2e41ed0: fd fd fa fa fa fa fa fa fa fa 00 00 00 00 00 00
  0x200ff2e41ee0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x200ff2e41ef0: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
=>0x200ff2e41f00: fd fd fd fd fd[fd]fd fd fd fd fd fd fd fd fa fa
  0x200ff2e41f10: fa fa fa fa fa fa fd fd fd fd fd fd fd fd fd fd
  0x200ff2e41f20: fd fd fd fd fd fd fd fd fd fd fd fd fa fa fa fa
  0x200ff2e41f30: fa fa fa fa fd fd fd fd fd fd fd fd fd fd fd fd
  0x200ff2e41f40: fd fd fd fd fd fd fd fd fd fd fa fa fa fa fa fa
  0x200ff2e41f50: fa fa fd fd fd fd fd fd fd fd fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc 

A security advisory was issued but the developers state this is a not a security concern, therefore, posting here to keep track of the core issue.