Crashes observed from torrent::Chunk::to_buffer on v0.16.5

[anthonyryan1]

I have observed an infrequent crash originating from torrent::Chunk::to_buffer, that I suspect may be a Chunk use-after-free.

While investigating this bug, I grep'd through the code looking at everywhere chunks would be allocated or freed, trying to check my theory.

I came across lines 137 - 142 in ChunkList::get which creates a new chunk and deletes the old one. I also noticed that on line 130 of that function, we're checking that node->blocking() != 0 but we're not checking node->references() != 0 currently.

Based on an unconfirmed theory that we might be seeing chunks with blocking = 0 and references != 0. I experimented with changing that line to if (node->blocking() != 0 || node->references() != 0) { and after that change I have not seen this crash happen again since. Given the infrequency, and the number of other infrequent crashes we are currently tracking in parallel, I can't say with certainty that I didn't just change the crash signature or mask the real bug.

I still haven't been able to figure out what sequence of events could lead a chunk having blocking = 0 but references != 0 in this code path.

I figured I would share my findings so far, in case the missing pieces of this bug are immediately obvious to someone more familiar. Verify that blocking() = 0 and references() != 0 is a valid state (seems like it should be), or if it would be helpful for us to add some assertions into chunk_list_node.h to throw on unexpected conditions like negative references, or inter-property constraint violations.

I appreciate any insight you can add.

Stack:

Program terminated with signal SIGABRT, Aborted.
#0  0x00007f45006a168b in pthread_kill () from /usr/lib64/libc.so.6
[Current thread is 1 (Thread 0x7f4500ce2e80 (LWP 74778))]

Thread 5 (Thread 0x7f44f27fa6c0 (LWP 74780)):
#0  0x00007f45006a7b72 in ?? () from /usr/lib64/libc.so.6
#1  0x00007f450069bc8c in ?? () from /usr/lib64/libc.so.6
#2  0x00007f4500721a11 in epoll_wait () from /usr/lib64/libc.so.6
#3  0x00007f4500d9a79c in torrent::net::Poll::poll (this=this@entry=0x55928c572c70, timeout_usec=<optimized out>) at net/poll_epoll.cc:170
#4  0x00007f4500d9aa41 in torrent::net::Poll::do_poll (this=0x55928c572c70, timeout_usec=<optimized out>) at net/poll_epoll.cc:156
#5  0x00007f4500dc29c3 in torrent::utils::Thread::event_loop (this=this@entry=0x55928c56c960) at /usr/lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/unique_ptr.h:192
#6  0x00007f4500dc32a7 in torrent::utils::Thread::enter_event_loop (thread=0x55928c56c960) at utils/thread.cc:130
#7  0x00007f450069f7a1 in ?? () from /usr/lib64/libc.so.6
#8  0x00007f450072172c in ?? () from /usr/lib64/libc.so.6

Thread 4 (Thread 0x7f44f16f76c0 (LWP 74786)):
#0  0x00007f45006a7b72 in ?? () from /usr/lib64/libc.so.6
#1  0x00007f450069bc8c in ?? () from /usr/lib64/libc.so.6
#2  0x00007f4500721a11 in epoll_wait () from /usr/lib64/libc.so.6
#3  0x00007f4500d9a79c in torrent::net::Poll::poll (this=this@entry=0x55928c571e60, timeout_usec=<optimized out>) at net/poll_epoll.cc:170
#4  0x00007f4500d9aa41 in torrent::net::Poll::do_poll (this=0x55928c571e60, timeout_usec=<optimized out>) at net/poll_epoll.cc:156
#5  0x00007f4500dc29c3 in torrent::utils::Thread::event_loop (this=this@entry=0x55928c57f9a0) at /usr/lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/unique_ptr.h:192
#6  0x00007f4500dc32a7 in torrent::utils::Thread::enter_event_loop (thread=0x55928c57f9a0) at utils/thread.cc:130
#7  0x00007f450069f7a1 in ?? () from /usr/lib64/libc.so.6
#8  0x00007f450072172c in ?? () from /usr/lib64/libc.so.6

Thread 3 (Thread 0x7f44f2ffb6c0 (LWP 74779)):
#0  0x00007f45006a7b72 in ?? () from /usr/lib64/libc.so.6
#1  0x00007f450069bc8c in ?? () from /usr/lib64/libc.so.6
#2  0x00007f4500721a11 in epoll_wait () from /usr/lib64/libc.so.6
#3  0x00007f4500d9a79c in torrent::net::Poll::poll (this=this@entry=0x55928c572c50, timeout_usec=<optimized out>) at net/poll_epoll.cc:170
#4  0x00007f4500d9aa41 in torrent::net::Poll::do_poll (this=0x55928c572c50, timeout_usec=<optimized out>) at net/poll_epoll.cc:156
#5  0x00007f4500dc29c3 in torrent::utils::Thread::event_loop (this=this@entry=0x55928c572320) at /usr/lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/unique_ptr.h:192
#6  0x00007f4500dc32a7 in torrent::utils::Thread::enter_event_loop (thread=0x55928c572320) at utils/thread.cc:130
#7  0x00007f450069f7a1 in ?? () from /usr/lib64/libc.so.6
#8  0x00007f450072172c in ?? () from /usr/lib64/libc.so.6

Thread 2 (Thread 0x7f44f1ff96c0 (LWP 74781)):
#0  0x00007f45006a7b72 in ?? () from /usr/lib64/libc.so.6
#1  0x00007f450069bc8c in ?? () from /usr/lib64/libc.so.6
#2  0x00007f4500721a11 in epoll_wait () from /usr/lib64/libc.so.6
#3  0x00007f4500d9a79c in torrent::net::Poll::poll (this=this@entry=0x55928c572d10, timeout_usec=<optimized out>) at net/poll_epoll.cc:170
#4  0x00007f4500d9aa41 in torrent::net::Poll::do_poll (this=0x55928c572d10, timeout_usec=<optimized out>) at net/poll_epoll.cc:156
#5  0x00007f4500dc29c3 in torrent::utils::Thread::event_loop (this=this@entry=0x55928c57c440) at /usr/lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/unique_ptr.h:192
#6  0x00007f4500dc32a7 in torrent::utils::Thread::enter_event_loop (thread=0x55928c57c440) at utils/thread.cc:130
#7  0x00007f450069f7a1 in ?? () from /usr/lib64/libc.so.6
#8  0x00007f450072172c in ?? () from /usr/lib64/libc.so.6

Thread 1 (Thread 0x7f4500ce2e80 (LWP 74778)):
#0  0x00007f45006a168b in pthread_kill () from /usr/lib64/libc.so.6
#1  0x00007f45006420e6 in raise () from /usr/lib64/libc.so.6
#2  0x00007f4500625315 in abort () from /usr/lib64/libc.so.6
#3  0x00005592653707dc in handle_sigbus (signum=<optimized out>, sa=0x7ffcdf42ea30, ptr=<optimized out>) at main.cc:612
#4  <signal handler called>
#5  0x00007f4500781e41 in memcpy () from /usr/lib64/libc.so.6
#6  0x00007f4500dda73f in memcpy (__dest=0x55928e1b0430, __src=<optimized out>, __len=16384) at /usr/include/bits/string_fortified.h:29
#7  torrent::Chunk::to_buffer (this=0x55928e15d000, buffer=0x55928e1b0430, position=<optimized out>, length=length@entry=16384) at data/chunk.cc:227
#8  0x00007f4500e29969 in torrent::PeerConnectionBase::up_chunk_encrypt (this=0x55928e116860, quota=16384) at protocol/peer_connection_base.cc:691
#9  torrent::PeerConnectionBase::up_chunk (this=this@entry=0x55928e116860) at protocol/peer_connection_base.cc:720
#10 0x00007f4500e3157c in torrent::PeerConnection<(torrent::Download::ConnectionType)1>::event_write (this=0x55928e116860) at protocol/peer_connection_leech.cc:598
#11 0x00007f4500d9a953 in torrent::net::Poll::process (this=0x55928c546a70) at net/poll_epoll.cc:229
#12 0x00007f4500d9aa65 in torrent::net::Poll::do_poll (this=<optimized out>, timeout_usec=<optimized out>) at net/poll_epoll.cc:165
#13 0x00007f4500dc29c3 in torrent::utils::Thread::event_loop (this=0x55928c546510) at /usr/lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/unique_ptr.h:192
#14 0x00005592653ce544 in main (argc=1, argv=0x7ffcdf42f4b8) at main.cc:504

[rakshasa]

SIGBUS on mmap'ed memory usually indicates the kernel invalidated the memory mapped file in some ways.

IIRC this can happen when running out of disk space or memory, so check those first.