Busy public node segfaults since 4.3 update

[Jayd603]

monerod is segfaulting roughly once per few hours (docker container autorestarts). I have rolled back to 4.2 to see if the issue goes away.

[1699348.429995] monerod[1354261]: segfault at a3 ip 0000561c0c54c018 sp 00007f1159080120 error 4 in monerod[561c0c474000+795000]

Update: rolling back actually didn't fix this.. it seems like there is a flood of traffic, hundreds of IPs are blocked all within one second then the crash happens...

2025-10-13 11:16:09.378	I Host 104.248.206.131 blocked.
2025-10-13 11:16:09.379	I Host 104.248.231.61 blocked.
2025-10-13 11:16:09.380	I Host 116.203.249.163 blocked.

.... x 200+ IPs blocked within a second.

2025-10-13 11:16:16.610	I ----- BLOCK ADDED AS ALTERNATIVE ON HEIGHT 3515415
2025-10-13 11:16:16.610	I id:	<c644920dd719720dece040056b1a512b7963fda79de2cf65786ececd449bd08b>
2025-10-13 11:16:16.610	I PoW:	<b662ea2c8f01df66a75315c8019ea1c28559b389a0909d3c6d34960000000000>
2025-10-13 11:16:16.610	I difficulty:	561959871867
2025-10-13 11:35:58.991	I Monero 'Fluorine Fermi' (v0.18.4.2-release)
2025-10-13 11:35:58.991	I Initializing cryptonote protocol...

I don't have debugging on but will do it. Other VMs on the hypervisor are not experiencing any segfaults so I don't believe it is a system hardware issue.

[Jayd603]

FROM ubuntu:latest
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y \
  build-essential \
  cmake \
  make \
  pkg-config \
  libboost-all-dev \
  libssl-dev \
  libzmq3-dev \
  libpgm-dev \
  libnorm-dev \
  libunbound-dev \
  libsodium-dev \
  libunwind8-dev \
  liblzma-dev \
  libreadline6-dev \
  libldns-dev \
  libexpat1-dev \
  libgtest-dev \
  ccache \
  doxygen \
  graphviz \
  qttools5-dev-tools \
  libhidapi-dev \
  libusb-1.0-0-dev \
  libprotobuf-dev \
  protobuf-compiler \
  libudev-dev \
  git \
  screen \
  && rm -rf /var/lib/apt/lists/*
WORKDIR /usr/src/gtest
RUN cmake . && make; mv lib/libg* /usr/lib/
WORKDIR /root/sources
RUN git clone --branch v0.18.4.3 --recursive https://github.com/monero-project/monero
WORKDIR /root/sources/monero
RUN git checkout -b release-v0.18; git submodule update --init --force; make -j7
EXPOSE 18080 18081
VOLUME /mnt/monero_data
WORKDIR /root/xmr
RUN cp /root/sources/monero/build/Linux/release-v0.18/release/bin/* .
ENTRYPOINT ["/root/xmr/monerod"]
CMD ["--non-interactive", \
    "--data-dir=/mnt/monero_data", \
    "--restricted-rpc", \
    "--rpc-bind-ip=0.0.0.0", \
    "--confirm-external-bind", \
    "--max-concurrency=5", \
    "--out-peers=3600", \
    "--prep-blocks-threads=4", \
    "--enable-dns-blocklist", \
    "--public-node", \
    "--limit-rate-down=8000000", \
    "--limit-rate-up=8000000", \
    "--block-sync-size=20", \
    "--max-log-file-size=10000000", \
    "--max-log-files=5", \
    "--db-sync-mode=safe" \
    ]

[selsta]

Could it be that you have a corrupted blockchain? That usually causes crashes or mass banning of peers because it can't verify blocks correctly.

[Jayd603]

This is the bitmonero.log around that time. I guess I will download the blockchain again and see what happens.

2025-10-13 11:16:02.504	[P2P1]	INFO	stacktrace	src/common/stack_trace.cpp:134	Exception: boost::wrapexcept<boost::bad_weak_ptr>
2025-10-13 11:16:02.504	[P2P1]	INFO	stacktrace	src/common/stack_trace.cpp:135	Unwound call stack:
2025-10-13 11:16:02.504	[P2P1]	INFO	stacktrace	src/common/stack_trace.cpp:173	    [1]  0xb0) [0x559b9624c60c]:__cxa_throw+0xb0) [0x559b9624c60c]
2025-10-13 11:16:02.504	[P2P1]	INFO	stacktrace	src/common/stack_trace.cpp:173	    [2] /root/xmr/monerod(+0x8ee51) [0x559b96224e51] 
2025-10-13 11:16:02.504	[P2P1]	INFO	stacktrace	src/common/stack_trace.cpp:173	    [3] /root/xmr/monerod(+0x465e23) [0x559b965fbe23] 
2025-10-13 11:16:02.504	[P2P1]	INFO	stacktrace	src/common/stack_trace.cpp:173	    [4] /root/xmr/monerod(+0x47c9c1) [0x559b966129c1] 
2025-10-13 11:16:02.504	[P2P1]	INFO	stacktrace	src/common/stack_trace.cpp:173	    [5] /root/xmr/monerod(+0x48e8f6) [0x559b966248f6] 
2025-10-13 11:16:02.504	[P2P1]	INFO	stacktrace	src/common/stack_trace.cpp:173	    [6] /root/xmr/monerod(+0x4ebe74) [0x559b96681e74] 
2025-10-13 11:16:02.504	[P2P1]	INFO	stacktrace	src/common/stack_trace.cpp:173	    [7] /root/xmr/monerod(+0x4ee368) [0x559b96684368] 
2025-10-13 11:16:02.504	[P2P1]	INFO	stacktrace	src/common/stack_trace.cpp:173	    [8] /root/xmr/monerod(+0x4efc85) [0x559b96685c85] 
2025-10-13 11:16:02.504	[P2P1]	INFO	stacktrace	src/common/stack_trace.cpp:173	    [9] /root/xmr/monerod(+0x461a77) [0x559b965f7a77] 
2025-10-13 11:16:02.504	[P2P1]	INFO	stacktrace	src/common/stack_trace.cpp:173	    [10] /root/xmr/monerod(+0x49437e) [0x559b9662a37e] 
2025-10-13 11:16:02.504	[P2P1]	INFO	stacktrace	src/common/stack_trace.cpp:173	    [11] /root/xmr/monerod(+0x4a99eb) [0x559b9663f9eb] 
2025-10-13 11:16:02.504	[P2P1]	INFO	stacktrace	src/common/stack_trace.cpp:173	    [12] /root/xmr/monerod(+0x44936d) [0x559b965df36d] 
2025-10-13 11:16:02.504	[P2P1]	INFO	stacktrace	src/common/stack_trace.cpp:173	    [13] /root/xmr/monerod(+0x48171a) [0x559b9661771a] 
2025-10-13 11:16:02.504	[P2P1]	INFO	stacktrace	src/common/stack_trace.cpp:173	    [14]  0x1699f) [0x7f19fd0e199f]:_64-linux-gnu/libboost_thread.so.1.83.0(+0x1699f) [0x7f19fd0e199f]
2025-10-13 11:16:02.504	[P2P1]	INFO	stacktrace	src/common/stack_trace.cpp:173	    [15]  0x9caa4) [0x7f19fcb7baa4]:_64-linux-gnu/libc.so.6(+0x9caa4) [0x7f19fcb7baa4]
2025-10-13 11:16:02.504	[P2P1]	INFO	stacktrace	src/common/stack_trace.cpp:173	    [16]  0x44) [0x7f19fcc08a64]:_64-linux-gnu/libc.so.6(__clone+0x44) [0x7f19fcc08a64]
2025-10-13 11:16:02.504	[P2P1]	INFO	stacktrace	src/common/stack_trace.cpp:173	
2025-10-13 11:16:09.378	[P2P1]	INFO	global	src/p2p/net_node.inl:327	Host 104.248.206.131 blocked.

[200 blocked host messages omitted]

2025-10-13 11:16:16.610	[P2P9]	INFO	global	src/cryptonote_core/blockchain.cpp:2110	----- BLOCK ADDED AS ALTERNATIVE ON HEIGHT 3515415
2025-10-13 11:16:16.610	[P2P9]	INFO	global	src/cryptonote_core/blockchain.cpp:2110	id:	<c644920dd719720dece040056b1a512b7963fda79de2cf65786ececd449bd08b>
2025-10-13 11:16:16.610	[P2P9]	INFO	global	src/cryptonote_core/blockchain.cpp:2110	PoW:	<b662ea2c8f01df66a75315c8019ea1c28559b389a0909d3c6d34960000000000>
2025-10-13 11:16:16.610	[P2P9]	INFO	global	src/cryptonote_core/blockchain.cpp:2110	difficulty:	561959871867
2025-10-13 11:17:04.659	[P2P4]	INFO	stacktrace	src/common/stack_trace.cpp:134	Exception: boost::wrapexcept<boost::bad_weak_ptr>
2025-10-13 11:17:04.659	[P2P4]	INFO	stacktrace	src/common/stack_trace.cpp:135	Unwound call stack:

[nahuhh]

Go back further on the log

that bad weak pointer unwound log isnt related (it probably prints every few mins)

[Jayd603]

Nothing else seems to show in the log - I re-downloaded the entire blockchain, the same issue is occurring. I'll move the machine to new hardware in case that's it.

[nahuhh]

Remove this
"--out-peers=3600"

[nahuhh]

monerod is segfaulting roughly once per few hours (docker container autorestarts).

is this rebuilding monero every hour?

FROM ubuntu:latest
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y \
  build-essential \
  cmake \
  make \
  pkg-config \
  libboost-all-dev \
  libssl-dev \
  libzmq3-dev \
  libpgm-dev \
  libnorm-dev \
  libunbound-dev \
  libsodium-dev \
  libunwind8-dev \
  liblzma-dev \
  libreadline6-dev \
  libldns-dev \
  libexpat1-dev \
  libgtest-dev \
  ccache \
  doxygen \
  graphviz \
  qttools5-dev-tools \
  libhidapi-dev \
  libusb-1.0-0-dev \
  libprotobuf-dev \
  protobuf-compiler \
  libudev-dev \
  git \
  screen \
  && rm -rf /var/lib/apt/lists/*
WORKDIR /usr/src/gtest
RUN cmake . && make; mv lib/libg* /usr/lib/
WORKDIR /root/sources
RUN git clone --branch v0.18.4.3 --recursive https://github.com/monero-project/monero
WORKDIR /root/sources/monero
RUN git checkout -b release-v0.18; git submodule update --init --force; make -j7
EXPOSE 18080 18081
VOLUME /mnt/monero_data
WORKDIR /root/xmr
RUN cp /root/sources/monero/build/Linux/release-v0.18/release/bin/* .
ENTRYPOINT ["/root/xmr/monerod"]
CMD ["--non-interactive", \
    "--data-dir=/mnt/monero_data", \
    "--restricted-rpc", \
    "--rpc-bind-ip=0.0.0.0", \
    "--confirm-external-bind", \
    "--max-concurrency=5", \
    "--out-peers=3600", \
    "--prep-blocks-threads=4", \
    "--enable-dns-blocklist", \
    "--public-node", \
    "--limit-rate-down=8000000", \
    "--limit-rate-up=8000000", \
    "--block-sync-size=20", \
    "--max-log-file-size=10000000", \
    "--max-log-files=5", \
    "--db-sync-mode=safe" \
    ]

unrelated: where did you get this ..?

[Jayd603]

I removed the out-peers argument and no change. I tried reverting to 4.0 , same thing. I will try a new server but I'm not sure when I have time to migrate. Other VMs are not having these issues so far.

@nahuhh that is a Dockerfile for building a monerod docker image. That is only run once to build the image. I start a container from that image and the container is what restarts when monerod crashes.

Here is a log that may be relevant or might not. The main behavior is - tons of addresses get blocked, then it crashes.

2025-10-09 21:54:35.073	I ----- BLOCK ADDED AS ALTERNATIVE ON HEIGHT 3490631
2025-10-09 21:54:35.073	I id:	<cf2c5511ac67a5d6dc329302740a469ba6d4a11b2d893e9ad892df9d1ca72393>
2025-10-09 21:54:35.073	I PoW:	<3334215661842162bf7e545c0626ce347abbfc50900c2de1b18ef10000000000>
2025-10-09 21:54:35.073	I difficulty:	603536740310
2025-10-10 00:29:49.711	E Failed to parse transaction from blob
2025-10-10 01:34:44.173	I p2p net loop stopped
2025-10-10 01:34:44.209	I Stopping core RPC server...
2025-10-10 01:34:44.209	I Node stopped.

[nahuhh]

tons of addresses get blocked

youre using the dns blocklist, are you sure these arent blocklist addresses?

[Jayd603]

tons of addresses get blocked

youre using the dns blocklist, are you sure these arent blocklist addresses?

I'm sure they are - but there are a huge number all at once. I disabled dns-blocklist in case it's related, the crash still happens.

[Jayd603]

Just a quick update - I will be moving this to a new server to rule out the system and OS - eta 1 wk

[Jayd603]

I ended up doing this sooner than expected, new server - new OS (ubuntu 24.04 LTS), new docker image build. Same exact crash. This is an actual bug, it's just a question of where it is... the new blockchain download is not even complete yet and the daemon already crashed.

[Thu Oct 23 21:51:26 2025] monerod[1802]: segfault at a3 ip 000058ef4a403042 sp 00007e68651d80c0 error 4 in monerod[58ef4a32b000+792000] likely on CPU 1 (core 1, socket 0)

[selsta]

@Jayd603 which version did you use?