Exception escaping `node_server::idle_worker()` permanently disables all p2p maintenance (out-peers → 0, connections never reaped)

[sethforprivacy]

Version: v0.18.5.1-release (Fluorine Fermi), Linux x86_64, musl/Alpine build, Docker.

Summary

boosted_tcp_server::global_timer_handler() re-arms its deadline_timer after invoking the
handler. If the handler throws, the re-arm is skipped and that idle-handler chain is permanently
dead for the lifetime of the process — silently, with no recovery path.

When the handler in question is node_server::idle_worker(), the node loses every piece of p2p
maintenance at once:

• connections_maker() stops → outbound peers decay to 0 and are never re-established
• peer_sync_idle_maker() stops → dead peers are never detected or dropped
• gray_peerlist_housekeeping(), check_incoming_connections(), store_config(),
  update_dns_blocklist() all stop

The node keeps accepting inbound connections and serving RPC, keeps reporting
"status": "OK" and "synchronized": true, and stays at chain tip via inbound peers only.
Externally it looks healthy. Internally it accumulates connection objects and file descriptors
forever.

cryptonote::core::on_idle is registered as a separate idle_callback_conext with its own
timer (net_node.inl ~L1042-1043), so it keeps running — which makes the failure even harder to
spot, since some periodic activity continues.

Observed on a production node

16-day-old container, RestartCount=0, chain at tip, status: OK. Onset was ~day 12; the
preceding 12 days were normal.

get_info:  incoming_connections_count=1660   outgoing_connections_count=0
           height=3734124  synchronized=true  status=OK
           white_peerlist_size=1000  grey_peerlist_size=3980

get_connections — 1660 connection objects:

incoming	state	count
true	before_handshake	1346
false	before_handshake	280
true	normal	36

• before_handshake live_time: min 97 s, p50 199 463 s (2.3 d), max 350 039 s (4.05 d)
• age histogram by day: {0: 311, 1: 373, 2: 456, 3: 484, 4: 2} — a hard cliff at day 4 dating
  onset precisely
• 1511 / 1626 before_handshake connections have send_count == 0; 886 have recv_count == 0

File descriptors (pid 1 in container, nofile = 65536, so this is not fd exhaustion):

total fds        1689
socket fds       1669
tcp rows in netns 311      <- CLOSE_WAIT 267, ESTAB 33, TIME_WAIT 6, LISTEN 5
socket fds with NO /proc/net/tcp entry:  1365

Those 1365 are sockets the kernel has fully torn down and unhashed, whose fd was never close()d.

CLOSE_WAIT breakdown — the leak is p2p, not RPC:

OUTBOUND -> remote port 18080   232     <- leaked back-ping sockets
INBOUND    local port 18080      28
everything else                   8

Three healthy nodes running the identical image/config/version for the same 16 days show 0–1
before_handshake connections (all sub-4-second) and 12 outbound peers each.

outgoing_connections_count = 0 is truthful, not an accounting bug

get_outgoing_connections_count() (src/p2p/net_node.inl L2029) counts
!cntxt.m_is_income && !cntxt.is_ping. All 280 non-income connection objects on the affected node
are is_ping back-pings, so they are correctly excluded. The node genuinely has zero real
outbound peers and depends entirely on inbound peers for block propagation.

net.p2p:INFO over 120 s confirms it — every OUT connection is a back-ping opened milliseconds
after an INC and closed ~10 ms later:

[45.142.30.74:36274 ... INC] NEW CONNECTION
[45.142.30.74:18080 ... OUT] NEW CONNECTION
[45.142.30.74:18080 ... OUT] CLOSE CONNECTION     <- 7 ms later
[<none> OUT] back ping connect failed to 79.204.56.99:18080

Proof that idle_worker() is not running

m_connections_maker_interval is once_a_time_seconds<1> (src/p2p/net_node.h L463), and
try_to_connect_and_handshake_with_new_peer() logs MDEBUG("Connecting to " << ...) as its first
statement after the guards. With out_peers = 0 against max_out_connection_count = 12, plus 5
configured --add-priority-node peers, this should fire continuously.

Over a 90-second window at net.p2p:DEBUG:

Connecting to                             0
Failed to connect to any, trying seeds    0
COMMAND_TIMED_SYNC                       26     <- inbound handling is fine
COMMAND_HANDSHAKE                         2
back ping connect failed                  1

Zero outbound attempts in 90 opportunities, while inbound event-driven work runs normally.

This is not lock contention. Sampling /proc/<pid>/task/*/syscall: 26 threads parked in futex,
on 26 distinct futex addresses — independent idle waits, no shared lock. The thread pool is
idle waiting for work that is never posted, not deadlocked.

Mechanism

contrib/epee/include/net/abstract_tcp_server2.h L490-498:

template<class t_handler>
bool global_timer_handler(/*const boost::system::error_code& err, */boost::shared_ptr<idle_callback_conext<t_handler>> ptr)
{
  //if handler return false - he don't want to be called anymore
  if(!ptr->call_handler())
    return true;
  ptr->m_timer.expires_from_now(boost::posix_time::milliseconds(ptr->m_period));
  ptr->m_timer.async_wait(boost::bind(&boosted_tcp_server<t_protocol_handler>::global_timer_handler<t_handler>, this, ptr));
  return true;
}

If call_handler() throws, both the expires_from_now and the async_wait are skipped. Nothing
ever re-arms that timer. There is no supervision and no restart path.

The exception then unwinds into worker_thread()
(contrib/epee/include/net/abstract_tcp_server2.inl L1433):

while(!m_stop_signal_sent)
{
  try { io_context_.run(); return true; }
  catch(const std::exception& ex) { _erro("Exception at server worker thread, what=" << ex.what()); }
  catch(...)                      { _erro("Exception at server worker thread, unknown execption"); }
}

The thread survives and re-enters run() — which is why the process looks fine — but the idle
chain is gone.

Why there is no trace of it in the log

abstract_tcp_server2.inl L56-57 sets MONERO_DEFAULT_LOG_CATEGORY "net", and the default
category string (contrib/epee/src/mlog.cpp L103) contains net:FATAL:

*:WARNING,net:FATAL,net.http:FATAL,net.ssl:FATAL,net.p2p:FATAL,net.cn:FATAL,daemon.rpc:FATAL,...

_erro is ERROR, which is below FATAL, so the one line that would identify the triggering
exception is discarded under the stock configuration. On the affected node the container log
covers the full 16 days from startup and contains no occurrence — consistent with suppression
rather than absence.

Candidate throw sites reachable from idle_worker()

try_to_connect_and_handshake_with_new_peer() (src/p2p/net_node.inl L1383) opens with:

network_zone& zone = m_network_zones.at(na.get_zone());

std::map::at throws std::out_of_range for any peerlist entry whose zone is not configured on
this node. This is reachable from idle_worker() → connections_maker() → connect_to_peerlist() / make_expected_connections_count(). I have not proven this is the specific throw that fired here —
the log line that would say so was suppressed — but it is a reachable, unguarded .at() on
attacker-influenceable peerlist data in exactly the call path that died.

Secondary failure: fd numbers crossing FD_SETSIZE break libunbound

Once the leak pushed monerod's fd numbers past 1024, libunbound's select() backend could no longer
register descriptors:

[1785839031] libunbound[1:0] error: event_add failed. in cpsl.       (4399 occurrences)
WARNING: no two valid DNS TXT records were received                  (43 occurrences)

First occurrence 2026-08-04 10:23:51 UTC, versus leak onset 2026-08-02 07:25 UTC — ~2 days later,
so this is a consequence of the leak, not its cause. Zero occurrences on the three healthy
nodes. Worth noting independently: any monerod holding >1024 fds loses DNS resolution
(seed nodes, DNS checkpoints, DNS blocklist), and raising nofile above 1024 — which is necessary
for a busy public node — makes this reachable.

Recovery

A restart clears it completely and immediately — consistent with a dead timer chain rather than
corrupted state:

before restart   in=1665  out=0   fds=1695  close_wait=270  before_handshake=1632
t+60s            in=1     out=12  fds=40    close_wait=0    before_handshake=0
t+120s           in=3     out=12  fds=42    close_wait=0    before_handshake=0
t+300s           in=6     out=12  fds=46    close_wait=0    before_handshake=0

Outbound peers return to the full --out-peers complement within a minute. Zero event_add failed
and zero DNS TXT warnings after the new startup banner, confirming the libunbound failure was purely
a function of fd numbering.

Worth noting for the exit path: docker restart -t 180 consumed the entire 180 s grace period
and the daemon had to be SIGKILLed — no orderly-shutdown messages were logged. Tearing down 1665
stuck connection objects with the p2p timer machinery already dead appears to hang the exit path,
which looks like the same underlying condition reported in #9482 ("monerod exit gets stuck/hangs at
deinitializing p2p"). LMDB survived it — no recovery or corruption messages on the next start.

Suggested fixes

1. Make the idle chain exception-safe. Either re-arm the timer before invoking the handler, or
   wrap call_handler() in try/catch inside global_timer_handler() so a single exception
   cannot permanently disable p2p maintenance. A transient failure should not be terminal.
2. Stop suppressing the diagnostic. Exception at server worker thread is a
   never-should-happen event logged under a category defaulted to FATAL. Raise it to FATAL, or
   move it to a category that is visible by default. As it stands this failure is undiagnosable
   from a stock log.
3. Guard the .at(). Use find() + a checked miss in
   try_to_connect_and_handshake_with_new_peer() rather than m_network_zones.at().
4. Consider a liveness assertion — e.g. warn if connections_maker() has not run in N minutes
   while outgoing_connections_count < max_out_connection_count.

Operator-visible signature

For anyone else hitting this: the node reports status: OK, stays synchronized, serves RPC
normally, and shows outgoing_connections_count: 0 with a large and growing
incoming_connections_count. fd usage is a poor signal — this node sat at 2.6 % of its 65536
limit. outgoing_connections_count == 0 on a node with a non-zero --out-peers is the reliable
alert. Only a restart recovers it.

[selsta]

Unfortunately without more detailed logs it's not possible to reconstruct what exactly went wrong.

[sethforprivacy]

I can try to pull more, anything specific you'd be looking for?

[selsta]

Hitting this bug with log level 2 would be ideal.

[sethforprivacy]

Will try to reproduce 🫡

[sethforprivacy]

Sorry this is a bit AI-driven, ran into this on one of our nodes during routine checks and wanted to raise it in case it was useful. Setup monitoring to see if I can catch it again but I can't find a way to reproduce it ATM.

[nahuhh]

Is this related to the missing tx (tx broadcasted but missing on the node, never mined, not in mempool) issue?

[sethforprivacy]

No, unrelated AFAICT.

[sethforprivacy]

Was able to hit it again on a prod node and grab a lot more data/logs, dropping now.

[sethforprivacy]

Is this related to the missing tx (tx broadcasted but missing on the node, never mined, not in mempool) issue?

It is related, actually, more details coming shortly from a full debug session.

[sethforprivacy]

Caught it again on a different node, and this time I had level-2 logging on the daemon while it was still in the broken state, as @selsta asked for. Two things came out of it that change the picture:

1. This failure silently stops the node broadcasting locally-submitted transactions, which is much worse than the peer-count symptom I originally reported.
2. There is a second, separable defect that makes those transactions permanently unrelayable — a restart does not recover them.

Also, @nahuhh — you were right and I was wrong to wave this off. This does look related to the "tx submitted but never appears / never mined" reports.

---

Environment

v0.18.5.1-release, Alpine/musl build, 2 vCPU, public node serving wallet clients (--public-node, restricted RPC on 18089, --rpc-max-connections=2000, no --tx-proxy/--anonymous-inbound, so a single public zone). Daemon up 4d13h, RestartCount=0. Nine sibling nodes on identical config and version were unaffected throughout, which is what makes the comparisons below meaningful.

Onset 2026-08-10 22:59–23:06 UTC, detected ~2.5 h later.

Symptom 1: locally-submitted txs enter the pool and are never broadcast

At detection, 59 of 87 pool entries had:

relayed:            false
last_relayed_time:  18446744073709551615   (UINT64_MAX)
do_not_relay:       false
fee:                > 0
relay method:       1  (local)

Diffing get_transaction_pool_hashes against four sibling nodes: 51 of 58 were held by no other node in the fleet. They never reached the network. Accumulation was ~23 tx/hour and monotonic (num_not_relayed 0 → 22 → 44 → 59 over three hours; 30 days of history on nine sibling nodes never exceeds 1).

fill_block_template also skips them, correctly, at tx_pool.cpp:1598 — relay_method::local doesn't match relay_category::legacy — so they're inert for mining as well as relay:

Considering <txid>, weight 1536, ..., relay method 1
  tx relay method is 1                       <- skipped
Considering <txid>, weight 2215, ..., relay method 4
  added, new block weight 2215/599400        <- fluff txs are fine

Everything arriving from peers as fluff (method 4) relayed and mined normally the whole time. The daemon was at chain tip, synchronized: true, status: OK, healthcheck green.

Root cause of symptom 1: dandelionpp_notify never runs

tx_pool.cpp::add_tx parks a new local/stem tx at last_relayed_time = UINT64_MAX and relies on set_relayed() to replace it with a real embargo time:

auto last_relayed_time = std::numeric_limits<decltype(meta.last_relayed_time)>::max();
if (tx_relay == relay_method::forward) { ... }
// else the `set_relayed` function will adjust the time accordingly later

For local/stem in a public zone, the only thing that calls set_relayed() is dandelionpp_notify::operator() in levin_notify.cpp, dispatched onto zone_->strand. Note it marks before sending, and its failure path also marks and falls back to fluff — so either branch executing would set relayed = true.

With net.p2p.tx at DEBUG, over a capture window in which 14 transactions entered the pool and num_not_relayed continued to climb, neither branch logged even once:

string	count
Sent N transaction(s) to <id> using Dandelion++ stem (MDEBUG, success)	0
Unable to send transaction(s) via Dandelion++ stem (MERROR, failure)	0
Unable to send transaction(s) to <zone> - no available outbound connections	0

So the handler is not executing at all — not failing. Contrast with the same node's earlier history, which does contain 11 Unable to send transaction(s) via Dandelion++ stem, all before the breakage minute.

Note that fluff txs bypass this: notify::send_txs calls on_transactions_relayed() synchronously for relay_method::fluff before dispatching fluff_notify to the same strand. That's why relayed=true still appeared on peer-received txs and masked the problem.

Symptom 2 (separable bug): a missed stem dispatch strands the tx permanently, across restarts

Once last_relayed_time == UINT64_MAX, get_relayable_transactions can never pick the tx up again, because get_relay_delay takes signed time_t:

uint64_t get_relay_delay(time_t last_relay, time_t received)
{
  time_t d = (last_relay - received + MIN_RELAY_TIME) / MIN_RELAY_TIME * MIN_RELAY_TIME;
  if (d > MAX_RELAY_TIME)
    d = MAX_RELAY_TIME;      // never taken: d is negative
  return d;                  // negative -> wraps to ~1.8e19 as uint64_t
}

UINT64_MAX as time_t is -1, so with received = 1786402000, MIN_RELAY_TIME = 300, MAX_RELAY_TIME = 14400:

d (signed)            = -1786401900        -> clamp not applied
returned as uint64_t  = 18446744071923149716
now - last_relayed_time (uint64 wrap) = 1786413001
skip condition: 1786413001 <= 18446744071923149716  -> true, forever

So the relay_method::local arm of get_relayable_transactions skips it on every pass for the full CRYPTONOTE_MEMPOOL_TX_LIVETIME (3 days).

Confirmed empirically: the sentinel is persisted in LMDB, so after restarting the daemon all 44 still-stranded txs reloaded with relayed=false, last_relayed_time=UINT64_MAX and stayed unrelayed while newly arriving txs relayed normally. I had to force them out one at a time with the relay_tx RPC; 32 then propagated fleet-wide, and the other 12 were correctly rejected as already-spent (those users had retried elsewhere in the meantime).

This part is arguably independent of idle_worker() and may deserve its own issue — any missed stem dispatch, for any reason, is unrecoverable. Clamping negative d to MIN_RELAY_TIME (or comparing before the unsigned conversion) would at least make the relay loop a real safety net.

The idle-chain death is partial this time

My original report assumed the whole chain dies and out decays to 0. Here it did not — outgoing_connections_count sat at 10 for hours, so an out == 0 alert never fired. Measured over 779 s, against nine unaffected siblings:

	broken node	siblings
half-open (before_handshake) conns	60, aged to 2.4 h, 0 reaped, ages tracking wall-clock exactly	0
new outbound conns created	0	—
grey peerlist	4955 → 4946 → 4939, monotonic	pinned 4999–5000
CLOSE_WAIT sockets	56	0
leaked socket fds (in fd/, absent from net/tcp{,6})	27	—

So connections_maker(), gray_peerlist_housekeeping() and check_incoming_connections() were all dead, while the daemon kept answering RPC, syncing and serving peers. A useful log tell: Failed to resolve ::ffff:<ip> from abstract_tcp_server2.inl:1886 (failed outbound connects) ran 12–19/h for four days and stopped dead at 23:05:56, never appearing again.

All of the above cleared immediately on restart: out 10 → 12, grey 4939 → 4999, half-open 60 → 0, CLOSE_WAIT 56 → 0.

Correlated trigger

Sustained Failed to do_send (levin_protocol_handler_async.h:662) began 23:03:22, four minutes before the last-ever outbound connect attempt, against an inbound peer roughly 639k blocks behind that was deep-syncing history from this node. It went from ~1 isolated occurrence per hour to a steady 2/min and never stopped; the paired Failed to send message, dropping it (line 749) changed character from ~1 per distinct peer (normal churn, present on all siblings) to ~25 repeats against a frozen set of 9 peers that were never dropped. This node is the fleet's heaviest uploader — 140 GB sent vs 36 GB received over 4.5 days.

do_send failing routes through wait_consume in abstract_tcp_server2.inl, which blocks the calling thread on a condition variable for a random 5000–6000 ms when a peer's send queue exceeds ABSTRACT_SERVER_SEND_QUE_MAX_COUNT (1000), then calls terminate_async() on timeout. Those failures were logged from five different P2P threads (P2P1/4/5/6/8). I want to be clear this is correlation, not a demonstrated mechanism — see below.

Ruled out

• Resource exhaustion — 444 fds of 65536, 1 GB RSS of 47 GB, CPU 11%.
• Thread loss or starvation — 29 threads, identical to siblings. Per-thread syscall distribution was indistinguishable from three healthy nodes (22–23 in futex_wait, 4–5 in epoll_pwait), so I could not substantiate the io-thread-starvation theory the wait_consume path suggests.
• A wedged strand — fluff_flush runs in the same zone_->strand and was clearly still executing.
• Anything remote/adversarial — the 54 stuck peers spanned ~50 distinct /16s with no clustering, and nine identically-configured nodes on the same public network were fine. Remote peers also can't influence which outbound peers this node dials.
• A stuck handler — two gdb samples ~25 s apart caught only transient work (one handle_request_get_objects → Blockchain::get_transactions_blobs → LMDB read inside a strand, gone by the next sample).

Suggested detection

num_not_relayed from get_transaction_pool_stats is a far sharper signal than peer counts for this failure: flat 0 (max 1) across 30 days on nine nodes, versus a clean monotonic ramp on the broken one, hours before any peer-count symptom appeared.

I have the full capture — level-2 log window, get_connections, pool metadata, socket tables, per-thread state, gdb backtraces. Happy to attach or send whatever is useful; I've held it back so far only because the pool dump contains user transaction blobs.

[selsta]

#10840 fixes part of what you are reporting, for the other things I would need all the logs you have to investigate further. I'll message you on Matrix.