Desktop x0xd leaks ~1.4-1.6 GB/h RSS; terminal state is host-wide UDP buffer exhaustion = the #312 'studio1 black-hole'

[dirvine]

Live capture 2026-08-20, tic-tac-toe v0.5.2 sidecars (x0xd 0.37.4), macOS

Two machines, same artifact:

host	uptime	RSS	rate
studio1 (M3 Ultra) pid 694 (--name ttt)	3d21h	142.6 GB	~1.5 GB/h
studio1 pid 746 (--name managed-…)	3d21h	122.6 GB	~1.3 GB/h
macbook (--name ttt)	7h24m	11.9 GB	~1.6 GB/h

Rate is ~100× the #288 VPS figure (10–15 MB/h) — desktop/ttt workload specific (mDNS? presence? LAN churn?), consistent across 3 instances on 2 machines.

vmmap (pid 694)

• DefaultMallocZone: 278.2 GB VIRT / 139.1 GB RES / 42.2 GB dirty / 2.6 GB swapped
• 20,740,536 live allocations, only 15.4 GB allocated — millions of small objects
• 29.4 GB heap fragmentation (66%)

Terminal state = the #312 black-hole

studio1 kernel: 4,173,337 UDP datagrams "dropped due to full socket buffers". Starved recv loops stop draining the QUIC socket → inbound UDP dies host-wide while outbound still works. Observed live: durable DM A→B times out (24 s), B→A dies waiting for receive-ACK on a conn B believes is live; raw nc probes prove transit A→studio1 is fine ("no socket" counter +50/50) while x0xd-bound traffic vanishes. #312 is plausibly not environmental — it is this leak's end state.

Correlated unbounded structure

/diagnostics/gossip pubsub_stages.peer_scores_v2 holds 10,412 peers (peer_scores 4,107) on a box with ~20 live conns; macbook already 3,407 after 7 h. Churned peers appear never pruned. JSON size of the maps (~5 MB) doesn't itself explain 15 GB, so the alloc bulk likely sits behind what the snapshot summarizes.

Evidence bundle: studio1:~/x0x-leak-evidence-20260820/ (vmmap-694.txt, diag-{gossip,dm,groups,history}.json).

Related: #288 (slow VPS variant), #312 (terminal symptom), #262 (driver starvation class).

🤖 Generated with Claude Code

[dirvine]

Control experiment 2026-08-20 17:26–19:31 — the app is exonerated; v0.39.0 still leaks

Bare x0xd 0.39.0 (official release binary, fresh data dir, no API clients at all, default bootstrap → prod mesh, on the same LAN as 2 other daemons) vs the ttt-attached 0.37.4 daemon, RSS sampled every 5 min for 2h05:

daemon	start	end	rate
control (0.39.0, no app)	205 MB	2.27 GB	~1.0 GB/h
ttt sidecar (0.37.4 + app)	2.50 GB	4.69 GB	~1.05 GB/h

Identical slopes ⇒ the desktop app workload is not the driver, and the leak is alive in v0.39.0. Same-age VPS daemons (0.39.0, post-convergence restart) sit at ~250–300 MB. The environmental difference is the desktop LAN: multiple x0xd instances discovering each other (ant-quic first-party mDNS, LAN peer churn) and/or the macOS allocator (vmmap earlier: 20.7M live allocations, 66% frag).

Control's early slope was steeper (~2 GB/h first 40 min) then settled ~0.5–0.6 GB/h — warm-up plus steady leak.

Next: dhat-instrumented bare run on this LAN (in progress) — no synthetic load needed; 30–45 min should capture the allocation stacks. Full CSV in session scratchpad (leakctl/rss.csv).

🤖 Generated with Claude Code

[dirvine]

Refined model (20:40Z): leak scales with NAT-traversal exposure, not LAN or app

daemon (all 0.39.0 unless noted)	environment	rate
VPS fleet (public IP)	prod mesh	~40–75 MB/h (250 → 375–555 MB over 3.3h; 3–5× the old #288 figure)
macbook LAN-only (--no-hard-coded-bootstrap, mDNS+exchange, ~12 peers)	NAT'd, few remote peers	0.65 → 0.17 GB/h, plateauing (395 → 774 MB / 55 min)
macbook control (prod bootstrap, no app)	NAT'd, full mesh	~0.95 GB/h, still linear (3.1 GB @ 3h17)
ttt sidecars (0.37.4 + app)	NAT'd, full mesh	~1.0–1.5 GB/h

Same leak everywhere; amplification tracks how much per-remote-peer connectivity work the node does. A NAT'd client continuously runs hole-punch/path-validation/candidate/relay machinery against every mesh peer; a public VPS almost never does. LAN-only plateaus because its remote-peer set is small and stabilises.

Leading hypothesis: NAT-traversal attempt state retained per peer/attempt on the ant-quic side (candidate pairs, path-validation or relay sessions, observed-address/bootstrap-cache entries) — cf. ant-quic #210 (failed sessions pinning Connection handles). Corroborating: shutdown logs from the dhat run show IHAVE 'node not initialized' send-failure spam at ~10 Hz to ~10 LAN peers — send-path failures against peers whose connection state is half-dead.

dhat stacks (in progress) should be read with this lens: look for ant_quic::nat_traversal, connection_establishment, candidate/path-validation, relay, and bootstrap-cache allocations first; gossip peer_scores second.

🤖 Generated with Claude Code

[dirvine]

Allocation profile — desktop macOS, bare daemon (no app load)

Setup: v0.39.0 release build, named instance leakprobe, --api-port 12799 --no-hard-coded-bootstrap, MallocStackLogging=1, mDNS active (~4–10 LAN peers discovered over the run). No synthetic load (matches @claude's bare-control slope). Captured at 57 min uptime via malloc_history -allBySize / leaks / sample (dhat capture is impossible on this build — see shutdown note below).

RSS curve (plain build): 301 MB @1 min → 506 MB @16 min → 592 MB @31 min → 661 MB @46 min → 1.27 GB @57 min (~1.3 GB/h, bare). Live heap at capture: 571,247 blocks / 1.05 GB across 2,320 stack groups.

Top live-allocation stacks (by bytes)

Bytes	Calls	Stack (leaf ← root, abridged)
262 MB (+71 +43 +14 +10 ≈ 400 MB total)	~21k	ant_quic streams::send::ByteSlice::pop_chunk ← Send::write ← SendStream::write ← NetworkNode::send_to_peer ← **saorsa_gossip_pubsub::PlumtreePubSub::send_to_peer_with_timeout / parallel_send_to_peers**
113 MB (+86 +60 +46 +46 +38 ≈ 390 MB total)	~380k	bytes::Bytes::copy_from_slice ← ant_quic **connection::assembler::Assembler::insert** ← StreamsState::received ← Connection::process_payload ← process_decrypted_packet
36.8 MB	36	tokio blocking-pool thread stacks (baseline, not leak)
13.8 MB (+4.8 +3.8)	49+	saorsa_gossip_pubsub::TopicState::with_cache_config ← PlumtreePubSub publish/record_verified_inbound_from_peer (per-topic message cache)
11.9 MB	824	Bytes::copy_from_slice ← serde_core Deserialize ← PlumtreePubSub (inbound gossip frames)

leaks proper: 160 root leaks / 48 MB, largest being TopicState::with_cache_config ×29 — the remaining ~1 GB is retained (reachable) memory, i.e. unbounded growth, not orphaned blocks.

Reading

The two dominant structures are per-peer QUIC stream buffers in both directions, fed by the Plumtree fan-out path: outbound SendStream::write buffers from send_to_peer_with_timeout/IHAVE flusher that never drain, and inbound Assembler::insert copies from process_payload that never get consumed/released. This matches the NAT-exposure correlation in the VPS/LAN/NAT datapoints: a peer whose path is lossy or half-established (NAT traversal, path validation failing) stops draining its streams; the connection is never torn down, so every retry (and every IHAVE/GRAFT volley) both (a) appends to the stuck connection's send buffers and (b) re-inserts received fragments into assembler queues. The existing "Peer cooled after repeated PubSub send timeouts" cooldown (30–60s, observed in logs) throttles new sends but does not drop or reset the cooled connection, so its accumulated buffers persist forever.

Secondary: sg TopicState message caches (49 topics, 13.8 MB + 48 MB leaked roots on this LAN) — real but an order of magnitude smaller than the stream buffers.

Shutdown livelock (blocks dhat, separate bug)

SIGTERM does not terminate the daemon: sample during shutdown shows all 16 tokio workers pegged 100% in saorsa_gossip_pubsub::PlumtreePubSub::spawn_ihave_flusher → ShardedTopicMap::write_all allocating continuously ("IHAVE per-peer send failed: node not initialized" WARN at ~10 Hz per peer; 10.4 MB of WARN logs; RSS grew 661 MB → 1.46 GB during the 13+ min after SIGTERM before I had to SIGKILL). Two consecutive runs, same shape. The IHAVE flusher never observes shutdown and its failed sends keep it looping.

Fix proposal (one paragraph)

Bound the per-peer gossip egress and expire stuck connections: (1) in saorsa-gossip-pubsub, make the send-cooldown destructive — after N consecutive send_to_peer timeouts, drop the peer's connections (and their send buffers) so ant-quic rebuilds a clean path instead of cooling-and-retaining; (2) in ant-quic, cap per-stream send-buffer bytes (SendStream::write backpressure → drop policy for unreliable fan-out traffic) and bound/reap Assembler queues for streams with no consumer — both accumulation sites above grow only because nothing ever enforces a limit; (3) cap sg TopicState message caches by bytes/age (config exists but is effectively unbounded here). (1) is the highest-leverage and smallest change; (2) is the durable fix but lives in ant-quic internals. Independently of the leak, the IHAVE flusher must select on the shutdown token — that's the shutdown hang.

Artifacts kept locally (/tmp/x0x-leakprobe/: malloc-allbysize.txt 17 MB, malloc-calltree.txt, leaks.txt, steady-state.sample, shutdown-storm.sample) — happy to attach trimmed versions or a memgraph on request.

[dirvine]

Design decided (.planning/design-368.md): PR A in x0x — reset timed-out SendStreams explicitly, escalate cooldown to a peer-id-keyed connection close, IHAVE flusher honours the shutdown token (also fixes #371), proof counters in /diagnostics/gossip, black-holed-peer integration test, 60-min soak gate (< 100 MB/h). Follow-up defences filed upstream: ant-quic #255 (per-connection buffer cap + assembler reaping), saorsa-gossip #41 (TopicState / peer_scores bounds).

[dirvine]

Soak-2 (PR A, destructive cooldown, final semantics): merge gate FAILED — negative result

Fresh x0xd from fix/368-destructive-cooldown @45d36a1 (timeout→Err, close after 5 consecutive timeouts, 60 s/peer close limiter), prod bootstrap, no app, 2-min independent samples:

elapsed	RSS
5 m	824 MB
11 m	1376 MB
15 m	1607 MB
23 m	1787 MB

Late slope ~1.35 GB/h (gate was < 100 MB/h); early slope ~4 GB/h, worse than the unpatched ~2 GB/h warm-up. Counters at 23 min: conns_closed_by_cooldown = 166, streams_reset_on_timeout = 3711.

Interpretation: a fixed 1200 ms transport timeout sits under saorsa-gossip's adaptive per-peer timeout (floor 1500 ms, grows under load), so on a NAT'd node with real WAN RTT and relay/hole-punch paths most 'stalls' are false. Closing connections on them (~7/min) drives redial → handshake → NAT-traversal churn whose per-attempt state allocates (ant-quic #210 class). We traded pinned stream buffers for connection churn and lost. Earlier soak-1 (pre-correction, timeout reported as Ok) was worse still at 1.9 GB/h — it blinded sg's cooldown.

Mechanism stands (buffers pinned on stalled streams); remedy changes: v2 = per-peer in-flight send cap (semaphore, K=8; saturated peer ⇒ immediate Err without opening a stream; no fixed timeout; close only after 60 s of continuous saturation, rate-limited). Bounds outbound pinning to K×msg per peer with zero churn and lets sg's cooldown throttle. Inbound Assembler pinning remains ant-quic #255's to solve, so a ~50% slope reduction with closes ≈ 0 is the realistic PR-A pass bar.

🤖 Generated with Claude Code

[dirvine]

Soak results — destructive cooldown (PR A v1) is a NEGATIVE result, design superseded

Two 60-minute-class bare soaks on this Mac (prod bootstrap, mDNS LAN, named instances), both worse than the ~1.3 GB/h baseline profile. Full data + the v2 design rationale below.

Soak 1 — pre-review semantics (2-consecutive-timeout close, timed-out send reported Ok)

elapsed	RSS
0:44	221 MB
5:44	793 MB
16:14	1,343 MB
1:01:15	2,145 MB

≈ 1.9 GB/h — worse than baseline. Reading: reporting Ok on a timed-out send blinded saorsa-gossip's own failure accounting, so nothing throttled the fan-out to stalled peers.

Soak 2 — reviewed semantics (Err on timeout, N=5 consecutive, close rate-limited 1/peer/60 s)

elapsed	RSS	conns_closed_by_cooldown	streams_reset_on_timeout
1:08	319 MB	1	41
5:58	883 MB	32	1,309
10:48	1,388 MB	77	1,996
15:38	1,656 MB	106	2,615
20:28	1,727 MB	138	3,126
25:18	1,949 MB	166	3,731
30:08	2,027 MB	200	4,207

Early slope ~4 GB/h, late ~1.35 GB/h — merge gate failed. ≈7 closes/min, ≈2.7 timeouts/s.

Why (mechanism reading, shared with @claude's independent samples)

A fixed 1,200 ms transport timeout undercuts saorsa-gossip's ADAPTIVE per-peer timeout (floor 1,500 ms, grows under load) — on a NAT'd node with real WAN RTT plus relay/hole-punch paths, most "timeouts" are false stalls. Closing on them creates redial → handshake → NAT-traversal churn whose per-attempt state allocates more than the teardown frees (the ant-quic #210 class): we traded pinned stream buffers for connection churn and got faster growth.

The mechanism from the heap profile (buffers pinned on stalled streams) stands; the remedy was wrong. v2 (in flight): stop OPENING new streams to a saturated peer instead of tearing the peer down — per-peer in-flight send cap (K=8, try_acquire, Err without opening a stream), no fixed transport timeout (sg's adaptive outer timeout governs again), connection close only for a peer saturated >60 s continuously, rate-limited 1/peer/5 min, counters renamed (sends_rejected_saturated, peers_saturated_now, conns_closed_dead_peer). Inbound Assembler pinning remains ant-quic #255 territory — the v2 soak gate accepts an outbound-only partial win (slope ≈ half of baseline with near-zero closes) and escalates #255.

Artifacts: curves above; soak logs in /tmp/x0x-leaksoak*/.

[dirvine]

Measurement (gate 1, bare 0.39.0+/diagnostics/transport, 30 min, 2-min sampling)

t	RSS	ant-quic active	x0x peers	successful_connections (cumulative)
04:40	321 MB	12	12	12
04:48	482 MB	16	16	27
05:00	529 MB	17	17	47
05:10	564 MB	17	17	59

failed_connections = 0 throughout; final 582 MB @ 34:49.

Findings. (1) No zombie accumulation: ant-quic active_connections ≈ x0x peer count at every sample. (2) Heavy replacement churn: 47 new connections in 30 min against a flat ~16 active — average connection lifetime ≈ 40 s on a stable peer set. (3) RSS growth tracks successful_connections, not active: ≈ 5.5 MB of residue per replaced connection.

Premises retired by data this session: app workload (control run), destructive cooldown on send timeout (soak-2: churn made it worse), per-peer in-flight cap (sg serialises per-peer sends; ant-quic ≥0.27.39 already resets abandoned streams on drop), zombie connections (this run). Per-connection retention is bounded by the fork's 16 MiB receive / 8×RWND send windows, so the leak is churn × per-close residue.

Next (gate 2, measurement only): add proto-level Endpoint::open_connections() to the endpoint (residue = retained closed connections?), and count the churn driver — x0x-initiated redials of already-connected peers vs inbound replacements (superseded connection reaped events) — before designing either fix.

🤖 Generated with Claude Code