Paths continually opened/abandoned.

[NfNitLoop]

Describe the bug

Iroh continues to try to open extra paths for a connection, though the application keeps abandoning them.

Original report from Discord:

Playing around with Iroh and reading debug logs. I'm wondering what might be causing this in debug logs. It seems to be using Path(1), but keeps trying to open to more paths (one v4, one v6) and they get abandoned with messages like this:

iroh::_events::path::abandoned: remote=... conn_id=... path_id=... network_path=Ip(...) reason=ApplicationClosed { error_code: 62 }

[...]
(Connections seem to be working fine, this is just me trying to understand multipath and learn more.)
[...]
FWIW I’m seeing the behavior when I’m running behind a NAT, connecting to another service behind the [same] NAT. No UPnP support.
Using only n0 discovery. (Plan to add mdns support but haven’t enabled yet.)

Relevant Logs

client.txt

server.txt

Expected behavior

I'm running a custom ping command that sends 10 pings, waiting 1 second between each. The pings all happen on the same Connection, so I'd expect paths to stabilize and get re-used. (I only open a new bidi stream for each ping.)

Plus, @flub on Discord said this might be a bug and asked me to submit these traces. :)

Iroh

Version:

iroh v1.0.0
├── iroh feature "default"
│   └── st v0.1.0 (/home/codyc/code/iroh/st/crates/st)
│       └── st feature "default" (command-line)
├── iroh feature "fast-apple-datapath"
│   └── iroh feature "default" (*)
├── iroh feature "metrics"
│   └── iroh feature "default" (*)
├── iroh feature "portmapper"
│   └── iroh feature "default" (*)
└── iroh feature "tls-ring"
    └── iroh feature "default" (*)

Endpoint configuration:

Server:

    let ep = Endpoint::builder(presets::N0)
        .secret_key(conf.secret_key)
        .alpns(
            vec![
                " ", "_", "h3",
            ]
            .into_iter()
            .map(|it| it.to_owned().into_bytes())
            .collect(),
        )
        .bind()
        .await?;

Client:

    let ep = Endpoint::builder(presets::N0)
        .secret_key(secret_key)
        .bind()
        .await?;

Platform(s)
Desktop: Linux ludilo 7.0.14-arch1-1 #1 SMP PREEMPT_DYNAMIC Sat, 27 Jun 2026 16:15:10 +0000 x86_64 GNU/Linux

• This issue was created by a human that thought critically about
  the issue being reported and wrote this as concisely and clearly
  as they could. Taking full responsibility for the issue being
  accurate.
• This issue isn't slop, and is backed up by evidence.

[NfNitLoop]

I've been looking into this a bit myself today, out of curiosity.

apply_selected_path() always closes all but one "IP path":

iroh/iroh/src/socket/remote_map/remote_state.rs

Lines 675 to 682 in 3e1e4b7

	/// Propagates a change of [`State::selected_path`] to noq.
	///
	/// Iterates over all connections and applies the selected path as follows:
	/// - Closes non-selected IP paths (but keeps one IP path open still)
	/// - Sets all non-selected paths to [`PathStatus::Backup`]
	/// - Opens the selected path if it does not exist on the connection
	/// - Sets the selected path to [`PathStatus::Available`]
	fn apply_selected_path(&mut self) {

And I bet something is just re-probing (re-opening) paths every so often to check if a better path has become available? (Digging some more...)

It seems like:

• iroh calls its do_holepunching()
• which calls noq's initiate_nat_traversal_round().
• That fires off NAT traversal packets
• When noq gets back those responses, it opens paths for them
• Which then calls apply_selected_path() to close them.

The first do_holepunching() call (round=1) is triggerd by handle_message, when the RemoteStageMessage::AddConnection. This makes sense when setting up a new connection. 👍

The second (round=2) is triggered within the same second, because: "remote addrs updated, triggering holepunching". This seems valid, since the remote_candidates inside the last HolepunchAttempt were empty. 👍

The third (round=3) do_holepunching() is triggered 5 seconds later due to "scheduled holepunching", which is throttled to 5 seconds:

iroh/iroh/src/socket/remote_map/remote_state.rs

Lines 47 to 51 in 3e1e4b7

	/// How often to attempt holepunching.
	///
	/// If there have been no changes to the NAT address candidates, holepunching will not be
	/// attempted more frequently than at this interval.
	const HOLEPUNCH_ATTEMPTS_INTERVAL: Duration = Duration::from_secs(5);

That third round seems to have been scheduled because there's a bit of a race condition with the event handling in handle_path_resopnse.

When we kicked off round=2 of holepunching, all remote_candidates were already known:

[2026-07-06 10:53:37] TRACE endpoint{id=b055e1f18b}:RemoteStateActor{remote=1a7eb57c9c}: iroh::socket::remote_map::remote_state: remote addrs updated, triggering holepunching id=ConnId(94871686634512) evt=Ok(AddressAdded([2607:fb92:2207:4208:ebf9:c018:87a7:3b6f]:54720))
[2026-07-06 10:53:37] TRACE endpoint{id=b055e1f18b}:RemoteStateActor{remote=1a7eb57c9c}: iroh::socket::remote_map::remote_state: candidates to holepunch? last_hp=HolepunchAttempt { when: Instant { tv_sec: 2031, tv_nsec: 475225298 }, local_candidates: {10.1.1.104:54720, 172.56.243.59:45127, [2607:fb92:2207:4208::2c7]:33860, [2607:fb92:2207:4208:ebf9:c018:87a7:3b6f]:33860}, remote_candidates: {} } local_candidates={10.1.1.104:54720, 172.56.243.59:45127, [2607:fb92:2207:4208::2c7]:33860, [2607:fb92:2207:4208:ebf9:c018:87a7:3b6f]:33860} remote_candidates={10.1.1.104:49637, 172.56.243.59:62251, [2607:fb92:2207:4208::2c7]:54720, [2607:fb92:2207:4208:ebf9:c018:87a7:3b6f]:54720}

But immediately after that, another AddressAdded event is received, which causes the scheduling of round 3, even though that address is already being holepunched:

[2026-07-06 10:53:37] TRACE endpoint{id=b055e1f18b}:RemoteStateActor{remote=1a7eb57c9c}: iroh::socket::remote_map::remote_state: remote addrs updated, triggering holepunching id=ConnId(94871686634512) evt=Ok(AddressAdded([2607:fb92:2207:4208::2c7]:54720))
[...]
[2026-07-06 10:53:37] TRACE endpoint{id=b055e1f18b}:RemoteStateActor{remote=1a7eb57c9c}: iroh::socket::remote_map::remote_state: candidates to holepunch? last_hp=HolepunchAttempt { when: Instant { tv_sec: 2031, tv_nsec: 629213769 }, local_candidates: {10.1.1.104:54720, 172.56.243.59:45127, [2607:fb92:2207:4208::2c7]:33860, [2607:fb92:2207:4208:ebf9:c018:87a7:3b6f]:33860}, remote_candidates: {10.1.1.104:49637, 172.56.243.
59:62251, [2607:fb92:2207:4208::2c7]:54720, [2607:fb92:2207:4208:ebf9:c018:87a7:3b6f]:54720} } local_candidates={10.1.1.104:54720, 172.56.243.59:45127, [2607:fb92:2207:4208::2c7]:33860, [2607:fb92:2207:4208:ebf9:c018:87a7:3b6f]:33860} remote_candidates={10.1.1.104:49637, 172.56.243.59:62251, [2607:fb92:2207:4208::2c7]:54720, [2607:fb92:2207:4208:ebf9:c018:87a7:3b6f]:54720}
[2026-07-06 10:53:37] TRACE connect{me=b055e1f18b alpn=  remote=1a7eb57c9c}:drive{id=8}:pkt{path_id=0}: noq_proto::connection: got Data packet (1200 bytes) from (local: 2607:fb92:2207:4208:ebf9:c018:87a7:3b6f, remote: [2607:fb92:2207:4208:ebf9:c018:87a7:3b6f]:54720) using id 0d9f8af9fb5ad564
[2026-07-06 10:53:37] TRACE endpoint{id=b055e1f18b}:RemoteStateActor{remote=1a7eb57c9c}: iroh::socket::remote_map::remote_state: not holepunching: no new addresses scheduled_in=4.99835637s

So round 3 seems unnecessary. Maybe the fix is to just not schedule a holepunch in that case?

iroh/iroh/src/socket/remote_map/remote_state.rs

Lines 561 to 563 in 3e1e4b7

	trace!(scheduled_in = ?(next_hp - now), "not holepunching: no new addresses");
	self.state.scheduled_holepunch = Some(next_hp);
	return;

All that said, I've learned a couple things that probably lower the priority of this issue:

• Iroh will try to find a better route every 60 seconds, via check_connections(), which it does by calling trigger_holepunching(). Combined with the fact that extra IP Paths are closed, it seems expected that we'll have paths opening and closing every 60 seconds at least.
• After the three rounds of holepunching during connection setup, I don't see any more holepunching being performed/scheduled, and so no more paths are opened/closed in the short term. So this is an O(1) problem instead of O(n).

FWIW, a couple few things that were confusing in this investigation:

• In my mind, "hole punching" has a specific meaning, but in iroh/noq, it seems to ... just be the way you open up new Paths? Even if they don't actually need NAT hole punching?
  (response from Discord) "turns out that doing holepunching on a path that doesn't need holepunchingt is entirely harmless. it just immediately validates the path and you can start using it. so there's no reason to try and figure out if a path needs holepunching or not"
• I naively assumed that "multi-pathing" meant that multiple direct connections would remain open, and traffic could be multiplexed between them. But it seems to mean that one direct ("IP") connection and one relay connection will remain open?
  (response from Discord) "current impl tries to send only traffic over one path at a time. we will at some point work on transmitting over multiple paths concurrently, but that's for later. and the gains of that are only in very specific situations. so likely this would never(?) be switched on automatically"
• The debug/trace messages about the path being closed have an event Abandoned { reason: ApplicationClosed { error_code: 62 } } which makes it seem like there's some error preventing the path from functioning. And it wasn't clear that the client itself was responsible for closing the path!

[flub]

Can you describe the network setup and devices that trigger this behaviour a bit more please?

[NfNitLoop]

Can you describe the network setup and devices that trigger this behaviour a bit more please?

Both client and server are running on the same machine, an Arch Linux desktop at my house. The local network is dual-stack IPv4/6, with (double-NATted 😭) private IPv4 in the 10.0.0.x/24 network, and publicly routable IPv6 addresses. (Though, my ISP, T-Mobile, does not allow inbound IPv6 packets unless they are for an established connection.)

So, the device-to-cloud path is:

• Linux desktop, wifi connection.
• Unifi Dream Machine router, (WAP LAN inside, wired ethernet outside)
• T-Mobile consumer 5G Wireless Internet.
• ???
• The Internet.

I can see that sometimes connection setup ends up choosing the 10.x.x.x IPv4 address for the connection (PathId=1), and sometimes it chooses one of the v6 addresses. I assume that's just whichever packet wins the race.

Anything else that would be helpful to know?