Merge pull request #178 from compoundingtech/fix/offline-peer-idle-cost

Prove bounded offline peer cost

Author: myobie

docs/fabric-sync-process-plan.md

@@ -415,13 +415,15 @@ Each implementation pull request merges when its required tests and CI pass.
 Silber.cos owns the step 7 activation gate and every release and deployment
 gate.
 
-Before any release, run every ignored test that needs a real machine. Record
-the machine, commit, command, and result.
+Before any release, run every named real-machine gate. Some gates are ignored
+tests, and some are matched live measurements. Record the machine, commit,
+command, and result.
 
 | Platform | Test | Exact proof |
 | --- | --- | --- |
 | macOS | `a_real_launchd_supervisor_rolls_back_and_removes_its_job` | An isolated real launchd job detects a deliberately broken pair. The pair-aware reader restores both members. The plist and loaded job disappear. |
 | Linux | `a_real_systemd_supervisor_rolls_back_and_removes_its_jobs` | An isolated real systemd timer starts its service over a deliberately broken pair. The pair-aware reader restores both members. Both units disappear. |
+| Linux | `offline_peer_cost_matches_control` | A configured peer that never answers does not move healthy-peer latency, CPU use, or resident memory outside a matched control window. |
 
 These tests use isolated paths and services. They do not prove the production
 service names, install paths, home permissions, or service definitions.
@@ -455,6 +457,24 @@ expected exactly one `fabric` member. It changed no executable or staging file.
 This clean refusal enforces the Release A order for that deployed reader. It is
 not a property of the Release A pair-aware reader.
 
+The first `offline_peer_cost_matches_control` gate passed on hetz at exact tag
+`v0.2.4+9b425d6` on 2026-09-05. The treatment added one peer that never answered
+and then restored the exact prior file digest. It sent 300 healthy-peer pings
+over 91.663 seconds. All pings passed, and none took more than one second.
+
+The matched resource traces each sampled the same daemon PID once per second
+for 379.095 seconds. Treatment used 5.925% of one core and had a 155,824 KiB RSS
+span. Control used 11.942% and had a 196,048 KiB span. Normal machine work was
+larger than the effect under test. Thus, the offline peer's cost was below this
+machine's noise floor. The daemon did not restart during the gate.
+
+For each release, run the treatment and control on the same Linux machine and
+deployed commit. Record CPU time, RSS minimum and maximum, and 300 healthy-peer
+ping times. Sample each resource window once per second for at least six
+minutes. Restore the exact prior `peers.toml` digest after treatment. The gate
+fails if treatment exceeds the control by more than the control's own 60-second
+variation. It also fails if a healthy ping fails or exceeds one second.
+
 Add each future real-machine test to this named list when the test is added.
 The measurement-only ignored tests are not release gates unless this list names
 them.

docs/failure-modes.md

@@ -3,11 +3,11 @@
 fabric connects two machines that are not on the same network. Networks fail, so
 this page is about what fabric does when they do.
 
-**Every number here was measured**, on one machine running two fabric daemons
-against each other, by the test named in the last column. Where there is no
-test, the row says `NOT PROVEN` and stays in the table. A page that lists only
-the failures we happened to test would read as a complete list of what can go
-wrong, and it would not be one.
+**Every number here was measured.** Most measurements use one machine running
+two fabric daemons, with the test named in the last column. A fleet measurement
+names its machine and window. Where there is no test, the row says `NOT PROVEN`
+and stays in the table. A page that lists only the failures we happened to test
+would read as a complete list of what can go wrong, and it would not be one.
 
 ## The two questions
 
@@ -33,6 +33,7 @@ nothing tells you to look.
 | **The far machine restarts** — you restart your dev server while a browser is connected | The open connection does not survive because the process that owned it is gone. A new request during the outage fails within Fabric's three-second initial-connect bound. A client can then retry. A new request works when the peer returns. See "Whose problem is a page that stops updating" below. | During the outage 3.006 s; after restart 91.681 ms; one 9.87 s focused run on 2026-09-02 | `a_peer_restarting_mid_session_restores_service_without_intervention` |
 | The direct path between the machines dies while a relay is available | `NOT PROVEN.` Two daemons on one machine cannot lose a direct path they never had, so this cannot be forced in a test here. It is not hypothetical: on the three-machine fleet today, 1,569 connections used a direct path and 1,463 used a relay, so both are in constant use. Proving the switch needs two real machines. | Unmeasured | `NOT PROVEN` |
 | A machine's address changes mid-session, as a laptop moving between networks does | The session survives without restarting the process, and the machine keeps its identity. Proven for one kind of tunnel. | Not separately measured | `generic_tunnel_survives_client_endpoint_recycle_without_process_restart`. **`NOT PROVEN` for TCP tunnels specifically.** |
+| A configured peer stays offline | Its failed connection attempt stays isolated. Healthy peer streams still open. Failed probes retain no connection. | Under 250 ms in the regression test. On hetz, 300 of 300 healthy pings passed over 91.663 seconds. | `offline_peer_cost_is_bounded_and_healthy_peer_stays_fast` |
 
 ## What you see while it is broken
 
@@ -79,6 +80,24 @@ network is back, for the same reason, while a new request is immediate.
 application notices is up to the application; fabric restores the tunnel but
 cannot resurrect a socket the far process no longer has.
 
+**An offline peer still gets a health probe every 20 seconds.** This is extra
+work, but its fleet cost was not detectable on 2026-09-05. A 91.663-second
+treatment had six failed probes. All 300 healthy-peer pings passed, with no ping
+above one second.
+
+Two matched resource traces each used 380 one-second samples over 379.095
+seconds. The offline-peer treatment used 5.925% of one core. The no-offline-peer
+control used 11.942%, because unrelated work made the control busier. Treatment
+RSS spanned 155,824 KiB. Control RSS spanned 196,048 KiB. Both traces crossed the
+daemon's 128 MiB allocator sawtooth. These results show no attributable cost at
+this fleet size. They do not show that a failed probe costs nothing.
+
+Remove a truly retired peer from `peers.toml` on every machine. This file is a
+local allow list, so removal on one machine does not remove trust elsewhere.
+`fabric doctor` can report an unreachable peer, but it cannot know that the peer
+was retired. The fleet has no authoritative peer set today. An operator must
+compare every machine's `peers.toml` to find this drift.
+
 ## Why `send-file` is not shaped like scp
 
 `scp` lets the sender choose where a file lands on the far machine. **fabric

src/mux.rs

@@ -1280,6 +1280,119 @@ mod tests {
         Ok(())
     }
 
+    /// A peer that never answers must keep a fixed amount of manager state and
+    /// must not delay a healthy peer while 32 repeated probes fail.
+    ///
+    /// The live control on hetz used 4.81 CPU-seconds over 93.339 seconds. A
+    /// 342.087-second window with one offline peer used 17.09 CPU-seconds, or
+    /// 4.996% of one core against the control's 5.153%. RSS followed the same
+    /// bounded 128 MiB allocator sawtooth with and without the offline peer.
+    /// This test pins the deterministic causes behind that result: each probe
+    /// owns one bounded attempt, failed probes retain no connection, and a
+    /// healthy peer never waits for the offline peer's gate.
+    #[tokio::test(flavor = "multi_thread", worker_threads = 4)]
+    async fn offline_peer_cost_is_bounded_and_healthy_peer_stays_fast() -> Result<()> {
+        const FAILED_PROBES: usize = 32;
+        const FAILED_PROBE_WINDOW: Duration = Duration::from_millis(350);
+        const HEALTHY_STREAM_BUDGET: Duration = Duration::from_millis(250);
+
+        let client = Endpoint::builder(presets::N0).bind().await?;
+        let (opened_tx, _opened_rx) = mpsc::unbounded_channel();
+        let manager = Arc::new(PeerConnections::new(client.id(), opened_tx));
+        let healthy = Router::builder(
+            Endpoint::builder(presets::N0)
+                .alpns(vec![MUX_ALPN.to_vec()])
+                .bind()
+                .await?,
+        )
+        .accept(
+            MUX_ALPN,
+            MuxEcho {
+                connections: Arc::new(AtomicUsize::new(0)),
+                headers: Arc::new(Mutex::new(Vec::new())),
+            },
+        )
+        .spawn();
+        healthy.endpoint().online().await;
+        let healthy_addr = healthy.endpoint().addr();
+        let offline_addr = EndpointAddr::new(iroh::SecretKey::generate().public());
+
+        let mut first = manager
+            .open_mux_stream(&client, 0, &healthy_addr, "control", StreamActivity::Probe)
+            .await?;
+        prove_measurement_stream(&mut first).await?;
+        drop(first);
+        let attempts_before = manager.mux_connect_attempts.load(Ordering::SeqCst);
+
+        for probe in 0..FAILED_PROBES {
+            let offline_manager = manager.clone();
+            let offline_client = client.clone();
+            let offline_addr = offline_addr.clone();
+            let offline = tokio::spawn(async move {
+                tokio::time::timeout(
+                    FAILED_PROBE_WINDOW,
+                    offline_manager.open_mux_stream(
+                        &offline_client,
+                        0,
+                        &offline_addr,
+                        "offline",
+                        StreamActivity::Probe,
+                    ),
+                )
+                .await
+            });
+
+            let expected_attempts = attempts_before + probe + 1;
+            tokio::time::timeout(Duration::from_secs(1), async {
+                while manager.mux_connect_attempts.load(Ordering::SeqCst) < expected_attempts {
+                    tokio::task::yield_now().await;
+                }
+            })
+            .await
+            .with_context(|| format!("offline probe {probe} never started"))?;
+
+            let mut stream = tokio::time::timeout(
+                HEALTHY_STREAM_BUDGET,
+                manager.open_mux_stream(
+                    &client,
+                    0,
+                    &healthy_addr,
+                    "healthy",
+                    StreamActivity::Probe,
+                ),
+            )
+            .await
+            .with_context(|| {
+                format!("offline probe {probe} delayed the healthy peer past the budget")
+            })??;
+            prove_measurement_stream(&mut stream).await?;
+            drop(stream);
+
+            let offline = offline.await?;
+            assert!(
+                !matches!(offline, Ok(Ok(_))),
+                "the offline peer unexpectedly accepted probe {probe}"
+            );
+        }
+
+        assert_eq!(
+            manager.mux_connect_attempts.load(Ordering::SeqCst) - attempts_before,
+            FAILED_PROBES,
+            "a probe must own one connection attempt and no hidden retry task"
+        );
+        assert_eq!(
+            manager.conns.lock().await.len(),
+            1,
+            "failed probes must retain no connection beside the healthy one"
+        );
+        assert!(manager.legacy_notices.lock().await.is_empty());
+        assert!(manager.legacy_fallbacks.lock().await.is_empty());
+
+        healthy.shutdown().await?;
+        client.close().await;
+        Ok(())
+    }
+
     async fn assert_stream_failure_recovery(
         failures: usize,
         expect_same_connection: bool,