tests: add patchbay tests (#3986)

## Description

Adds a first round of tests using our new network simulation framework,
[patchbay](https://github.com/n0-computer/patchbay). Patchbay uses Linux
network namespaces to create isolated network topologies with routers,
NATs, and link impairment. This lets us test actual holepunching and NAT
traversal in realistic conditions, without needing any external
infrastructure.

The test suite covers establishing a direct path in different scenarios:

- Holepunching: two devices behind destination-independent NATs connect
via relay, then upgrade to direct.
- Uplink switching: a device changes its network uplink (v4 and v6)
mid-connection and recovers a direct path.
- Interface changes: network interfaces are added or removed while
connected
- Link outage recovery: a device goes offline and comes back, direct
connection resumes
- Degraded links: increasing levels of impairment (latency, jitter,
packet loss) are applied to either side of a connection and holepunching
still succeeds

Some tests that iroh doesn't pass yet are included but marked
`#[ignore]`. They should be un-ignored as we improve things.

### Running the tests

On Linux, things just work with user namespace support:
```sh
cargo make patchbay
# expands to:
cargo nextest run -p iroh --features qlog --test patchbay --profile patchbay

# run a specific test with logs:
RUST_LOG=trace cargo make patchbay holepunch_simple --nocapture
```

Test output is saved in `./target/testdir-current/patchbay/<test-name>`.
Patchbay currently collects tracing logs for each device, qlog files (if
the feature is enabled), and endpoint metrics.
There's also a browser UI for viewing timelines, topologies, and logs.
The UI can also be used to compare different test runs.
```sh
cargo install --git https://github.com/n0-computer/patchbay patchbay-cli
patchbay serve --testdir --open
```

On macOS you'll need to run the patchbay tests in a VM or container. The
`patchbay` CLI includes a tool to set up a container or QEMU VM for
this.
```sh
# both commands default to the container backend on macOS and native on linux
patchbay test -p iroh --test patchbay
patchbay serve --testdir --open
# use --vm to force VM mode even on linux
```

See the [patchbay docs](https://n0-computer.github.io/patchbay/) and
[README](https://github.com/n0-computer/patchbay) for more details.

## Notes and open questions

There's a draft PR with some more tests: #4065, but they need more
thought and work to really cover what we want to test.

## Breaking Changes

None.

## Change checklist

- [x] Self-review.
- [x] Documentation updates following the [style
guide](https://rust-lang.github.io/rfcs/1574-more-api-documentation-conventions.html#appendix-a-full-conventions-text),
if relevant.
- [x] Tests if relevant.

Author: Frando

.config/nextest.toml

@@ -1,5 +1,5 @@
 [test-groups]
-run-in-isolation = { max-threads = 32 } 
+run-in-isolation = { max-threads = 32 }
 # these are tests that must not run with other tests concurrently. All tests in
 # this group can take up at most 32 threads among them, but each one requiring
 # 16 threads also. The effect should be that tests run isolated.
@@ -15,4 +15,12 @@ test-group = 'run-in-isolation'
 threads-required = 32
 
 [profile.default]
+default-filter = 'not binary(patchbay)'
 slow-timeout = { period = "10s", terminate-after = 3 }
+
+[profile.patchbay]
+fail-fast = false
+retries = 1
+test-threads = 4
+default-filter = 'binary(patchbay)'
+slow-timeout = { period = "30s", terminate-after = 4 }

.github/workflows/ci.yml

@@ -31,6 +31,8 @@ jobs:
     if: "github.event_name != 'pull_request' || ! contains(github.event.pull_request.labels.*.name, 'flaky-test')"
     timeout-minutes: 30
     runs-on: [self-hosted, linux, X64]
+    env:
+      RUSTFLAGS: "-Dwarnings --cfg skip_patchbay"
     strategy:
       fail-fast: false
       matrix:
@@ -135,6 +137,8 @@ jobs:
       matrix:
         target:
           - i686-unknown-linux-gnu
+    env:
+      RUSTFLAGS: "-Dwarnings --cfg skip_patchbay"
     steps:
       - name: Checkout
         uses: actions/checkout@v6

.github/workflows/patchbay.yml

@@ -0,0 +1,56 @@
+name: Patchbay Tests
+
+on:
+  pull_request:
+  push:
+    branches:
+      - main
+
+concurrency:
+  group: patchbay-${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
+  cancel-in-progress: true
+
+env:
+  RUST_BACKTRACE: 1
+  SCCACHE_CACHE_SIZE: "10G"
+  IROH_FORCE_STAGING_RELAYS: "1"
+  NEXTEST_VERSION: "0.9.80"
+
+jobs:
+  patchbay_tests:
+    name: Patchbay Tests
+    timeout-minutes: 15
+    runs-on: [self-hosted, linux, X64]
+    env:
+      RUSTC_WRAPPER: "sccache"
+    steps:
+      - name: Enable unprivileged user namespaces
+        run: sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0
+        continue-on-error: true
+
+      - uses: actions/checkout@v6
+      - uses: dtolnay/rust-toolchain@stable
+      - uses: mozilla-actions/sccache-action@v0.0.9
+      - name: Install cargo-make and cargo-nextest
+        uses: taiki-e/install-action@v2
+        with:
+          tool: nextest@${{ env.NEXTEST_VERSION }},cargo-make
+
+      - name: Build patchbay tests
+        run: cargo make patchbay --no-run
+
+      - name: Run patchbay tests
+        id: tests
+        run: cargo make patchbay
+        env:
+          RUST_LOG: ${{ runner.debug && 'trace' || 'debug' }}
+          PATCHBAY_LOG: trace
+
+      - name: Upload test results
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: patchbay-testdir-${{ github.sha }}
+          path: target/testdir-current/
+          retention-days: 7
+          if-no-files-found: ignore

.gitignore

@@ -1,3 +1,4 @@
 /target
 /logs
 iroh.config.toml
+/.patchbay