Slow validate clusters with big OpenShift clusters

[nirs]

Description

validate clusters takes ~348 seconds (nearly 6 minutes) when gathering from large OpenShift clusters over WAN. The bottleneck is log streaming over high-latency network, with secondary impact from hundreds of empty list API calls.

Using --remote gather (as in #407) should reduce this to ~30–50 seconds.

Example — validate clusters with high-latency network

% du -sh out/clusters/validate-clusters.data
282M    out/clusters/validate-clusters.data

% du -sh out/clusters/validate-clusters.data/*
 99M    out/clusters/validate-clusters.data/arft-site02
 91M    out/clusters/validate-clusters.data/arft-site03
 92M    out/clusters/validate-clusters.data/hub

Total wall time 348 s, almost all of it spent gathering data (345.8 s):

Wall (s)	Step
348.20	Total
345.80	gather
344.40	-- gather "arft-site02"
322.90	-- gather "hub"
125.40	-- gather "arft-site03"
0.95	check S3 profiles
0.05	validate clusters data
2.50	validate config

From the log — clusters gathered in parallel, wall time = slowest cluster:

2026-07-29T17:43:21.294  Gathering from clusters ["hub" "arft-site02" "arft-site03"]
2026-07-29T17:43:21.297  Gathering from cluster "arft-site03"
2026-07-29T17:43:21.297  Gathering from cluster "hub"
2026-07-29T17:43:21.297  Gathering from cluster "arft-site02"
2026-07-29T17:45:26.659  Gathered 6865 resources from cluster "arft-site03" in 125.36 seconds
2026-07-29T17:48:44.192  Gathered 4182 resources from cluster "hub" in 322.88 seconds
2026-07-29T17:49:05.668  Gathered 5206 resources from cluster "arft-site02" in 344.35 seconds

Per-cluster breakdown

Cluster	Resources	Data	Wall time
arft-site02	5,206	99 MiB	344.4 s
hub	4,182	92 MiB	322.9 s
arft-site03	6,865	91 MiB	125.4 s

arft-site03 has more resources yet finishes 2.7× faster — the difference is log streaming performance.

Why local gather was slow

1. Log streaming over WAN (main cause)

Log streaming throughput was 0.05–0.11 MiB/s, setting the wall-time floor:

Cluster	Log file	Size (MiB)	Duration (s)	Throughput
hub	ramen-hub-operator/manager/current.log	30.50	301.50	0.10 MiB/s
hub	odf-multicluster-console/current.log	18.70	284.40	0.07 MiB/s
hub	odfmo-controller-manager/current.log	12.70	273.60	0.05 MiB/s
arft-site02	ramen-dr-cluster-operator/previous.log	31.80	298.40	0.11 MiB/s
arft-site02	ramen-dr-cluster-operator/current.log	14.80	157.50	0.09 MiB/s
arft-site03	ramen-dr-cluster-operator/current.log	12.70	7.70	1.64 MiB/s

From the log — a single 30 MiB log takes 5 minutes to stream over WAN:

2026-07-29T17:48:24.939  hub.logs  Gathered "openshift-operators/ramen-hub-operator-55d4c66fbb-pzb92/manager/current.log" in 301.470 seconds (0.10 MiB/s)
2026-07-29T17:48:07.874  hub.logs  Gathered "openshift-operators/odf-multicluster-console-7549574c65-gcs9w/odf-multicluster-console/current.log" in 284.410 seconds (0.07 MiB/s)
2026-07-29T17:47:57.052  hub.logs  Gathered "openshift-operators/odfmo-controller-manager-954859499-v7cvt/odf-multicluster-orchestrator/current.log" in 273.588 seconds (0.05 MiB/s)
2026-07-29T17:48:22.197  arft-site02.logs  Gathered "openshift-dr-system/ramen-dr-cluster-operator-5b577cb69f-h875m/manager/previous.log" in 298.384 seconds (0.11 MiB/s)
2026-07-29T17:46:01.260  arft-site02.logs  Gathered "openshift-dr-system/ramen-dr-cluster-operator-5b577cb69f-h875m/manager/current.log" in 157.464 seconds (0.09 MiB/s)

Compare with site03 — same size log, 40× faster (different network path):

2026-07-29T17:43:30.752  arft-site03.logs  Gathered "openshift-dr-system/ramen-dr-cluster-operator-5b577cb69f-2bcm8/manager/current.log" in 7.698 seconds (1.64 MiB/s)

Hub and site02 wall time is bounded by the slowest log stream (~300 s each).
Site03 logs finished in <8 s (15–30× better throughput), so it wasn't bottlenecked.

2. WAN API latency (many empty list calls)

Cluster	Total lists	Empty lists	Median RTT	Sum empty-list time (worker-s)
arft-site02	859	679 (79%)	0.240 s	1,298 s
hub	569	406 (71%)	0.184 s	707 s
arft-site03	965	747 (77%)	0.253 s	447 s

Even "nothing here" responses cost 0.18–0.25 s round-trip over WAN.

From the log — typical empty list calls at ~0.23–0.25 s each:

2026-07-29T17:43:22.279  hub  Listed 0 "limitranges" in 0.229 seconds
2026-07-29T17:43:22.769  arft-site02  Listed 0 "limitranges" in 0.238 seconds
2026-07-29T17:49:05.415  arft-site02  Listed 0 "zen.cpd.ibm.com/zenextensions" in 0.238 seconds
2026-07-29T17:49:05.656  arft-site02  Listed 0 "metrics.k8s.io/pods" in 0.242 seconds

3. Slow API responses during log streaming

The "Listed" log measures only the k8s API List() call time (no queue waits).
Some List calls took 50–77 s for site02 and 30–40 s for hub — these are
genuinely slow API responses that correlate with concurrent log streaming:

Cluster	Slowest list call	# calls > 1 s	Sum of slow calls
arft-site02	77.40 s	41	1,125 s
hub	40.20 s	53	637 s
arft-site03	45.00 s	6	263 s

From the log — List calls for basic resources taking 35–77 s during active log
streaming:

2026-07-29T17:45:28.878  arft-site02  Listed 0 "batch/jobs" in 77.384 seconds
2026-07-29T17:45:28.878  arft-site02  Listed 0 "batch/jobs" in 77.274 seconds
2026-07-29T17:44:22.321  hub  Listed 1 "networking.k8s.io/servicecidrs" in 34.413 seconds
2026-07-29T17:44:23.387  hub  Listed 1 "networking.k8s.io/ingressclasses" in 35.477 seconds
2026-07-29T17:44:24.481  hub  Listed 100 "networking.k8s.io/ipaddresses" in 36.573 seconds

The median List RTT is 0.18–0.24 s, but during active log streaming (which runs
for ~300 s on hub and site02), some API calls are inflated by 100–300×. We don't
know why — possible causes:

• WAN link saturation — 500 Mbit/s WiFi locally, connecting via company VPN
  to servers in the US. Unknown networking between VPN endpoint and cluster.
• API server CPU/resource pressure — concurrent log streaming puts load on
  the API server (kubelet proxy for pod logs), which may starve List requests.
• IO limits on API server node — cloud instances often have IOPS limits that
  can throttle etcd reads when log streaming drives concurrent IO.

Note: the gather and inspect use separate unbuffered work queues (6 workers each).
"Listed" measures only the API call; "Gathered" includes the full loop
(list + inspect + dump) and can also stall when the inspect queue is full and
all inspect workers are busy streaming logs.

Where time goes per cluster

arft-site02 (344.4 s wall)

Activity	Duration	Notes
Log streaming (previous.log)	298.40 s	31.8 MiB @ 0.11 MiB/s — sets wall time
Log streaming (current.log)	157.50 s	14.8 MiB @ 0.09 MiB/s
List API calls (worker-time)	2,052.00 s	859 calls

2026-07-29T17:48:22.197  arft-site02.logs  Gathered ".../ramen-dr-cluster-operator-.../manager/previous.log" in 298.384 seconds (0.11 MiB/s)
2026-07-29T17:46:01.260  arft-site02.logs  Gathered ".../ramen-dr-cluster-operator-.../manager/current.log" in 157.464 seconds (0.09 MiB/s)
2026-07-29T17:49:05.668  Gathered 5206 resources from cluster "arft-site02" in 344.35 seconds

hub (322.9 s wall)

Activity	Duration	Notes
Log streaming (ramen-hub)	301.50 s	30.5 MiB @ 0.10 MiB/s — sets wall time
Log streaming (odf-console)	284.40 s	18.7 MiB @ 0.07 MiB/s
Log streaming (odfmo)	273.60 s	12.7 MiB @ 0.05 MiB/s
List API calls (worker-time)	1,841.00 s	569 calls

2026-07-29T17:48:24.939  hub.logs  Gathered ".../ramen-hub-operator-.../manager/current.log" in 301.470 seconds (0.10 MiB/s)
2026-07-29T17:48:07.874  hub.logs  Gathered ".../odf-multicluster-console-.../current.log" in 284.410 seconds (0.07 MiB/s)
2026-07-29T17:47:57.052  hub.logs  Gathered ".../odfmo-controller-manager-.../current.log" in 273.588 seconds (0.05 MiB/s)
2026-07-29T17:48:44.192  Gathered 4182 resources from cluster "hub" in 322.88 seconds

arft-site03 (125.4 s wall)

Activity	Duration	Notes
Log streaming (all 4)	17.00 s	Fast — not a bottleneck
List API calls (worker-time)	732.00 s	965 calls / 6 workers ≈ 122 s

2026-07-29T17:43:30.752  arft-site03.logs  Gathered ".../ramen-dr-cluster-operator-.../manager/current.log" in 7.698 seconds (1.64 MiB/s)
2026-07-29T17:43:27.042  arft-site03.logs  Gathered ".../tekton-operator-webhook-.../current.log" in 3.981 seconds (0.76 MiB/s)
2026-07-29T17:43:25.886  arft-site03.logs  Gathered ".../openshift-pipelines-operator-.../current.log" in 2.830 seconds (0.50 MiB/s)
2026-07-29T17:43:25.773  arft-site03.logs  Gathered ".../openshift-pipelines-operator-.../current.log" in 2.717 seconds (0.46 MiB/s)
2026-07-29T17:45:26.659  Gathered 6865 resources from cluster "arft-site03" in 125.36 seconds

Page cache effect — second gather is 7–15× faster

validate clusters ran first; validate application ran ~3 minutes later,
gathering the same logs from the same pods. Same log files, dramatically
different throughput:

Log	1st run (s)	1st MiB/s	2nd run (s)	2nd MiB/s	Speedup
hub: ramen-hub-operator current.log	301.50	0.10	28.50	1.08	10.6×
hub: odf-multicluster-console	284.40	0.07	25.00	0.75	11.4×
hub: odfmo-controller-manager	273.60	0.05	17.30	0.73	15.8×
arft-site02: ramen-dr-cluster current	157.50	0.09	41.80	0.55	3.8×
arft-site02: ramen-dr-cluster previous	298.40	0.11	43.50	0.73	6.9×
arft-site03: ramen-dr-cluster current	7.70	1.64	16.80	0.89	0.5×

From the validate-application log (2nd run):

2026-07-29T17:52:40.073  hub.logs  Gathered ".../ramen-hub-operator-.../manager/current.log" in 28.492 seconds (1.08 MiB/s)
2026-07-29T17:52:36.570  hub.logs  Gathered ".../odf-multicluster-console-.../current.log" in 25.006 seconds (0.75 MiB/s)
2026-07-29T17:52:28.873  hub.logs  Gathered ".../odfmo-controller-manager-.../current.log" in 17.304 seconds (0.73 MiB/s)
2026-07-29T17:53:00.631  arft-site02.logs  Gathered ".../ramen-dr-cluster-operator-.../manager/current.log" in 41.761 seconds (0.55 MiB/s)
2026-07-29T17:53:07.741  arft-site02.logs  Gathered ".../ramen-dr-cluster-operator-.../manager/previous.log" in 43.498 seconds (0.73 MiB/s)
2026-07-29T17:52:31.548  arft-site03.logs  Gathered ".../ramen-dr-cluster-operator-.../manager/current.log" in 16.777 seconds (0.89 MiB/s)

Hub and site02 logs went from 0.05–0.11 MiB/s to 0.55–1.08 MiB/s on the second
gather. The most likely explanation is page cache warming: the first gather
reads log data from storage (slow IO), and the second gather reads the same data
from the kernel page cache (fast).

arft-site03 is the exception — it was already fast on the first run (1.64 MiB/s)
and slightly slower on the second (0.89 MiB/s, more concurrent streams). This
suggests site03 logs were already cached before the first gather.

We cannot do anything about cold page cache, and it's very hard to test without
access to the nodes to drop caches. The takeaway is that we need to expect much
slower log gather in some cases (first gather after node restart, after memory
pressure evicts cached log data, or with large logs that exceed available cache).

Testing cold page cache

To reproduce the worst case (first gather after reboot), drop page cache on
every node before gathering:

for node in $(oc get nodes -o name); do
  oc debug "$node" -- chroot /host sh -c 'sync; echo 1 > /proc/sys/vm/drop_caches'
done

Value 1 drops page cache (file data in memory) — this is what makes log
streaming slow on cold nodes. No need for 3 (which also drops dentries/inodes);
the log IO cost dominates.

Test plan:

1. Drop caches on all nodes
2. Run validate clusters — expect ~300 s (cold)
3. Run validate clusters again immediately — expect ~50 s (warm)
4. Drop caches on all nodes again
5. Run validate clusters --remote — compare with step 2

This isolates page cache vs. WAN latency as the two independent factors.

Next steps

• More local gathers — compare page cache effect
• Cold cache test (see above)
• Remote gathers — compare with local

Dependencies

• Slow validate application with big OpenShift clusters #407 (Support remote gather)

[nirs]

Testing local vs remote gather performance

Test setup

• 3 OpenShift clusters (hub, site2, site3) — same clusters as the original report
• Same pods confirmed running (identical pod names, same ReplicaSet hashes)
• Gathering from namespaces: openshift-dr-system, openshift-operators + cluster-scoped resources
• Local machine: MacBook Pro, 500 Mbps WiFi, corporate VPN to US-based clusters

Test scripts

test-gather-clusters-local.sh

# Simulate ramenctl validate clusters gather:
# - gather from ramen hub and managed clusters namespaces
# - gather cluster data

echo "== Testing gather clusters - local =="

for i in 1 2 3; do
    echo "=== iteration $i ==="
    kubectl gather \
        --kubeconfig "kubeconfigs/hub.yaml,kubeconfigs/site2.yaml,kubeconfigs/site3.yaml" \
        --namespaces "openshift-dr-system,openshift-operators" \
        --cluster \
        --directory "out/kubectl-gather/clusters/local-$i"
    echo
    sleep 10
done

test-gather-clusters-remote.sh

# Simulate ramenctl validate clusters gather:
# - gather from ramen hub and managed clusters namespaces
# - gather cluster data

echo "== Testing gather clusters - remote =="

for i in 1 2 3; do
    echo "=== iteration $i ==="
    kubectl gather \
        --kubeconfig "kubeconfigs/hub.yaml,kubeconfigs/site2.yaml,kubeconfigs/site3.yaml" \
        --namespaces "openshift-dr-system,openshift-operators" \
        --cluster \
        --remote \
        --directory "out/kubectl-gather/clusters/remote-cold-$i"
    echo
    sleep 10
done

test-gather-clusters-remote-cold.sh (with drop-caches.sh)

# Simulate ramenctl validate clusters gather with cold cache:
# - gather from ramen hub and managed clusters namespaces
# - gather cluster data

set -eu

echo "== Testing gather clusters - remote with cold cache =="

for i in 1 2 3; do
    echo "=== iteration $i ==="

    echo "dropping caches"
    for kubeconfig in kubeconfigs/*.yaml; do
        ./drop-caches.sh $kubeconfig > drop-caches.log 2>&1 &
    done
    wait

    echo "gathering data"
    kubectl gather \
        --kubeconfig "kubeconfigs/hub.yaml,kubeconfigs/site2.yaml,kubeconfigs/site3.yaml" \
        --namespaces "openshift-dr-system,openshift-operators" \
        --cluster \
        --remote \
        --directory "out/kubectl-gather/clusters/remote-cold-$i"
    echo
done

#!/bin/bash
# drop-caches.sh - Drop page caches on all nodes in a cluster

set -eu

kubeconfig=${1:?Usage: drop-caches KUBECONFIG}

for node in $(oc get nodes -o name --kubeconfig "$kubeconfig"); do
    echo "dropping caches in $node"
    oc debug "$node" --kubeconfig "$kubeconfig" -- chroot /host sh -c 'sync; echo 1 > /proc/sys/vm/drop_caches' &
done

wait
echo "dropped caches on all nodes"

test-gather-cold-apiserver.sh

#!/bin/bash

# Test the effect of cold API server watch cache on gather performance.
# Restarts kube-apiserver on all clusters, then compares first vs second gather.

set -e

KUBECONFIGS_DIR="kubeconfigs"
KUBECONFIGS="$KUBECONFIGS_DIR/hub.yaml,$KUBECONFIGS_DIR/site2.yaml,$KUBECONFIGS_DIR/site3.yaml"
OUTPUT_DIR="out/kubectl-gather/clusters"

clusters=(hub site2 site3)

echo "=== Restarting API servers on all clusters ==="
for cluster in "${clusters[@]}"; do
    echo "Restarting kube-apiserver on $cluster..."
    oc --kubeconfig="$KUBECONFIGS_DIR/$cluster.yaml" delete pods \
        -n openshift-kube-apiserver -l app=openshift-kube-apiserver &
done
wait
echo "Delete requests sent."

echo ""
echo "=== Waiting for API servers to be ready ==="
for cluster in "${clusters[@]}"; do
    echo "Waiting for $cluster..."
    until oc --kubeconfig="$KUBECONFIGS_DIR/$cluster.yaml" get --raw /readyz &>/dev/null; do
        sleep 5
    done
    echo "$cluster API server ready."
done

echo ""
echo "=== First gather (cold watch cache) ==="
kubectl gather \
    --kubeconfig="$KUBECONFIGS" \
    --namespaces=openshift-dr-system,openshift-operators \
    --cluster \
    --remote \
    --directory="$OUTPUT_DIR/cold-apiserver"
echo ""
echo "=== Second gather (warm watch cache) ==="
kubectl gather \
    --kubeconfig="$KUBECONFIGS" \
    --namespaces=openshift-dr-system,openshift-operators \
    --cluster \
    --remote \
    --directory="$OUTPUT_DIR/warm-apiserver"
echo ""
echo "=== Done ==="

Results

Local vs Remote

Run	hub	site2	site3	Total
local-1	43.4s	53.2s	93.9s	93.9s
local-2	53.2s	47.0s	88.3s	88.3s
local-3	51.1s	58.8s	99.3s	99.3s
remote-1	29.8s	35.4s	38.2s	38.2s
remote-2	22.6s	33.7s	38.2s	38.2s
remote-3	24.2s	34.8s	44.0s	44.0s

Local gather is 2.3–2.5× slower than remote.

Remote hot vs cold page cache

Run	hub	site2	site3	Total
remote-1	29.8s	35.4s	38.2s	38.2s
remote-2	22.6s	33.7s	38.2s	38.2s
remote-3	24.2s	34.8s	44.0s	44.0s
remote-cold-1	23.4s	33.0s	35.1s	35.1s
remote-cold-2	22.6s	32.7s	44.0s	44.0s
remote-cold-3	23.6s	36.3s	38.1s	38.1s

No difference. Cold page cache (dropping caches on all nodes before gathering) does not affect remote gather performance. In-cluster disk IO is fast enough that it doesn't matter whether logs are served from page cache or disk.

Cold vs warm API server watch cache

Run	hub	site2	site3	Total
cold-apiserver	24.0s	37.2s	39.2s	39.2s
warm-apiserver	23.2s	33.4s	40.3s	40.3s

No difference. Restarting all kube-apiserver pods (clearing watch caches) does not affect gather performance.

Conclusions

Why remote gather is faster:

Remote gather eliminates WAN latency from the critical path:

• All API calls happen in-cluster (sub-millisecond RTT vs 200ms+ WAN RTT)
• Log streaming happens in-cluster (local network bandwidth vs WAN throughput)
• Only the final tar transfer goes over WAN (single bulk transfer)

What does NOT explain the original 344s:

• ❌ Cold page cache on cluster nodes — no effect
• ❌ Cold API server watch cache — no effect
• ❌ Different cluster installation — confirmed same pods running since before the original report

What we observed in local gathers:

• During log streaming, API List calls slow down from ~0.2s (baseline WAN RTT) to 5–12s
• After log streaming completes, API calls normalize back to ~0.2s
• This confirms bandwidth saturation: concurrent log streaming over WAN starves API calls
• Log throughput varies significantly between runs (0.47–0.87 MiB/s for the same log on the same day)
• The original 344s run showed ~0.04 MiB/s throughput — 10–20× worse than today's local gathers on the same clusters

Remaining issues

1. Log streaming slows down API List calls in local gather. During concurrent log streaming, WAN bandwidth saturation causes API calls to take 5–12s instead of 0.2s. This effectively serializes resource listing while logs are streaming. Possible improvement: limit concurrent log streams or prioritize API calls over log streaming.

2. Local gather performance is unpredictable. Same clusters, same machine, same WiFi — but 344s one day vs 94s another day. The VPN/WAN path introduces uncontrollable variability that makes local gather unreliable for large clusters.

3. Remote gather is consistently fast (~38–44s) regardless of page cache state or API server cache state. This is the recommended approach for large clusters.

[nirs]

Local vs Remote gather — workload visualization

The plot below shows all operations during a single local gather and a single remote
gather of the same 3 clusters (hub, site2, site3), same namespaces, ~330 MiB data.

How to read the plot

• X axis: wall clock time since gather started (seconds)
• Y axis: duration of each operation (seconds)
• Blue dots: local List API calls (each dot = one kubectl List request over WAN)
• Blue solid lines: local log streams (span shows start→end, height = duration)
• Orange dots: remote List API calls (in-cluster, sub-millisecond RTT)
• Orange solid lines: remote log streams (in-cluster)
• Orange dashed lines: remote pod startup (oc adm must-gather pod scheduling)
• Orange dotted lines: remote download (tar transfer of results back over WAN)

Analysis

Local gather (88s, wired):

• 9 concurrent log streams (3 clusters × 3 logs) compete for the same VPN link
• During log streaming (0–37s), List API calls are inflated from 0.2s to 3–5s
• After logs finish, List calls normalize to ~0.2s (baseline WAN RTT)
• The longest log stream (37s, site3 previous.log at 1.11 MiB/s) dominates the timeline
• 173 of 2,499 List calls (7%) exceeded 1 second

Remote gather (40s):

• Pod startup: 3–4s (pod scheduling, image already on nodes)
• In-cluster gather: 10–18s (all List calls + log streams happen in-cluster at full speed)
• Download: 11–19s (single bulk tar transfer over WAN)
• List calls never exceed 0.83s — no bandwidth contention with log streaming

Key insight

In local gather, 9 log streams saturate the WAN link, which serializes everything.
In remote gather, each cluster's work is independent — logs stream on the cluster
network while only the final result goes over WAN as a single transfer.

The remote gather overhead (pod startup + download) adds ~22s, but the actual gather
work is 4.8× faster (18s vs 88s) because it eliminates WAN contention entirely.

Consistency (3 runs each, MTU 1445)

Connection	Run 1	Run 2	Run 3	Average	Range
Remote	40.1s	43.0s	36.5s	39.9s	6.5s
Wired	87.8s	89.2s	83.3s	86.8s	5.8s
WiFi	130.4s	82.0s	96.8s	103.1s	48.4s

Wired and remote are consistent (6–7% variability). WiFi has high
variability (59%) — unreliable for benchmarking.

Not shown in this plot

Secret sanitization (PBKDF2 hashing) is a separate CPU bottleneck in remote gather,
tracked in kubectl-gather#176.
It does not affect local gather timing (dominated by WAN) but is the critical path
for remote gather on clusters with many secrets.

Next steps

• Consider prioritizing API List calls over log streaming in local gather to
  reduce the inflation of List call latency during concurrent log streaming.
• Try smaller inspection work queue size in kubectl-gather to limit concurrent log
  streams per cluster, reducing bandwidth contention with API calls.

[nirs]

MTU discovery

The corporate VPN derives the tunnel MTU from the physical interface MTU.
When the physical interface uses the default MTU (1500), the VPN subtracts
its overhead (~110 bytes) and sets the tunnel to ~1390. But the corporate
network adds further encapsulation (likely VXLAN, ~50 bytes), so 1500-byte
outer packets don't actually fit — they fragment or get dropped, causing
severe slowdowns (30x in our tests).

The only reliable way to find the right physical device MTU is manual
binary search:

1. Set en0 MTU to a candidate value (sudo ifconfig en0 mtu <value>)
2. Disconnect and reconnect the VPN (required — the tunnel MTU is computed
   at connect time)
3. Test whether the connection works (e.g. access a cluster via browser or API)
4. Repeat with higher/lower values until the boundary is found

Results

en0 MTU	utun MTU	Works?
1500	1390	Very slow / broken
1472	1362	Broken
1451	1342	Broken
1448	1326	Works
1445	1326	Works
1390	1280	Works

The highest working value is 1448. We used 1445 (3-byte safety margin)
for all benchmark runs. Same results on both WiFi and wired connections,
confirming the limit is in the corporate network path, not the local network.

Local gather wired (MTU 1445)

1 Gbit home router -> 2 Gbit fiber link. ~330 MiB gathered data.

Run	hub	site2	site3	Total
wired-1	45.8s	53.7s	87.8s	87.8s
wired-2	59.4s	47.5s	89.2s	89.2s
wired-3	71.8s	51.3s	83.3s	83.3s

No errors. Very consistent (range 5.8s across runs).

Local gather WiFi (MTU 1445)

500 Mbit mesh WiFi -> 1 Gbit home router -> 2 Gbit fiber link. ~330 MiB gathered data.

Run	hub	site2	site3	Total
wifi-1	58.7s	73.9s	130.4s	130.4s
wifi-2	44.7s	62.0s	82.0s	82.0s
wifi-3	51.4s	46.7s	96.8s	96.8s

No errors. wifi-1 (130s) was the first gather of the day and significantly
slower than the subsequent runs. WiFi also adds wireless jitter on top.

Remote gather (MTU 1445)

Run	hub	site2	site3	Total
remote-1	22.8s	35.1s	40.1s	40.1s
remote-2	22.7s	34.1s	43.0s	43.0s
remote-3	21.4s	32.0s	36.5s	36.5s

Very consistent (range 6.5s). In-cluster gather is ~10–18s; the rest is pod
startup (~4s) and tar download (~11–19s).

Conclusion

• The slow run in the issue description (348s) was caused by incorrect MTU.
  The default MTU (1500) does not work via the corporate VPN due to additional
  network encapsulation inside the corporate network.
• The physical device MTU must be found by binary search with VPN
  reconnects — we found 1448 is the maximum working value.
• No GOAWAY errors in any of the 9 MTU 1445 runs (the earlier MTU 1390 run
  had a GOAWAY on wired-3 that silently skipped a large log).
• Remote gather is significantly faster than local:

Method	Average	vs remote	Notes
Remote	~40s	—	~10–18s in-cluster + ~13–22s overhead
Local wired	~87s	2.2x
Local WiFi	~103s	2.6x

Note: data grew from ~303 MiB (MTU 1390 runs) to ~330 MiB (MTU 1445 runs),
so absolute times are not directly comparable with previous runs.