fix(e2e): make instance metadata endpoint unreachable on kind nodes (#1352)

Since 2026-08-29 the e2e tests fail on CI regardless of the branch under
test: the collector container of the daemonset collector does not become
ready in time, the BeforeAll node fails and every remaining spec is
skipped. c94a444 bought time by raising the startup probe to 120s and
the test timeout to 150s. This commit adds a workaround at the network
level.

This commit also reverts the workarounds from an earlier commit:
waitForCollectorToStart waits 60s again (instead of 150) and the
daemonset collector is back to the default startup probe timeouts.

A probe that ran the eight resource detectors of the daemonset
collector, one at a time (run
[33278618771](https://github.com/dash0hq/dash0-operator/actions/runs/33278618771)),
singled out gcp:

```
  eks 0.003s   ecs 0.000s   ec2 2.000s   gcp 27.947s   azure 2.000s
  aks 2.000s   k8snode 0.004s   system 0.000s   all 27.972s
```

ec2, azure and aks stop at the configured timeout of 2s, the rest finish
in milliseconds, and all eight together take no longer than gcp alone. A
second probe replayed the steps of the gcp detector inside a pod (run
33307695809):

```
  onGKE/metadata   12.824s  dial tcp 169.254.169.254:80: i/o timeout
  onGCE/metadata   13.928s  dial tcp 169.254.169.254:80: i/o timeout
  metadata.OnGCE    0.003s  false
  tcp-dial         10.001s  i/o timeout       (reference)
  dns-lookup        0.003s  no such host      (reference)
  CloudPlatform()  26.519s  UnknownPlatform
```

DNS is healthy and answers in 3ms, so name resolution was never
involved. The IP 169.254.169.254 is blackholed on the runner: the dials
end in "i/o timeout", not "connection refused", so the packets are
dropped rather than rejected.

The delay comes from CloudPlatform() of

github.com/GoogleCloudPlatform/opentelemetry-operations-go/detectors/gcp.
Two of the platform checks query the metadata server, and both pass
context.TODO(), which carries no deadline, so the timeout of the
resourcedetection processor cannot stop them:

onGKE() reads KUBERNETES_SERVICE_HOST, which is always set inside a pod,
and
then calls InstanceAttributeValueWithContext(context.TODO(),
"cluster-location")
  onGCE()  calls GetWithContext(context.TODO(), "instance/machine-type")

metadata.NewClient(nil) returns the package default client, which dials
with a timeout of 2s and retries up to five times, hence about 13s per
call. The daemonset collector runs the detectors once per pipeline and
has three, which is about 80s against a startup probe that grants 90s.

This is a known upstream defect, tracked in

GoogleCloudPlatform/opentelemetry-operations-go#1026,
and not fixed yet.

The workaround, on every kind node: 
* Add an unreachable route for the endpoint that makes connect() fail
immediately with EHOSTUNREACH, which the metadata client neither retries
nor treats as temporary, so the detector gives up immediately.
* Write a sysctl `net.ipv4.icmp_ratelimit=0`; without it, only the first
connections fail fast, then Linux throttles the generation of ICMP
errors per destination via net.ipv4.icmp_ratelimit, one message per
second by default, so every connection after the first gets no ICMP at
all and waits for the retransmission of its SYN instead. The retry burst
of the detector is exactly the pattern that runs into this, which is why
isolated calls end at 0s while the detector as a whole would still >
18s.

An iptables REJECT rule would have been the worse choice: both
--reject-with tcp-reset and the default icmp-port-unreachable produce
ECONNREFUSED, and syscallRetryable in retry_linux.go of
cloud.google.com/go/compute/metadata retries exactly ECONNRESET and
ECONNREFUSED, so the five retries would still run and only the dial
timeouts would be saved.

Disabling the rate limit on every node should make all of the calls fail
immediately rather than only the first few. There is still an
unexplained 1s delay for every consecutive call after the first call,
e.g. a total lag of ~2 seconds. But since that is coincidentally roughly
the same as the 2 second timeout that is configured for the
resourcedetection processor, it is acceptable.

Nothing in the kind cluster needs the instance metadata endpoint, so
making it unreachable costs nothing. Note that this only repairs CI. Any
cluster that blackholes the link-local range still pays about 80s of
collector startup, and the e2e suite no longer covers that case.

Author: basti1302

.github/workflows/e2e.yaml

@@ -128,6 +128,12 @@ jobs:
           echo "All pods:"
           kubectl get pods -A
 
+      # Workaround for resourcedetection bug in the gcp detector that makes the collector slow to become ready.
+      # https://github.com/GoogleCloudPlatform/opentelemetry-operations-go/issues/1026
+      - name: Make the instance metadata endpoint unreachable
+        run: |
+          ./.github/workflows/e2e/add-instance-metadata-unreachable-route.sh
+
       - name: Verify registry is running
         run: |
           echo "Checking registry container..."

.github/workflows/e2e/add-instance-metadata-unreachable-route.sh

@@ -0,0 +1,94 @@
+#!/usr/bin/env bash
+
+# SPDX-FileCopyrightText: Copyright 2026 Dash0 Inc.
+# SPDX-License-Identifier: Apache-2.0
+
+# Adds an unreachable route for the cloud instance metadata endpoint to every node of the kind cluster.
+#
+# The CI runner drops the packets to 169.254.169.254 rather than rejecting them. The gcp resource detector of the
+# collector queries that address twice while it starts, in onGKE() and in onGCE() of
+# github.com/GoogleCloudPlatform/opentelemetry-operations-go/detectors/gcp, and both calls pass context.TODO(), which
+# carries no deadline. See also: https://github.com/GoogleCloudPlatform/opentelemetry-operations-go/issues/1026.
+# The timeout of the resourcedetection processor therefore cannot stop them. Each call runs into the dial timeout of
+# 2s five times, which costs about 13s, so the collector needs about 80s to become ready instead of the 6s it needs when
+# the endpoint answers.
+#
+# An unreachable route makes connect() fail at once with EHOSTUNREACH. The metadata client neither retries that error
+# nor treats it as temporary, so the detector gives up immediately.
+#
+# The route on its own is not enough, though. Linux throttles the generation of ICMP errors per destination via
+# net.ipv4.icmp_ratelimit, one message per second by default. The first connections get their ICMP host unreachable at
+# once, the ones after that get nothing and fall back to the retransmission of the SYN, which is why a measurement of
+# the route alone showed the first two calls at 0s but the detector as a whole still at 18s, down from 28s. The rate
+# limit is therefore disabled as well, which is what makes every call fail immediately rather than only the first few.
+#
+# Note: an iptables REJECT rule is the worse choice here. Both --reject-with tcp-reset and the default
+# icmp-port-unreachable produce ECONNREFUSED, and syscallRetryable in retry_linux.go of
+# cloud.google.com/go/compute/metadata retries exactly ECONNRESET and ECONNREFUSED. The five retries would still run,
+# only the dial timeouts would be saved, not the backoff between them.
+#
+# Nothing in the kind cluster needs the instance metadata endpoint, so making it unreachable costs nothing. This works
+# around the CI environment, it does not fix the underlying problem, which is that the gcp detector discards the
+# deadline it is given, see https://github.com/GoogleCloudPlatform/opentelemetry-operations-go/issues/1026.
+
+set -euo pipefail
+
+metadata_ip=169.254.169.254
+
+add_route() {
+  local node=$1
+  docker exec "$node" ip route replace unreachable "$metadata_ip/32"
+}
+
+# Without this, only the first ICMP error per second reaches the caller and every connection after that waits for the
+# retransmission of its SYN instead of failing at once.
+disable_icmp_rate_limit() {
+  local node=$1
+  docker exec "$node" sysctl --quiet --write net.ipv4.icmp_ratelimit=0
+}
+
+# Verifies the two settings rather than the latency they produce. Measuring the latency from the node would prove
+# nothing: a request that starts on the node runs into the unreachable route during the route lookup and fails locally
+# in microseconds, without an ICMP packet ever being generated, whether the rate limit is in place or not. Only traffic
+# that the node forwards, which means traffic from a pod, makes the node generate the ICMP error that the rate limit
+# applies to. The latency that matters is therefore only measurable from a pod, and it is what the gcp detector probe
+# measures.
+verify_settings() {
+  local node=$1
+
+  local route
+  route=$(docker exec "$node" ip route show | grep -F "unreachable $metadata_ip" || true)
+  if [[ -z $route ]]; then
+    echo "error: node $node has no unreachable route for $metadata_ip."
+    exit 1
+  fi
+
+  local rate_limit
+  rate_limit=$(docker exec "$node" sysctl --values net.ipv4.icmp_ratelimit)
+  if [[ $rate_limit != 0 ]]; then
+    echo "error: node $node still limits the rate of ICMP errors, net.ipv4.icmp_ratelimit=$rate_limit."
+    exit 1
+  fi
+
+  printf '  %-34s %s, net.ipv4.icmp_ratelimit=%s\n' "$node" "$route" "$rate_limit"
+}
+
+main() {
+  local kind_cluster="${DASH0_KIND_CLUSTER:?DASH0_KIND_CLUSTER needs to be set}"
+
+  local nodes
+  nodes=$(docker ps --filter "label=io.x-k8s.kind.cluster=$kind_cluster" --format '{{.Names}}')
+  if [[ -z $nodes ]]; then
+    echo "error: no running node container of cluster $kind_cluster found."
+    exit 1
+  fi
+
+  local node
+  for node in $nodes; do
+    add_route "$node"
+    disable_icmp_rate_limit "$node"
+    verify_settings "$node"
+  done
+}
+
+main "$@"

test/e2e/collector.go

@@ -125,11 +125,7 @@ func waitForCollectorToStart(operatorNamespace string, operatorHelmChart string)
 			))).To(Succeed())
 	}
 
-	// The collector's resourcedetection processor queries the cloud instance metadata endpoints once per pipeline on
-	// startup. On a machine where these endpoints neither answer nor refuse the connection, each of these runs takes
-	// about 25 seconds instead of the configured two seconds, so the daemonset collector needs about 80 seconds to
-	// become ready. See also the startup probe that executeOperatorHelmChart grants it.
-	Eventually(verifyCollectorIsUp, 150*time.Second, time.Second).Should(Succeed())
+	Eventually(verifyCollectorIsUp, 60*time.Second, time.Second).Should(Succeed())
 
 	verifyCollectorHasOwnerReference(operatorNamespace, operatorHelmChart)
 }

test/e2e/operator.go

@@ -254,11 +254,6 @@ func executeOperatorHelmChart(
 		arguments = append(arguments, "--create-namespace")
 	}
 	arguments = append(arguments, "--set", "operator.developmentMode=true")
-	// The collector's resourcedetection processor queries the cloud instance metadata endpoints once per pipeline on
-	// startup. On a machine where these endpoints neither answer nor refuse the connection, each of these runs takes
-	// about 25 seconds instead of the configured two seconds, which exceeds the 90 seconds that the default startup
-	// probe grants the daemonset collector. 60 failures at a period of two seconds grant it 120 seconds.
-	arguments = append(arguments, "--set", "operator.collectors.daemonSetProbes.startup.failureThreshold=60")
 	if images != nil {
 		arguments = addHelmParametersForImages(arguments, *images)
 	}