fix(e2e): make instance metadata endpoint unreachable on kind nodes - #1352
Merged
Conversation
… nodes Since 2026-08-29 the e2e tests fail on CI regardless of the branch under test: the collector container of the daemonset collector does not become ready in time, the BeforeAll node fails and every remaining spec is skipped. c94a444 bought time by raising the startup probe to 120s and the test timeout to 150s. This commit adds a workaround at the network level. This commit also reverts the workarounds from an earlier commit: waitForCollectorToStart waits 60s again (instead of 150) and the daemonset collector is back to the default startup probe timeouts. A probe that ran the eight resource detectors of the daemonset collector, one at a time (run 33278618771), singled out gcp: eks 0.003s ecs 0.000s ec2 2.000s gcp 27.947s azure 2.000s aks 2.000s k8snode 0.004s system 0.000s all 27.972s ec2, azure and aks stop at the configured timeout of 2s, the rest finish in milliseconds, and all eight together take no longer than gcp alone. A second probe replayed the steps of the gcp detector inside a pod (run 33307695809): onGKE/metadata 12.824s dial tcp 169.254.169.254:80: i/o timeout onGCE/metadata 13.928s dial tcp 169.254.169.254:80: i/o timeout metadata.OnGCE 0.003s false tcp-dial 10.001s i/o timeout (reference) dns-lookup 0.003s no such host (reference) CloudPlatform() 26.519s UnknownPlatform DNS is healthy and answers in 3ms, so name resolution was never involved. The IP 169.254.169.254 is blackholed on the runner: the dials end in "i/o timeout", not "connection refused", so the packets are dropped rather than rejected. The delay comes from CloudPlatform() of github.com/GoogleCloudPlatform/opentelemetry-operations-go/detectors/gcp. Two of the platform checks query the metadata server, and both pass context.TODO(), which carries no deadline, so the timeout of the resourcedetection processor cannot stop them: onGKE() reads KUBERNETES_SERVICE_HOST, which is always set inside a pod, and then calls InstanceAttributeValueWithContext(context.TODO(), "cluster-location") onGCE() calls GetWithContext(context.TODO(), "instance/machine-type") metadata.NewClient(nil) returns the package default client, which dials with a timeout of 2s and retries up to five times, hence about 13s per call. The daemonset collector runs the detectors once per pipeline and has three, which is about 80s against a startup probe that grants 90s. This is a known upstream defect, tracked in GoogleCloudPlatform/opentelemetry-operations-go#1026, and not fixed yet. The workaround: An unreachable route for the endpoint on every kind node makes connect() fail at once with EHOSTUNREACH, which the metadata client neither retries nor treats as temporary, so the detector gives up immediately. An iptables REJECT rule would have been the worse choice: both --reject-with tcp-reset and the default icmp-port-unreachable produce ECONNREFUSED, and syscallRetryable in retry_linux.go of cloud.google.com/go/compute/metadata retries exactly ECONNRESET and ECONNREFUSED, so the five retries would still run and only the dial timeouts would be saved. Nothing in the kind cluster needs the instance metadata endpoint, so making it unreachable costs nothing. Note that this only repairs CI. Any cluster that blackholes the link-local range still pays about 80s of collector startup, and the e2e suite no longer covers that case.
basti1302
changed the title
The A/B run of the probe (run 33315392371) showed that the unreachable route on its own is not enough: step before after the route onGKE/metadata 13.250s 0s onGCE/metadata 13.933s 0s tcp-dial (10s cap) 10.000s 7.133s CloudPlatform() 27.782s 18.263s The route works, the error changes from "i/o timeout" to "connect: no route to host". But only the first connections fail fast. Linux throttles the generation of ICMP errors per destination via net.ipv4.icmp_ratelimit, one message per second by default, so every connection after the first gets no ICMP at all and waits for the retransmission of its SYN instead. The retry burst of the detector is exactly the pattern that runs into this, which is why the two isolated calls end at 0s while the detector as a whole still needs 18.3s. Disabling the rate limit on every node is what makes all of the calls fail immediately rather than only the first few. The check the script prints was misleading for the same reason: a single request fails fast even while the rate limit is in place. It now makes five requests in a row and prints all five durations.
The check the script printed after applying the workaround was worthless, and worse than nothing, because it looked like evidence. It measured five requests from the node and reported around 50 microseconds each, both before and after the ICMP rate limit was disabled, so it would have looked healthy in exactly the case it was added to catch. A request that starts on the node runs into the unreachable route during the route lookup and fails locally, without an ICMP packet ever being generated, so no rate limit can apply to it. Only traffic that the node forwards, that is, traffic from a pod, makes the node generate the ICMP error that is subject to net.ipv4.icmp_ratelimit. The latency that matters is therefore only measurable from a pod, which is what the gcp detector probe does. The script now verifies the two settings it applies, the unreachable route and the rate limit, and fails when either is missing. That is deterministic, needs no image and cannot be mistaken for a measurement of the effect. It reports, per node: unreachable 169.254.169.254 , net.ipv4.icmp_ratelimit=0 The workaround itself is confirmed and reproducible. Two A/B runs, each of which probes the gcp detector from a pod before and after applying the workaround within the same job: run CloudPlatform() before after 33316626554 28.086s 2.047s 33317232519 27.359s 2.045s That restores the behaviour from before the breakage: the last green run of the e2e tests detected resources in 2.000s, 2.001s and 2.001s per pipeline, which was the 2s timeout of the resourcedetection processor doing its job. The gcp detector now finishes just inside that budget again, so the three pipelines of the daemonset collector cost about 6s in total and the collector becomes ready about as quickly as it used to. One detail stays unexplained. Even with net.ipv4.icmp_ratelimit=0, only the first metadata call of a process fails instantly; every call after it costs about 1.02s, identically in both runs, which is what makes CloudPlatform() 2.045s rather than close to zero. Something on the path from the pod still emits about one ICMP error per second. Candidates are conntrack discarding the ICMP error of the flow, or a limit in a network namespace that the script does not set, but neither is measured, so neither is claimed here. At 2s against a startup probe of 90s it does not matter enough to chase.
Member
Author
|
With this fix, the e2e tests ran successfully again: https://github.com/dash0hq/dash0-operator/actions/runs/33317710641/job/99274088440 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Since 2026-08-29 the e2e tests fail on CI regardless of the branch under test: the collector container of the daemonset collector does not become ready in time, the BeforeAll node fails and every remaining spec is skipped. c94a444 bought time by raising the startup probe to 120s and the test timeout to 150s. This commit adds a workaround at the network level.
This commit also reverts the workarounds from an earlier commit: waitForCollectorToStart waits 60s again (instead of 150) and the daemonset collector is back to the default startup probe timeouts.
A probe that ran the eight resource detectors of the daemonset collector, one at a time (run 33278618771), singled out gcp:
ec2, azure and aks stop at the configured timeout of 2s, the rest finish in milliseconds, and all eight together take no longer than gcp alone. A second probe replayed the steps of the gcp detector inside a pod (run 33307695809):
DNS is healthy and answers in 3ms, so name resolution was never involved. The IP 169.254.169.254 is blackholed on the runner: the dials end in "i/o timeout", not "connection refused", so the packets are dropped rather than rejected.
The delay comes from CloudPlatform() of
github.com/GoogleCloudPlatform/opentelemetry-operations-go/detectors/gcp. Two of the platform checks query the metadata server, and both pass context.TODO(), which carries no deadline, so the timeout of the resourcedetection processor cannot stop them:
onGKE() reads KUBERNETES_SERVICE_HOST, which is always set inside a pod, and
then calls InstanceAttributeValueWithContext(context.TODO(), "cluster-location")
onGCE() calls GetWithContext(context.TODO(), "instance/machine-type")
metadata.NewClient(nil) returns the package default client, which dials with a timeout of 2s and retries up to five times, hence about 13s per call. The daemonset collector runs the detectors once per pipeline and has three, which is about 80s against a startup probe that grants 90s.
This is a known upstream defect, tracked in
GoogleCloudPlatform/opentelemetry-operations-go#1026, and not fixed yet.
The workaround, on every kind node:
net.ipv4.icmp_ratelimit=0; without it, only the first connections fail fast, then Linux throttles the generation of ICMP errors per destination via net.ipv4.icmp_ratelimit, one message per second by default, so every connection after the first gets no ICMP at all and waits for the retransmission of its SYN instead. The retry burst of the detector is exactly the pattern that runs into this, which is why isolated calls end at 0s while the detector as a whole would still > 18s.An iptables REJECT rule would have been the worse choice: both --reject-with tcp-reset and the default icmp-port-unreachable produce ECONNREFUSED, and syscallRetryable in retry_linux.go of cloud.google.com/go/compute/metadata retries exactly ECONNRESET and ECONNREFUSED, so the five retries would still run and only the dial timeouts would be saved.
Disabling the rate limit on every node should make all of the calls fail immediately rather than only the first few. There is still an unexplained 1s delay for every consecutive call after the first call, e.g. a total lag of ~2 seconds. But since that is coincidentally roughly the same as the 2 second timeout that is configured for the resourcedetection processor, it is acceptable.
Nothing in the kind cluster needs the instance metadata endpoint, so making it unreachable costs nothing. Note that this only repairs CI. Any cluster that blackholes the link-local range still pays about 80s of collector startup, and the e2e suite no longer covers that case.