fix(e2e): give the daemonset collector more time to become ready - #1350
Merged
Conversation
Since 2026-08-29 the e2e tests fail on CI regardless of the branch under test. The collector container of the daemonset collector does not become ready within the 60s the test suite waits for the rollout, the BeforeAll node fails and all 103 remaining specs are skipped. The cause is the resourcedetection processor of the collector, which runs the detectors eks, ecs, ec2, gcp, azure and aks once per pipeline on startup. It used to finish in exactly the configured 2s per pipeline, it now takes about 26s: last green run (2026-08-28 17:45 UTC) 2.000s 2.001s 2.001s ready after 6s first red run (2026-08-29 16:40 UTC) 26.233s 26.467s unfinished Three pipelines put the daemonset collector at about 80s, which is also close to the 90s that its default startup probe grants it. The deployment collector has only one pipeline and still becomes ready, which is why only the daemonset collector fails. Nothing in the operator changed. The runner image did: the job runs on a Depot runner, and between the last green and the first red run the image rolled from kubectl 1.36.3 to 1.36.4, with go 1.27.0 and kind v0.32.0 unchanged. Depot reports no incident for that period. The suspected mechanism is name resolution rather than the metadata endpoints themselves. The 2s timeout of the resourcedetection processor was honored to the millisecond before and is now exceeded by more than an order of magnitude, which fits a blocking DNS lookup: the Go resolver cannot abort an in-flight cgo lookup when the context is cancelled, so a name server that drops the query for metadata.google.internal instead of answering NXDOMAIN outlasts the deadline. The IMDSv2 hop limit, which has to be at least 2 for containers, would explain the same for the AWS and Azure detectors. To find out which of the two it is, add test-resources/bin/probe-instance-metadata-endpoints.sh and run it in the e2e workflow. It measures name resolution and connectivity to the metadata endpoints from the three network positions between the runner and a collector pod: the runner itself, every kind node container and a pod. A probe that runs into its timeout at one position but answers at the previous one shows at which hop the responses are dropped. Until the runner side is understood, keep CI green by granting the collector the time it currently needs. This is a workaround, not a fix for the slowdown: - Raise the startup probe of the daemonset collector from 90s to 120s for all e2e tests, via operator.collectors.daemonSetProbes.startup.failureThreshold. - Raise the timeout of waitForCollectorToStart from 60s to 150s.
Member
Author
|
/run-e2e |
Contributor
🚀 E2E tests triggered
Results will be posted here once the tests complete. You can also view the workflow runs directly. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Since 2026-08-29 the e2e tests fail on CI regardless of the branch under test. The collector container of the daemonset collector does not become ready within the 60s the test suite waits for the rollout, the BeforeAll node fails and all 103 remaining specs are skipped.
The cause is the resourcedetection processor of the collector, which runs the detectors eks, ecs, ec2, gcp, azure and aks once per pipeline on startup. It used to finish in exactly the configured 2s per pipeline, it now takes about 26s:
last green run (2026-08-28 17:45 UTC) 2.000s 2.001s 2.001s ready after 6s
first red run (2026-08-29 16:40 UTC) 26.233s 26.467s unfinished
Three pipelines put the daemonset collector at about 80s, which is also close to the 90s that its default startup probe grants it. The deployment collector has only one pipeline and still becomes ready, which is why only the daemonset collector fails.
Nothing in the operator changed. The runner image did: the job runs on a Depot runner, and between the last green and the first red run the image rolled from kubectl 1.36.3 to 1.36.4, with go 1.27.0 and kind v0.32.0 unchanged. Depot reports no incident for that period.
The suspected mechanism is name resolution rather than the metadata endpoints themselves. The 2s timeout of the resourcedetection processor was honored to the millisecond before and is now exceeded by more than an order of magnitude, which fits a blocking DNS lookup: the Go resolver cannot abort an in-flight cgo lookup when the context is cancelled, so a name server that drops the query for metadata.google.internal instead of answering NXDOMAIN outlasts the deadline. The IMDSv2 hop limit, which has to be at least 2 for containers, would explain the same for the AWS and Azure detectors.
To find out which of the two it is, add
test-resources/bin/probe-instance-metadata-endpoints.sh and run it in the e2e workflow. It measures name resolution and connectivity to the metadata endpoints from the three network positions between the runner and a collector pod: the runner itself, every kind node container and a pod. A probe that runs into its timeout at one position but answers at the previous one shows at which hop the responses are dropped.
Until the runner side is understood, keep CI green by granting the collector the time it currently needs. This is a workaround, not a fix for the slowdown: