Skip to content

fix(e2e): give the daemonset collector more time to become ready - #1350

Merged
basti1302 merged 1 commit into
mainfrom
investigate-e2e-failures
Aug 29, 2026
Merged

fix(e2e): give the daemonset collector more time to become ready#1350
basti1302 merged 1 commit into
mainfrom
investigate-e2e-failures

Conversation

@basti1302

Copy link
Copy Markdown
Member

Since 2026-08-29 the e2e tests fail on CI regardless of the branch under test. The collector container of the daemonset collector does not become ready within the 60s the test suite waits for the rollout, the BeforeAll node fails and all 103 remaining specs are skipped.

The cause is the resourcedetection processor of the collector, which runs the detectors eks, ecs, ec2, gcp, azure and aks once per pipeline on startup. It used to finish in exactly the configured 2s per pipeline, it now takes about 26s:

last green run (2026-08-28 17:45 UTC) 2.000s 2.001s 2.001s ready after 6s
first red run (2026-08-29 16:40 UTC) 26.233s 26.467s unfinished

Three pipelines put the daemonset collector at about 80s, which is also close to the 90s that its default startup probe grants it. The deployment collector has only one pipeline and still becomes ready, which is why only the daemonset collector fails.

Nothing in the operator changed. The runner image did: the job runs on a Depot runner, and between the last green and the first red run the image rolled from kubectl 1.36.3 to 1.36.4, with go 1.27.0 and kind v0.32.0 unchanged. Depot reports no incident for that period.

The suspected mechanism is name resolution rather than the metadata endpoints themselves. The 2s timeout of the resourcedetection processor was honored to the millisecond before and is now exceeded by more than an order of magnitude, which fits a blocking DNS lookup: the Go resolver cannot abort an in-flight cgo lookup when the context is cancelled, so a name server that drops the query for metadata.google.internal instead of answering NXDOMAIN outlasts the deadline. The IMDSv2 hop limit, which has to be at least 2 for containers, would explain the same for the AWS and Azure detectors.

To find out which of the two it is, add
test-resources/bin/probe-instance-metadata-endpoints.sh and run it in the e2e workflow. It measures name resolution and connectivity to the metadata endpoints from the three network positions between the runner and a collector pod: the runner itself, every kind node container and a pod. A probe that runs into its timeout at one position but answers at the previous one shows at which hop the responses are dropped.

Until the runner side is understood, keep CI green by granting the collector the time it currently needs. This is a workaround, not a fix for the slowdown:

  • Raise the startup probe of the daemonset collector from 90s to 120s for all e2e tests, via operator.collectors.daemonSetProbes.startup.failureThreshold.
  • Raise the timeout of waitForCollectorToStart from 60s to 150s.

Since 2026-08-29 the e2e tests fail on CI regardless of the branch under test.
The collector container of the daemonset collector does not become ready within
the 60s the test suite waits for the rollout, the BeforeAll node fails and all
103 remaining specs are skipped.

The cause is the resourcedetection processor of the collector, which runs the
detectors eks, ecs, ec2, gcp, azure and aks once per pipeline on startup. It
used to finish in exactly the configured 2s per pipeline, it now takes about
26s:

  last green run (2026-08-28 17:45 UTC)   2.000s   2.001s   2.001s   ready after 6s
  first red run  (2026-08-29 16:40 UTC)  26.233s  26.467s  unfinished

Three pipelines put the daemonset collector at about 80s, which is also close to
the 90s that its default startup probe grants it. The deployment collector has
only one pipeline and still becomes ready, which is why only the daemonset
collector fails.

Nothing in the operator changed. The runner image did: the job runs on a Depot
runner, and between the last green and the first red run the image rolled from
kubectl 1.36.3 to 1.36.4, with go 1.27.0 and kind v0.32.0 unchanged. Depot
reports no incident for that period.

The suspected mechanism is name resolution rather than the metadata endpoints
themselves. The 2s timeout of the resourcedetection processor was honored to the
millisecond before and is now exceeded by more than an order of magnitude, which
fits a blocking DNS lookup: the Go resolver cannot abort an in-flight cgo lookup
when the context is cancelled, so a name server that drops the query for
metadata.google.internal instead of answering NXDOMAIN outlasts the deadline.
The IMDSv2 hop limit, which has to be at least 2 for containers, would explain
the same for the AWS and Azure detectors.

To find out which of the two it is, add
test-resources/bin/probe-instance-metadata-endpoints.sh and run it in the e2e
workflow. It measures name resolution and connectivity to the metadata endpoints
from the three network positions between the runner and a collector pod: the
runner itself, every kind node container and a pod. A probe that runs into its
timeout at one position but answers at the previous one shows at which hop the
responses are dropped.

Until the runner side is understood, keep CI green by granting the collector the
time it currently needs. This is a workaround, not a fix for the slowdown:

- Raise the startup probe of the daemonset collector from 90s to 120s for all
  e2e tests, via operator.collectors.daemonSetProbes.startup.failureThreshold.
- Raise the timeout of waitForCollectorToStart from 60s to 150s.
@basti1302

Copy link
Copy Markdown
Member Author

/run-e2e

@github-actions

Copy link
Copy Markdown
Contributor

🚀 E2E tests triggered

  • Branch: investigate-e2e-failures
  • Triggered by: @basti1302

Results will be posted here once the tests complete. You can also view the workflow runs directly.

@basti1302
basti1302 marked this pull request as ready for review August 29, 2026 20:47
@basti1302
basti1302 requested a review from a team as a code owner August 29, 2026 20:47
@basti1302
basti1302 merged commit c94a444 into main Aug 29, 2026
15 of 16 checks passed
@basti1302
basti1302 deleted the investigate-e2e-failures branch August 29, 2026 20:47
@github-actions github-actions Bot locked and limited conversation to collaborators Aug 29, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant