Skip to content

Commit c0bf562

Browse files
L3n41cclaude
andauthored
[CI] Add Kubernetes memory limits to docker_image_build_otel job (#49233)
### What does this PR do? Adds `KUBERNETES_MEMORY_REQUEST` and `KUBERNETES_MEMORY_LIMIT` (32Gi) to the `docker_image_build_otel` CI job, which was the only OTel integration test job without Kubernetes memory limits. ### Motivation The `docker_image_build_otel` job has a **~12% flaky failure rate** on `main`, caused by OOM kills during `go mod download` of 40+ OTel modules inside a Docker-in-Docker build (see [Slack thread](https://dd.slack.com/archives/CR5TV8QBY/p1775901945411339)). **Root cause analysis** ([posted in thread](https://dd.slack.com/archives/CR5TV8QBY/p1775979407970919?thread_ts=1775901945.411339&cid=CR5TV8QBY)): - Datadog CI data shows **64 failures / 469 successes over 7 days** (~12%), with failures pre-dating the initially suspected PR #48777 (one failure on Apr 9, a day before the PR merged) - **60%** of failures are Kubernetes pod scheduling failures (insufficient memory, unmatched node affinity) - **25%** are silent process kills (exit code 255) during `go mod download` — classic OOM pattern - The two sibling OTel jobs already have memory limits: - `integration_tests_otel`: 16Gi - `datadog_otel_components_ocb_build`: 32Gi + 16 CPU - `docker_image_build_otel` had **none**, relying on runner defaults Setting REQUEST = LIMIT = 32Gi gives the pod a Guaranteed QoS class in Kubernetes, matching the OCB build job which performs the same workload (OCB generation + Go compilation). ### Describe how you validated your changes - The YAML structure matches the existing pattern used by all other jobs in this file - The 32Gi value matches `datadog_otel_components_ocb_build` which does the same work - Validation will come from observing the failure rate drop after merge ### Additional Notes The `.ddot_byoc_oci_build_test` template and `ddot_byoc_binary_build_test_ubuntu2004` job in the same file also lack memory limits and perform similar Docker builds. They may benefit from the same treatment as a follow-up. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent d575f76 commit c0bf562

1 file changed

Lines changed: 3 additions & 0 deletions

File tree

.gitlab/test/integration_test/otel.yml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,9 @@ docker_image_build_otel:
2424
image: registry.ddbuild.io/ci/datadog-agent-buildimages/docker_x64$CI_IMAGE_DOCKER_X64_SUFFIX:$CI_IMAGE_DOCKER_X64
2525
needs: ["integration_tests_otel"]
2626
tags: ["docker-in-docker:amd64"]
27+
variables:
28+
KUBERNETES_MEMORY_REQUEST: 32Gi
29+
KUBERNETES_MEMORY_LIMIT: 32Gi
2730
before_script:
2831
- mkdir -p /tmp/otel-ci
2932
- cp comp/otelcol/collector-contrib/impl/manifest.yaml /tmp/otel-ci/

0 commit comments

Comments
 (0)