Commit c0bf562
[CI] Add Kubernetes memory limits to
### What does this PR do?
Adds `KUBERNETES_MEMORY_REQUEST` and `KUBERNETES_MEMORY_LIMIT` (32Gi) to
the `docker_image_build_otel` CI job, which was the only OTel
integration test job without Kubernetes memory limits.
### Motivation
The `docker_image_build_otel` job has a **~12% flaky failure rate** on
`main`, caused by OOM kills during `go mod download` of 40+ OTel modules
inside a Docker-in-Docker build (see [Slack
thread](https://dd.slack.com/archives/CR5TV8QBY/p1775901945411339)).
**Root cause analysis** ([posted in
thread](https://dd.slack.com/archives/CR5TV8QBY/p1775979407970919?thread_ts=1775901945.411339&cid=CR5TV8QBY)):
- Datadog CI data shows **64 failures / 469 successes over 7 days**
(~12%), with failures pre-dating the initially suspected PR #48777 (one
failure on Apr 9, a day before the PR merged)
- **60%** of failures are Kubernetes pod scheduling failures
(insufficient memory, unmatched node affinity)
- **25%** are silent process kills (exit code 255) during `go mod
download` — classic OOM pattern
- The two sibling OTel jobs already have memory limits:
- `integration_tests_otel`: 16Gi
- `datadog_otel_components_ocb_build`: 32Gi + 16 CPU
- `docker_image_build_otel` had **none**, relying on runner defaults
Setting REQUEST = LIMIT = 32Gi gives the pod a Guaranteed QoS class in
Kubernetes, matching the OCB build job which performs the same workload
(OCB generation + Go compilation).
### Describe how you validated your changes
- The YAML structure matches the existing pattern used by all other jobs
in this file
- The 32Gi value matches `datadog_otel_components_ocb_build` which does
the same work
- Validation will come from observing the failure rate drop after merge
### Additional Notes
The `.ddot_byoc_oci_build_test` template and
`ddot_byoc_binary_build_test_ubuntu2004` job in the same file also lack
memory limits and perform similar Docker builds. They may benefit from
the same treatment as a follow-up.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>docker_image_build_otel job (#49233)1 parent d575f76 commit c0bf562
1 file changed
Lines changed: 3 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
24 | 24 | | |
25 | 25 | | |
26 | 26 | | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
27 | 30 | | |
28 | 31 | | |
29 | 32 | | |
| |||
0 commit comments