ci: pin build_base_venvs memory to prevent cold-cache OOM#18962
ci: pin build_base_venvs memory to prevent cold-cache OOM#18962vlad-scherbich wants to merge 2 commits into
Conversation
build_base_venvs compiles every native extension (Rust _native, Cython, IAST, profiling) at CMAKE_BUILD_PARALLEL_LEVEL=12. When a version's ext_cache is cold -- e.g. a newly added Python version such as 3.15 -- the full 12-way parallel compile peaks well above the ~5Gi the VPA tunes this job down to (observed "MemoryLimit changed from 6Gi to 5012971110"), and the job is OOMKilled (exit 137) deterministically. Pin CPU/memory and disable the VPA, matching the resource requests the native package build jobs (.build_base, "test sdist") already use. This keeps build parallelism intact and only affects cold-cache builds; warm builds barely compile anything.
Codeowners resolved as |
Circular import analysis
|
|
BenchmarksBenchmark execution time: 2026-07-09 21:10:05 Comparing candidate commit 9ff9307 in PR branch Found 0 performance improvements and 3 performance regressions! Performance is the same for 618 metrics, 10 unstable metrics. scenario:iastaspectsospath-ospathbasename_aspect
scenario:iastaspectssplit-rsplit_aspect
scenario:span-start
|
vlad-scherbich
added
the
changelog/no-changelog
There was a problem hiding this comment.
Pull request overview
This PR updates the GitLab CI build_base_venvs job to avoid cold-cache OOMKills by explicitly pinning Kubernetes CPU/memory requests/limits and disabling VPA tuning, aligning its runtime constraints with other native-build jobs in the pipeline.
Changes:
- Add
KUBERNETES_CPU_REQUEST,KUBERNETES_MEMORY_REQUEST, andKUBERNETES_MEMORY_LIMITtobuild_base_venvs. - Set
DD_DISABLE_VPAto prevent VPA from clamping memory during cold-cache native extension builds. - Document the rationale inline in the job’s variables section.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| # Pin resources and disable the VPA. This job compiles every native | ||
| # extension in parallel (CMAKE_BUILD_PARALLEL_LEVEL=12, set above) whenever | ||
| # the ext_cache is cold (e.g. a newly added Python version), which peaks | ||
| # well above the ~5Gi the VPA otherwise tunes it down to and gets OOMKilled. | ||
| # Matches the resource requests used by the native package build jobs. |
There was a problem hiding this comment.
Are those requests defined somewhere within dd-trace-py? If so, could you add a comment there to make sure we also update this file when they're updated?
Description
build_base_venvsis OOMKilled for cold-ext_cachePython versions. Currently this happens forv3.15only, which is new and has no warm cache. This was found while working on #17849.Root cause
KUBERNETES_MEMORY_*and does not disable the VPA, so the autoscaler tunes the limit down to ~5GB, based on the cheap warm-cache history.Changes
Pin CPU/memory and disable the VPA on
build_base_venvs, matching what.build_base,test sdistalready use for the same build targets:Test plan
Additional Notes
ext_cacherestore fails on every run and forces cold compiles every time. That's a runner/ddbuildplatform-side credential issue. Fixing this separately would restores warm-build speed.