PKG-1427: GC leaked docker buildx builders on cloud.cd workers#371
Closed
nogueiraanderson wants to merge 3 commits into
Closed
PKG-1427: GC leaked docker buildx builders on cloud.cd workers#371nogueiraanderson wants to merge 3 commits into
nogueiraanderson wants to merge 3 commits into
Conversation
- New init.groovy.d task: removes anonymous buildx_buildkit_* containers older than 24h plus their _state volumes on idle docker-labelled workers - Protects named shared builders (multiarch, pmmbuilder); their cache is age-pruned instead of removed - Engine build cache pruned with until=48h; no docker system prune - Layered bounds: per-proc and per-node timeouts, whole-sweep deadline, overlap guard, generation token for idempotent re-evaluation
- All GC logic now lives in dockerBuildxGc.sh (shellcheck-clean);
the groovy is a thin scheduler that re-reads the payload every tick,
so shell-side changes deploy via the S3/SSM sync alone
- Widen cloud's init_groovy_files glob to *.{groovy,sh}; Jenkins only
evaluates *.groovy, shipped .sh files are inert on the master
…t audit - Audit covered inline/freestyle job definitions on all 10 masters (zero buildx hits), jenkins-pipelines master/hetzner/feature branches, and every live-pinned fork branch - Adds multiarch-builder (hetzner branch: pmm rpm-build, rel valkey) and multiarch-wt (pmm-arm64-test branch) to the protect list
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bug
docker buildx create --usewith nobuildx rm; each run leaks a running buildkit container plus a named_statecache volume, both exempt fromdocker system prune, filling cloud.cd worker disks to 100% (builds then fail at image pull).Fix
buildx_buildkit_*containers older than 24h plus their volumes on idle docker-labelled workers, then age-bounded cache prunes.dockerBuildxGc.sh(shellcheck-clean); the groovy is a thin scheduler that re-reads the payload each tick, so shell-side changes deploy via the S3/SSM sync alone.multiarch,multiarch-builder,multiarch-wt,pmmbuilder, from a fleet-wide audit of live-pinned pipeline sources) are protected: never removed, cache pruned only.Tickets