CI: kill leaf jobs when aborting stale PR dispatchers#11491
Draft
Alexey-Rivkin wants to merge 1 commit into
Draft
CI: kill leaf jobs when aborting stale PR dispatchers#11491Alexey-Rivkin wants to merge 1 commit into
Alexey-Rivkin wants to merge 1 commit into
Conversation
Contributor
Author
|
/build |
Contributor
Author
Alexey-Rivkin
force-pushed
the
ci/abort-stale-with-leaves
branch
from
3c2a30a to
2abcb24
Compare
Contributor
Author
|
/build |
Contributor
Author
|
Verified by double /build, e.g.: |
Contributor
Author
|
/build |
1 similar comment
Contributor
Author
|
/build |
The OSS dispatcher already aborts the prior dispatcher build for the same PR, but the 8 leaf jobs it spawned via `build job: ..., wait: true` keep running until they finish naturally. That ties up build executors every time someone pushes a new commit to a PR. Port the pattern NIXL uses: scrape each stale dispatcher's console log for `Starting building: <name> #<num>` lines to find its children, kill the leaves first so the dispatcher's wait unblocks, then kill the dispatcher. Use `doKill()` instead of `doStop()` for a hard stop, and retry up to 3 times with a 5s gap so builds caught mid-startup still get torn down. Failures inside the abort block are still swallowed and logged - we never want stale cleanup to break a fresh build.
Alexey-Rivkin
force-pushed
the
ci/abort-stale-with-leaves
branch
from
2abcb24 to
5be718f
Compare
Contributor
Author
|
/build |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What?
Make stale-dispatcher cleanup also kill the leaf jobs it spawned, not just the dispatcher.
Why?
doStop()on the dispatcher leaves 8 child jobs (ucx-build-oss, ucx-test-gpu-oss, etc.) running until they finish naturally. Wastes executors on every PR push.How?
Scrape stale dispatcher's console for Starting building: ... #N to find leaves,
doKill()leaves first then dispatcher, retry 3x with 5s gap.