TEST/PERF: Compare device perftest BW/Lat against the PR base - #11531
Open
Alexey-Rivkin wants to merge 1 commit into
Open
TEST/PERF: Compare device perftest BW/Lat against the PR base#11531Alexey-Rivkin wants to merge 1 commit into
Alexey-Rivkin wants to merge 1 commit into
Conversation
Alexey-Rivkin
force-pushed
the
device-perf-regression-compare
branch
3 times, most recently
from
June 7, 2026 21:19
705f3ba to
76a0752
Compare
Contributor
Author
|
/build |
3 similar comments
Contributor
Author
|
/build |
|
/build |
Contributor
Author
|
/build |
Alexey-Rivkin
force-pushed
the
device-perf-regression-compare
branch
from
August 5, 2026 13:21
76a0752 to
66236d9
Compare
Alexey-Rivkin
marked this pull request as ready for review
August 5, 2026 13:44
Alexey-Rivkin
marked this pull request as draft
August 5, 2026 13:52
The GPU device-API (GDA / gdaki) perftests run in CI but only ever ran once on the PR, with no comparison, so a bandwidth/latency regression went unnoticed (e.g. the ~34% GDA bandwidth drop in openucx#11424). The OSU perf pipeline can't cover this: OSU is host-initiated and never enters the GPU-kernel-initiated path, and its perf nodes lack GDA hardware. Add a before/after check inside the existing GPU leg: build the PR base (HEAD^1 of the merge ref) and run test_types_ucp_device_cuda on both the base and head builds, interleaving the runs (alternating order) so both see the same node load, then compare per-test BW/Lat. A regression above the threshold (default 15%, looser than the OSU 5% since the GPU CI nodes are shared and noisier) is reported as a build warning and marks the stage SucceededWithIssues. The check is advisory and never fails the job - base build or base run failures only skip the comparison; blocking mode can follow once the signal-vs-noise ratio on the shared nodes is known. master / non-PR builds (no HEAD^2) just run once for coverage, unchanged. Signed-off-by: Alexey Rivkin <arivkin@nvidia.com>
Alexey-Rivkin
force-pushed
the
device-perf-regression-compare
branch
from
August 5, 2026 13:55
66236d9 to
391b9e8
Compare
Alexey-Rivkin
marked this pull request as ready for review
August 6, 2026 10:40
Contributor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What?
Add base-vs-head BW/Lat regression check for the GPU device-API (GDA) perftests (
test_types_ucp_device_cuda).Why?
They run in CI but only on the PR, with no comparison, so regressions slip through (e.g. the ~34% GDA bandwidth drop in #11424). The OSU perf pipeline can't cover this: it's host-initiated and its nodes lack GDA.
How?
On a PR build, build the base branch too, run the device config on both the base and head branches interleaved (so both see the same node load on the shared GPU CI nodes), and compare. A regression above the threshold is reported as a build warning - the check is advisory and non-blocking. The threshold is loose (15%) since these nodes are shared; blocking mode can follow once the signal-vs-noise ratio on the shared nodes is known.