test(perf): add many-NodePools e2e perf test (500 NodePools scale-in/out) - #3248
Draft
nathangeology wants to merge 3 commits into
Draft
test(perf): add many-NodePools e2e perf test (500 NodePools scale-in/out)#3248nathangeology wants to merge 3 commits into
nathangeology wants to merge 3 commits into
Conversation
Introduce common.LatencyHarness alongside KarpenterMetricsPoller. The harness scrapes /metrics at phase start and stop, then reduces per-series histogram bucket deltas into percentile stats (P50/P90/P95/P99, plus bucket truncation rate) and counter deltas over the observation window. The scrape helper is shared with KarpenterMetricsPoller (previously the poller inlined the API-server pod-proxy fetch + parse). Both callers now funnel through scrapeKarpenterMetricFamilies. Includes common.LatencySidecar (JSON schema for the artifact written alongside PerformanceReport when OUTPUT_DIR is set) and common.WriteLatencySidecar so performance-suite specs share one on-disk shape rather than each declaring its own.
…out) Adds an e2e performance test that puts pressure on the many-NodePools axis specifically. Provisions 500 NodePools with the standard test template, drives a scale-in perturbation (2 pods per NodePool then 1 pod per NodePool), then a scale-out perturbation (0 then 2 pods per NodePool). Captures Karpenter controller CPU / memory / latency samples through each phase. The LatencyHarness substrate lands here as an in-place copy from the Balanced-consolidation perf work; a follow-up rebase consumes the shared common/ helper once the sibling PR merges. Threshold assertions route through AssertUpperBound / AssertLowerBound / AssertTotalTimeUpperBound with keys manyNodePools/<phase>. The KARPENTER_PERF_THRESHOLDS env-var overrides from kubernetes-sigs#3165 apply as expected. Signed-off-by: Nathaniel Jones <jonesflp@amazon.com>
kubernetes-prow
Bot
added
the
do-not-merge/work-in-progress
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: nathangeology The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
kubernetes-prow
Bot
added
cncf-cla: yes
Wires the new Many NodePools performance spec into the kind-perf-e2e workflow so the DescribeTable sweep runs alongside the other scenarios on every push to main. Focus regex matches the Ginkgo Context in many_nodepools_test.go. Fan-out cost is one additional matrix row against the existing seven scenarios. If the largest sweep entry (500 NodePools) overflows the per-matrix-job wall clock on the first run, splitting the DescribeTable into two Its and adding a second matrix row is the mitigation path. Signed-off-by: Nathaniel Jones <jonesflp@amazon.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #N/A
Description
Adds an e2e performance test that puts pressure on the many-NodePools axis specifically. The scenario provisions 500 NodePools with the standard test template, drives a scale-in perturbation (2 pods per NodePool then 1 pod per NodePool), then a scale-out perturbation (0 then 2 pods per NodePool), and captures Karpenter controller CPU / memory / latency samples through each phase.
Test file:
test/suites/performance/many_nodepools_test.go. NodePool template matches the existing GinkgoDescribe("Performance", Label(debug.NoWatch), func() { ... })shape used bybasic_test.goandwide_deployments_test.go.The LatencyHarness substrate lands here as an in-place copy from the Balanced-consolidation perf work in #3247; a follow-up rebase consumes the shared
common/helper once #3247 merges.Threshold assertions route through
AssertUpperBound/AssertLowerBound/AssertTotalTimeUpperBoundwith keysmanyNodePools/<phase>, so theKARPENTER_PERF_THRESHOLDSenv-var overrides from #3165 apply as expected.How was this change tested?
go test -c ./test/suites/performance/produces a test binary (compile clean).make verifyrunning (>2 min) — will retry in fork CI on the pushed branch.AI Disclosure
I used an LLM to parallel program with me on this. That included drafting the PR description, the ordering of test phases against the scoping doc, and the LatencyHarness cherry-pick. The scoping and test-shape decisions came from me; the LLM did not invent the 500 x 2 / 1 / 0->2 scenario.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.