test(perf): add e2e perf tests for balanced consolidation

[nathangeology]

Fixes #N/A

Description

Adds end-to-end performance specs for the Balanced consolidation policy landed in #2962. The specs provide regression coverage for score-driven consolidation decisions on top of the existing basic and wideDeployments suites.

Three artifacts are added under test/:

1. LatencyHarness + LatencySidecar (in test/pkg/environment/common/) is the shared substrate. LatencyHarness scrapes Karpenter's /metrics at phase start and stop and reduces per-series histogram bucket deltas into percentile stats (P50 / P90 / P95 / P99 plus a bucket-truncation rate) and counter deltas over the observation window. LatencySidecar is the on-disk JSON schema written alongside PerformanceReport when OUTPUT_DIR is set. Both are consumed by the new specs so a PerformanceReport and its latency companion pair on disk for offline diff analysis.

2. test/suites/performance/balanced_baseline_marginal_test.go covers two specs. Balanced Baseline It reuses the 1000-pod / 700-pod fixture from basic_test.go, wrapped with the harness so hero histograms (scheduling_decision, voluntary_disruption_decision_evaluation, pods_bound) are captured for both scale-out and consolidation phases. This is the reference distribution the marginal runs compare against; suite-wide WhenEmptyOrUnderutilized applies. Balanced Marginal Move runs two variants that share the same fixture. The small deployment's pod-deletion-cost annotation is the only knob and shifts the pool's total_disruption_cost denominator to push the consolidation score across the 1/k=0.5 threshold. Approved variant runs at cost=0; rejected variant runs at cost=2e9. Both run under ConsolidationPolicyBalanced.

3. test/suites/performance/balanced_churn_heterogeneous_test.go covers churn-chain and heterogeneous NodePool scenarios. The churn-chain spec drives a 400-pod scale-out followed by three scale-in / scale-out rounds under each ConsolidationPolicy; the harness spans the churn window. The heterogeneous spec runs two family-restricted NodePools (KWOK c and m families) with distinct pod-density profiles and observes cross-pool consolidation on scale-down. Per-pool decisions are captured via the same paired-artifact scheme.

The paired-run pattern (WhenEmptyOrUnderutilized first, then Balanced) keeps the second run starting from an AfterEach-clean state so diff analysis stays stable. Neither variant asserts an exact karpenter_consolidation_moves_total count; KWOK timing blurs which candidates land in a given round, so the soft directional check hasScoreSeriesForDecision only logs when the expected decision series is absent.

Provider-specific threshold overrides continue to work through the KARPENTER_PERF_THRESHOLDS env var from #3165. The threshold-lift companion for the existing basic and wideDeployments suites is in #3245.

How was this change tested?

make verify passes locally. make presubmit passes including vulncheck. Compile is clean under go test -c ./test/suites/performance/... and unit tests in ./test/pkg/environment/common/... pass. The e2e specs themselves cannot be exercised outside the kind-perf-e2e workflow; fork CI will verify that path after push. Threshold values (for the new specs) were chosen with headroom over observed samples rather than from a full re-baseline capture; the rationale for the accompanying threshold-lift on the pre-existing suites is in #3245.

AI Disclosure

I used an LLM to parallel program with me on this. That included drafting the PR description, running the rebase against upstream/main, and the three small cleanup refactors (histogram reducer complexity, LatencySidecar hoist, and magic-number naming). All benchmark spec logic was authored by me previously; the LLM did not write the perf spec bodies.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[kubernetes-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: nathangeology
Once this PR has been reviewed and has the lgtm label, please assign jmdeal for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment