Benchmark Phase 1: Enable framework and chatops action for benchmarking (#900)

* Feat: Add benchmark framework with scale-up-latency scenario and /benchmark kind ChatOps trigger

Add test/benchmark/ package with a Ginkgo-based scale-up-latency benchmark
that measures autoscaler performance through 4 phases: baseline, spike,
sustained, and cooldown. Collects scale-up/down latency, max replicas,
KV cache usage, queue depth, and replica oscillation via Prometheus range
queries.

- test/benchmark/suite_test.go: test suite with Prometheus port-forward setup
- test/benchmark/config.go: env-based config with tunable phase durations
- test/benchmark/benchmark_test.go: 4-phase ordered scenario using e2e fixtures
- test/benchmark/prometheus.go: QueryRangeAvg helper for Prometheus range queries
- test/utils/e2eutils.go: export PrometheusClient.API() for range queries
- .github/workflows/ci-benchmark.yaml: /benchmark kind ChatOps workflow
- Makefile: add test-benchmark and test-benchmark-with-setup targets,
  exclude benchmark from unit tests

* Feat: Add ephemeral Grafana snapshot capture to benchmark framework

Deploy an in-cluster Grafana instance during benchmarks via helm upgrade
on the existing kube-prometheus-stack, create a 5-panel dashboard
(replicas, desired replicas, KV cache, queue depth, saturation metrics),
and capture a snapshot covering the full benchmark time range.

- deploy/grafana/: Helm values (anonymous admin, Prometheus datasource,
  sidecar dashboard provisioning) and dashboard JSON
- test/benchmark/grafana.go: DeployGrafana, NewGrafanaClient,
  CreateSnapshot, RenderPanel
- test/benchmark/config.go: GrafanaEnabled, GrafanaSnapshotFile fields
- test/benchmark/suite_test.go: wire Grafana deploy/teardown into
  BeforeSuite/AfterSuite
- test/benchmark/benchmark_test.go: capture snapshot in AfterAll,
  include URL in results JSON
- ci-benchmark.yaml: enable Grafana, upload snapshot, link in PR comment

* Feat: Export Grafana snapshot JSON and render panels to PNG

Persist benchmark Grafana data to GitHub Actions artifacts so results
survive the ephemeral Kind cluster:

1. Snapshot JSON export: fetch full snapshot via GET /api/snapshots/:key
   and save as re-importable JSON (POST to any Grafana to restore).
2. Panel PNG rendering: enable grafana-image-renderer plugin and render
   all 5 dashboard panels to individual PNG files.

- grafana.go: CreateSnapshot now returns SnapshotResult (key+URL),
  add ExportSnapshotJSON and RenderAllPanels methods
- config.go: add GrafanaSnapshotJSONFile and GrafanaPanelDir fields
- benchmark-dashboard.json: add explicit panel IDs for stable rendering
- benchmark-grafana-values.yaml: enable imageRenderer with resource limits
- ci-benchmark.yaml: pass new env vars, upload JSON + PNGs as artifacts

* Fix wrong Prometheus metric name in benchmark (gpu_cache → kv_cache)

The benchmark queried vllm:gpu_cache_usage_perc which doesn't exist.
The actual metric emitted by the vLLM simulator is vllm:kv_cache_usage_perc
as defined in internal/constants/metrics.go.

* Fix staticcheck SA4004: remove unconditionally terminated loop

Use direct index access for the single Grafana pod instead of
iterating with a for-range that always returns on first iteration.

* Fix Grafana image pull: use docker.io with Kind pre-load

Grafana only publishes images to Docker Hub, not quay.io.
Pre-load the image into Kind before running the benchmark
and set imagePullPolicy: IfNotPresent to avoid runtime pulls.

* Match benchmark spike phase to e2e parallel load test flow

- Add 30s ramp-up wait after load generation starts (like e2e)
- Monitor VA status for scale-up intent before checking deployment
- Monitor HPA for scale-up confirmation (separate stage)
- Monitor deployment for actual replica changes (10m timeout)
- Add detailed diagnostics every 30s: VA status, HPA conditions,
  HPA metrics, load pod phases, Prometheus metric values
- Clean up existing jobs before creating new ones (like e2e)
- Log service endpoint count during readiness check

* Fix benchmark load generation: reduce workers to 1, add failed pod log collection

Match e2e's maxSingleReplicaWorkers=1 for single-replica deployments to avoid
overwhelming the simulator's max-num-seqs queue. Also collect pod logs when load
pods fail to aid diagnosis of connectivity/runtime issues.

* Add in-cluster connectivity probe to diagnose load pod failures

The load pod cannot connect to the service (24 attempts, all fail).
Adding a diagnostic probe pod that runs curl -v and DNS resolution
to determine if the issue is DNS, routing, or HTTP status code mismatch.

* Add Grafana image renderer sidecar for panel PNG export

The base grafana:11.4.0 image has no image renderer installed, causing
"no image renderer available/installed" when rendering panel PNGs.
Adding grafana-image-renderer:3.11.6 as a sidecar container and
pre-loading it into Kind in CI.

* Fix Grafana dashboard to use actual WVA metric names

- Panel 1: Replace kube_deployment_spec_replicas (requires kube-state-metrics)
  with wva_desired_replicas and wva_current_replicas
- Panel 2: Replace wva_desired_replicas with wva_desired_ratio (more useful)
- Panel 5: Replace non-existent wva_saturation_score/wva_capacity_score
  with wva_desired/current_replicas and scaling rate
- Fix label references: variant_name not variant

* Embed Grafana panel images in PR comment via release assets

Upload rendered panel PNGs as prerelease assets and embed them
directly in the PR comment under a collapsible details section.
Also fix dashboard queries to use actual WVA metric names.

* Fix Grafana dashboard datasource for file-provisioned mode

File-provisioned dashboards do not resolve ${DS_PROMETHEUS} template
variables. Remove uid from all panel datasource references so Grafana
auto-selects the default Prometheus datasource.

* Fix WVA metric queries and CI permissions for panel images

WVA metrics get namespace="workload-variant-autoscaler-system" from
Prometheus scraping (not the VA namespace), so remove the
namespace=~"llm-d.*" filter from WVA metric queries in the dashboard.
vLLM metrics keep the filter since they are scraped from llm-d-sim.

Change benchmark-kind job permissions to contents:write so the workflow
can create GitHub releases to host rendered panel PNG images.

* Route benchmark load through Gateway/EPP (full llm-d stack)

Change benchmark to send load through the Gateway service instead of
directly to the model service. Traffic now flows through the full
llm-d stack: Gateway → HTTPRoute → InferencePool → EPP → model pods.

The benchmark model service pods already have the
llm-d.ai/inferenceServing label, so the InferencePool discovers
them automatically.

Add GatewayServiceName/GatewayServicePort config fields (env:
GATEWAY_SERVICE_NAME, GATEWAY_SERVICE_PORT) and EPP/Gateway
readiness checks in BeforeSuite.

* Fix EPP pod label selector to match inferencepool chart

The inferencepool chart labels EPP pods with
inferencepool=<epp-service-name>, not
app.kubernetes.io/name=inferencepool. Use the same label selector
as the e2e scale-from-zero test.

* Address PR #900 review comments: dedup config, extract setup, streamline suite

- Extract shared test config to test/testconfig/config.go; both E2EConfig
  and BenchmarkConfig now embed testconfig.SharedConfig (comment 5)
- Move BenchmarkResults to results.go for reuse across scenarios (comment 6)
- Rename benchmark_test.go to scale_up_latency_benchmark_test.go (comment 7)
- Add scenario description comment (comment 8)
- Extract common setup to setup_test.go with SetupBenchmarkScenario(),
  CaptureResultsAndGrafana(), GatewayTargetURL() (comment 9)
- Remove redundant infra verification from BeforeSuite — install.sh
  already verifies WVA, Gateway, EPP, Prometheus (comment 12)
- Move Grafana deployment to install.sh via INSTALL_GRAFANA env var;
  remove DeployGrafana() call from suite_test.go (comments 2, 13)
- Remove "(matches e2e flow)" comments (comment 10)
- Fix vaAttempt % 3 spacing (comment 11)

* Remove assertions from benchmark phases — observe, don't test

Benchmark phases should only observe and record metrics, not assert
on replica counts. Prometheus already monitors replicas for all
deployments.

- Phase 2: Replace VA/HPA/deployment Eventually blocks with simple
  observation loop that logs and records scale-up time + max replicas
- Phase 3: Replace Expect on deployment Get with warning log
- Phase 4: Replace Expect/IsNotFound on deployment Get with warning log
- Remove DeployGrafana and dumpGrafanaDiagnostics from grafana.go
  (Grafana deployed via install.sh INSTALL_GRAFANA env var)
- Remove unused imports: fmt, corev1, errors

* Pre-check Grafana service existence before SetUpPortForward

SetUpPortForward uses Expect() internally which fatally fails the
entire suite if the service is not found. Add a pre-check in
NewGrafanaClient that returns an error gracefully, allowing the
suite to continue without Grafana when the service is missing.

* Create benchmark-dashboard ConfigMap before Grafana deployment

The Grafana deployment references a benchmark-dashboard ConfigMap
volume but deploy_benchmark_grafana() never created it, causing
the pod to fail with CreateContainerConfigError.

* Address review: make setup helpers private and add GinkgoHelper

- Rename SetupBenchmarkScenario → setupBenchmarkScenario (private)
- Rename CaptureResultsAndGrafana → captureResultsAndGrafana (private)
- Rename GatewayTargetURL → gatewayTargetURL (private)
- Add GinkgoHelper() to setup/capture functions so failures
  report at the caller's line instead of inside the helper
- Update doc comment to clarify fresh resource creation

Author: ev-shindin

.github/workflows/ci-benchmark.yaml

@@ -91,7 +91,7 @@ vet: ## Run go vet against code.
 
 .PHONY: test
 test: manifests generate fmt vet setup-envtest helm ## Run tests.
-	KUBEBUILDER_ASSETS="$(shell $(ENVTEST) use $(ENVTEST_K8S_VERSION) --bin-dir $(LOCALBIN) -p path)" PATH="$(LOCALBIN):$(PATH)" go test $$(go list ./... | grep -v /e2e) -coverprofile cover.out
+	KUBEBUILDER_ASSETS="$(shell $(ENVTEST) use $(ENVTEST_K8S_VERSION) --bin-dir $(LOCALBIN) -p path)" PATH=$(LOCALBIN):$(PATH) go test $$(go list ./... | grep -v /e2e | grep -v /benchmark) -coverprofile cover.out
 
 # Creates a multi-node Kind cluster
 # Adds emulated GPU labels and capacities per node
@@ -269,7 +269,32 @@ test-e2e-smoke-with-setup: deploy-e2e-infra test-e2e-smoke
 # Convenience target that deploys infra + runs full test suite.
 # Set DELETE_CLUSTER=true to delete Kind cluster after tests (default: keep cluster for debugging).
 .PHONY: test-e2e-full-with-setup
-test-e2e-full-with-setup: deploy-e2e-infra test-e2e-full 
+test-e2e-full-with-setup: deploy-e2e-infra test-e2e-full
+
+# Benchmark targets
+.PHONY: test-benchmark
+test-benchmark: manifests generate fmt vet ## Run benchmark tests (scale-up-latency scenario)
+	@echo "Running benchmark tests..."
+	KUBECONFIG=$(KUBECONFIG) \
+	ENVIRONMENT=$(ENVIRONMENT) \
+	WVA_NAMESPACE=$(CONTROLLER_NAMESPACE) \
+	LLMD_NAMESPACE=$(E2E_EMULATED_LLMD_NAMESPACE) \
+	MONITORING_NAMESPACE=$(E2E_MONITORING_NAMESPACE) \
+	USE_SIMULATOR=$(USE_SIMULATOR) \
+	SCALER_BACKEND=$(SCALER_BACKEND) \
+	MODEL_ID=$(MODEL_ID) \
+	go test ./test/benchmark/ -timeout 30m -v -ginkgo.v \
+		-ginkgo.label-filter="benchmark"; \
+	TEST_EXIT_CODE=$$?; \
+	echo ""; \
+	echo "=========================================="; \
+	echo "Benchmark execution completed. Exit code: $$TEST_EXIT_CODE"; \
+	echo "=========================================="; \
+	exit $$TEST_EXIT_CODE
+
+# Convenience target that deploys infra + runs benchmark tests.
+.PHONY: test-benchmark-with-setup
+test-benchmark-with-setup: deploy-e2e-infra test-benchmark
 
 .PHONY: lint
 lint: golangci-lint ## Run golangci-lint linter

Makefile

@@ -0,0 +1,212 @@
+{
+  "annotations": {
+    "list": [
+      {
+        "builtIn": 1,
+        "datasource": { "type": "grafana", "uid": "-- Grafana --" },
+        "enable": true,
+        "hide": true,
+        "iconColor": "rgba(0, 211, 255, 1)",
+        "name": "Annotations & Alerts",
+        "type": "dashboard"
+      }
+    ]
+  },
+  "editable": true,
+  "fiscalYearStartMonth": 0,
+  "graphTooltip": 1,
+  "id": null,
+  "links": [],
+  "panels": [
+    {
+      "id": 1,
+      "title": "Deployment Replicas",
+      "type": "timeseries",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 0 },
+      "datasource": { "type": "prometheus" },
+      "fieldConfig": {
+        "defaults": {
+          "color": { "mode": "palette-classic" },
+          "custom": {
+            "drawStyle": "line",
+            "lineWidth": 2,
+            "fillOpacity": 10,
+            "pointSize": 5,
+            "showPoints": "auto",
+            "spanNulls": true
+          },
+          "unit": "short",
+          "min": 0
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "wva_desired_replicas",
+          "legendFormat": "desired {{variant_name}}",
+          "refId": "A"
+        },
+        {
+          "expr": "wva_current_replicas",
+          "legendFormat": "current {{variant_name}}",
+          "refId": "B"
+        }
+      ],
+      "options": { "legend": { "displayMode": "list", "placement": "bottom" } }
+    },
+    {
+      "id": 2,
+      "title": "WVA Desired Ratio",
+      "type": "timeseries",
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 0 },
+      "datasource": { "type": "prometheus" },
+      "fieldConfig": {
+        "defaults": {
+          "color": { "mode": "palette-classic" },
+          "custom": {
+            "drawStyle": "line",
+            "lineWidth": 2,
+            "fillOpacity": 10,
+            "spanNulls": true
+          },
+          "unit": "short",
+          "min": 0
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "wva_desired_ratio",
+          "legendFormat": "ratio {{variant_name}}",
+          "refId": "A"
+        }
+      ],
+      "options": { "legend": { "displayMode": "list", "placement": "bottom" } }
+    },
+    {
+      "id": 3,
+      "title": "KV Cache Usage",
+      "type": "timeseries",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 8 },
+      "datasource": { "type": "prometheus" },
+      "fieldConfig": {
+        "defaults": {
+          "color": { "mode": "palette-classic" },
+          "custom": {
+            "drawStyle": "line",
+            "lineWidth": 2,
+            "fillOpacity": 20,
+            "spanNulls": true
+          },
+          "unit": "percentunit",
+          "min": 0,
+          "max": 1,
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              { "color": "green", "value": null },
+              { "color": "yellow", "value": 0.7 },
+              { "color": "red", "value": 0.9 }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "vllm:kv_cache_usage_perc{namespace=~\"llm-d.*\"}",
+          "legendFormat": "{{pod}}",
+          "refId": "A"
+        },
+        {
+          "expr": "avg(vllm:kv_cache_usage_perc{namespace=~\"llm-d.*\"})",
+          "legendFormat": "avg",
+          "refId": "B"
+        }
+      ],
+      "options": { "legend": { "displayMode": "list", "placement": "bottom" } }
+    },
+    {
+      "id": 4,
+      "title": "Queue Depth (Requests Waiting)",
+      "type": "timeseries",
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 8 },
+      "datasource": { "type": "prometheus" },
+      "fieldConfig": {
+        "defaults": {
+          "color": { "mode": "palette-classic" },
+          "custom": {
+            "drawStyle": "line",
+            "lineWidth": 2,
+            "fillOpacity": 10,
+            "spanNulls": true
+          },
+          "unit": "short",
+          "min": 0
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "vllm:num_requests_waiting{namespace=~\"llm-d.*\"}",
+          "legendFormat": "{{pod}} waiting",
+          "refId": "A"
+        },
+        {
+          "expr": "vllm:num_requests_running{namespace=~\"llm-d.*\"}",
+          "legendFormat": "{{pod}} running",
+          "refId": "B"
+        }
+      ],
+      "options": { "legend": { "displayMode": "list", "placement": "bottom" } }
+    },
+    {
+      "id": 5,
+      "title": "Scaling Activity",
+      "type": "timeseries",
+      "gridPos": { "h": 8, "w": 24, "x": 0, "y": 16 },
+      "datasource": { "type": "prometheus" },
+      "fieldConfig": {
+        "defaults": {
+          "color": { "mode": "palette-classic" },
+          "custom": {
+            "drawStyle": "line",
+            "lineWidth": 2,
+            "fillOpacity": 10,
+            "spanNulls": true
+          },
+          "unit": "short",
+          "min": 0
+        },
+        "overrides": []
+      },
+      "targets": [
+        {
+          "expr": "wva_desired_replicas",
+          "legendFormat": "desired {{variant_name}}",
+          "refId": "A"
+        },
+        {
+          "expr": "wva_current_replicas",
+          "legendFormat": "current {{variant_name}}",
+          "refId": "B"
+        },
+        {
+          "expr": "rate(wva_replica_scaling_total[2m])",
+          "legendFormat": "scaling rate {{variant_name}} {{direction}}",
+          "refId": "C"
+        }
+      ],
+      "options": { "legend": { "displayMode": "list", "placement": "bottom" } }
+    }
+  ],
+  "schemaVersion": 39,
+  "tags": ["benchmark", "wva", "autoscaling"],
+  "templating": { "list": [] },
+  "time": { "from": "now-30m", "to": "now" },
+  "timepicker": {},
+  "timezone": "utc",
+  "title": "WVA Benchmark: Scale-Up Latency",
+  "uid": "wva-benchmark-scaleup",
+  "version": 1
+}