Open
Conversation
Contributor
|
Unsigned commits detected! Please sign your commits. For instructions on how to set up GPG/SSH signing and verify your commits, please see GitHub Documentation. |
Contributor
GPU Pre-flight Check ✅GPUs are available for e2e-openshift tests. Proceeding with deployment.
|
kahilam
force-pushed
the
feat/benchmark-phase3-openshift
branch
from
f45fe9b to
b806063
Compare
Contributor
GPU Pre-flight Check ✅GPUs are available for e2e-openshift tests. Proceeding with deployment.
|
Collaborator
Author
|
/benchmark openshift |
Made-with: Cursor
kahilam
force-pushed
the
feat/benchmark-phase3-openshift
branch
from
b806063 to
580a917
Compare
Contributor
GPU Pre-flight Check ✅GPUs are available for e2e-openshift tests. Proceeding with deployment.
|
Made-with: Cursor
Contributor
GPU Pre-flight Check ✅GPUs are available for e2e-openshift tests. Proceeding with deployment.
|
…ken for openshift Made-with: Cursor
Contributor
GPU Pre-flight Check ✅GPUs are available for e2e-openshift tests. Proceeding with deployment.
|
Made-with: Cursor
Contributor
GPU Pre-flight Check ✅GPUs are available for e2e-openshift tests. Proceeding with deployment.
|
Made-with: Cursor
Contributor
GPU Pre-flight Check ✅GPUs are available for e2e-openshift tests. Proceeding with deployment.
|
Collaborator
Author
|
/benchmark openshift |
Made-with: Cursor
Contributor
GPU Pre-flight Check ✅GPUs are available for e2e-openshift tests. Proceeding with deployment.
|
Made-with: Cursor
Contributor
GPU Pre-flight Check ✅GPUs are available for e2e-openshift tests. Proceeding with deployment.
|
Made-with: Cursor
Contributor
GPU Pre-flight Check ✅GPUs are available for e2e-openshift tests. Proceeding with deployment.
|
Made-with: Cursor
Contributor
GPU Pre-flight Check ✅GPUs are available for e2e-openshift tests. Proceeding with deployment.
|
Made-with: Cursor
Contributor
GPU Pre-flight Check ✅GPUs are available for e2e-openshift tests. Proceeding with deployment.
|
Made-with: Cursor
Contributor
GPU Pre-flight Check ✅GPUs are available for e2e-openshift tests. Proceeding with deployment.
|
Made-with: Cursor
Contributor
Benchmark: scale-up-latency (OpenShift)
Environment
|
The deploy script already enables flow control via the ENABLE_EXPERIMENTAL_FLOW_CONTROL_LAYER env var on the EPP. Patching the EPP with --config-file on v0.5.0-rc.1 causes it to restart and break Gateway routing (HTTP 500). The scale-up latency test proves the Gateway works when the EPP is left untouched. - Skip ensureEPPConfig() call so the EPP is not modified - Restore direct vLLM fallback as safety net if Gateway still fails - Keep EPP config helpers in codebase for future use - All EPP queue metric sampling is retained Made-with: Cursor
The deploy script sets ENABLE_EXPERIMENTAL_FLOW_CONTROL_LAYER env var on the EPP. Using --config-file with featureGates: [flowControl] caused a conflict on v0.5.0-rc.1 that broke Gateway routing (HTTP 500). New approach: - Use --config-text to pass EndpointPickerConfig inline (no volume mount) - Remove the deprecated ENABLE_EXPERIMENTAL_FLOW_CONTROL_LAYER env var since config-text featureGates supersede it - Wait for Gateway health after EPP rollout (5min timeout) - Gateway is now a hard requirement (no fallback to direct vLLM) - Scorer weights: queue=2, kv-cache=2, prefix-cache=3 Made-with: Cursor
The Helm chart already deploys the EPP with --config-file pointing to its own ConfigMap (with scorer weights 2/2/3). Adding a second --config-file or --config-text flag broke the EPP and caused Gateway HTTP 500. New approach: - Find the EPP deployment's existing ConfigMap volume - Update the ConfigMap data to add featureGates: [flowControl] - Trigger a rollout restart via annotation (no arg/volume/env changes) - Wait for Gateway health after EPP restart - Gateway is a hard requirement — no fallback to direct vLLM Made-with: Cursor
Contributor
Benchmark: scale-up-latency (OpenShift)
Environment
|
The CI workflow sets E2E_TESTS_ENABLED=true, which causes install.sh to already configure the EPP with: - Image v0.5.0-rc.1 - ENABLE_EXPERIMENTAL_FLOW_CONTROL_LAYER=true env var - ConfigMap with scorer weights queue=2, kv-cache=2, prefix-cache=3 Any modification to the EPP (adding featureGates to config, adding --config-text, etc.) conflicts with the existing env var and breaks Gateway routing (HTTP 500). Replace ensureEPPConfig with verifyEPPConfig that only inspects and logs the EPP state without modifying it. Gateway connectivity is validated with a 5-minute retry before benchmark starts. Made-with: Cursor
Contributor
Benchmark: scale-up-latency (OpenShift)
Environment
|
The Gateway returns HTTP 500 with empty body even when EPP is correctly configured (flow control enabled, weights 2/2/3, pod Running/Ready). This is a pre-existing infrastructure issue — not caused by our EPP modifications. Add diagnostics to capture in a single CI run: - EPP pod logs (last 50 lines) after failure - Gateway/Istio pod logs (last 30 lines) - All service ports (not just first port) - InferencePool and InferenceModel resources (via unstructured client) - Verbose curl output with response body and debug headers Made-with: Cursor
Contributor
Benchmark: scale-up-latency (OpenShift)Environment
|
Contributor
Benchmark: scale-up-latency (OpenShift)
Environment
|
The cluster's Istio 1.29 only watches inference.networking.k8s.io/v1 InferencePool resources, but the v0.3.0 llm-d guide creates them with inference.networking.x-k8s.io/v1alpha2. This caused Istio to ignore the InferencePool entirely, resulting in cluster_not_found errors from Envoy. install.sh now auto-detects when the cluster supports the v1 CRD and patches the gaie values before helmfile deploy. The HTTPRoute backendRef group is also updated to match. Made-with: Cursor
Contributor
Benchmark: scale-up-latency (OpenShift)
Benchmark: prefill-heavy-workload (OpenShift)
HPA Replica Timeline (44 snapshots)
WVA Replica Timeline (48 snapshots)
Dashboard Panels (4)prefill comparison
prefill metrics timeline
prefill percentiles
prefill replica timeline
Environment
|
- Add model_id workflow_dispatch input so benchmarks can be triggered with any HuggingFace model (default: unsloth/Meta-Llama-3.1-8B) - Generate per-autoscaler PDF reports (3-page) matching the colleague's benchmark report format: config summary, time-series charts, and percentile distributions - Show model name dynamically in run-name and PR comment Made-with: Cursor
Contributor
Benchmark: scale-up-latency (OpenShift)
Benchmark: prefill-heavy-workload (OpenShift)
HPA Replica Timeline (45 snapshots)
WVA Replica Timeline (45 snapshots)
Dashboard Panels (4)prefill comparison
prefill metrics timeline
prefill percentiles
prefill replica timeline
Environment
|
- HPA test now creates VA(min=1, max=2, cost=10) + HPA(min=1, max=10) to match colleague's setup instead of pure CPU-based HPA - WVA test cost changed from 30.0 to 10.0 for consistency - Added model_id, va_config, hpa_config, achieved_rps, error_count, incomplete_count fields to result JSON - Enhanced PDF reports with detailed autoscaler configuration section, error/incomplete request tracking, and achieved RPS - PR comment table now includes failed/incomplete requests and RPS rows Made-with: Cursor
The external metrics API (prometheus-adapter) can be transiently unavailable, causing all benchmarks to fail. Change the check from a hard failure to a best-effort warning with diagnostics, so the benchmark runs and collects data even when HPA cannot scale. Made-with: Cursor
KEDA on the OpenShift cluster continuously reclaims the external.metrics.k8s.io APIService, preventing prometheus-adapter from serving wva_desired_replicas. The existing guard only ran during the deploy step and was dead by the time tests started. Add a background guard loop that re-patches the APIService every 8 seconds during the actual benchmark run so HPA can scale. Made-with: Cursor
The cluster already has a working prometheus-adapter setup in workload-variant-autoscaler-monitoring with wva_desired_replicas rules configured. Using SCALER_BACKEND=prometheus-adapter was deploying a second adapter and re-patching the APIService, which then got reclaimed by KEDA, breaking all external metrics. Switch to SCALER_BACKEND=none to preserve the existing working external metrics API setup. Made-with: Cursor
The throughput/ttft/itl fields use omitempty in Go — when GuideLLM metric extraction fails, these keys are absent from the JSON results. Add a safe accessor helper and use .get() throughout the plotting code to handle missing fields gracefully. Made-with: Cursor
The APIService guard patch was failing with: "spec.insecureSkipTLSVerify: Invalid value: true: may not be true if caBundle is present" KEDA sets a caBundle when it reclaims the APIService, which is mutually exclusive with insecureSkipTLSVerify=true. Adding "caBundle": null to the merge patch clears it before setting insecureSkipTLSVerify, matching the state that worked on April 2. Also switches SCALER_BACKEND back to prometheus-adapter and re-adds the APIService guard to the CI run step. Made-with: Cursor
- TTFT mean in PR comment showed "ms" but value was already divided by 1000 (should be "s") - Achieved RPS was always 0.00 because GuideLLM may not expose rate.completed_rate; add fallback: completed_requests / duration Made-with: Cursor
Contributor
Benchmark: scale-up-latency (OpenShift)
Benchmark: prefill-heavy-workload (OpenShift)
HPA Replica Timeline (44 snapshots)
WVA Replica Timeline (45 snapshots)
Dashboard Panels (4)prefill comparison
prefill metrics timeline
prefill percentiles
prefill replica timeline
Environment
|
…aults The benchmark was deploying vLLM with --max-num-seqs=5 (only 5 concurrent requests per pod), causing 2-3% KV cache utilization and ~1 RPS instead of the expected 60-100% KV cache and ~9 RPS. Removing this allows vLLM to use its default (256), matching the colleague's benchmark configuration. Also aligns WVA saturation thresholds (kvSpareTrigger, queueSpareTrigger) to chart defaults (0.1, 3) to match the colleague's setup. Made-with: Cursor
Contributor
Benchmark: scale-up-latency (OpenShift)
Benchmark: prefill-heavy-workload (OpenShift)
HPA Replica Timeline (46 snapshots)
WVA Replica Timeline (45 snapshots)
Dashboard Panels (4)prefill comparison
prefill metrics timeline
prefill percentiles
prefill replica timeline
Environment
|
V1 analyzer scales by +1 replica per 30s cycle and blocks during pod transitions, limiting scaling to ~3 replicas in a 600s test. V2 uses demand-based calculation (ceil(requiredCapacity / perReplicaCapacity)) and can jump to the needed replica count in one decision, matching the colleague's benchmark behavior. Made-with: Cursor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.