Fix automatic image tag selection by version#1
Closed
hubunt wants to merge 21 commits into
Closed
Conversation
Reproduce the bug with a template-level test that renders the harness pod and inspects the launch script. Before the fix, the rendered pod args only contain the harness command, so the ConfigMap mounted at /workspace/harnesses is never referenced before llm-d-benchmark.sh runs. Copy scripts from /workspace/harnesses into /usr/local/bin and chmod only the copied targets before executing the existing harness command. This lets the llmdbench-harness-scripts ConfigMap override image-baked harness scripts while preserving the existing launcher path. Signed-off-by: hubunt <150658615+hubunt@users.noreply.github.com>
Removed the '--disable-log-requests' option from the command. Signed-off-by: maugustosilva <marcio.a.silva@ibm.com>
Signed-off-by: Radhika Lakhtakia <rlakhtakia@google.com>
Reproduce the run-only failure with an inference-perf lifecycle fixture, run_metadata.yaml, and a workload config passed through --config_file without any Kubernetes service environment or standup ConfigMap. Before the fix, report conversion crashes while deriving the cluster id because the Kubernetes API certificate fallback can be None. Use run metadata endpoint or namespace as the cluster-id fallback when Kubernetes service details are unavailable, and load the workload config from the parsed --config_file path so run-only/local analysis preserves the native config. Signed-off-by: hubunt <150658615+hubunt@users.noreply.github.com>
Signed-off-by: hubunt <150658615+hubunt@users.noreply.github.com>
This commit completes the migration of the Workload Variant Autoscaler (WVA)
from the deprecated prometheus-adapter to KEDA (Kubernetes Event-driven
Autoscaling).
## Key Changes
### Removed
- prometheus-adapter Helm values template (21_prometheus-adapter-values.yaml.j2)
- HPA template for external metrics (28_wva-hpa.yaml.j2)
- prometheus-adapter references from defaults.yaml
### Added
- KEDA TriggerAuthentication CR template (21_keda-triggerauthentication.yaml.j2)
- KEDA ScaledObject CR template (28_wva-scaledobject.yaml.j2)
- Per-namespace RBAC (ServiceAccount, ClusterRoleBinding)
- Documentation: PR description + migration guide + deployment paths
### Fixed Bugs
- create_prometheus_auth_secret() was missing stack_path parameter
(critical: prevented KEDA TriggerAuthentication from applying)
- Smoketest validator updated to check KEDA ScaledObject instead of
prometheus-adapter Deployment
## Deployment Architecture
The WVA KEDA deployment uses a hybrid Kustomize + Helm approach:
- Kustomize: WVA controller (from upstream overlay)
- Helm: ModelService inference pods (decode/prefill vLLM)
- Jinja2 templates: VariantAutoscaling + ScaledObject CRs
## Test Verification
- Command: llmdbenchmark --spec guides/workload-autoscaling standup \
-p asmalvan-test-2 -m Qwen/Qwen3-0.6B
- Result: All KEDA resources deployed successfully
- ScaledObject (targets decode deployment, min=1, max=10)
- TriggerAuthentication (Prometheus bearer token auth)
- Secret + ServiceAccount + ClusterRoleBinding (RBAC)
- WVA controller (1/1 Ready)
- Model deployment (2 replicas, 2/2 Ready each)
- Tests: 603/603 unit tests passing
## Future Work
Proposal to move completely to Kustomize for unified deployment pipeline:
- Replace Helm rendering with Kustomize generators
- Create local overlays for modelservice vLLM pods
- Consolidate per-namespace RBAC into kustomization.yaml
- Estimated effort: 3-6 hours
- Benefits: simpler templating, no Helm dependency, easier version control
## Breaking Changes
- prometheus-adapter support removed (no backward compatibility)
- KEDA CRDs must exist on cluster (verified during standup)
- KEDA operator installation is cluster-admin responsibility
Signed-off-by: Abhishek Malvankar <asm582@nyu.edu>
Co-authored-by: Claude Haiku 4.5 <noreply@anthropic.com>
Signed-off-by: hubunt <150658615+hubunt@users.noreply.github.com>
* standup: clear wedged router Helm release before apply A router Helm release left in a transitional state (`uninstalling`, or `pending-*`) by a previously interrupted teardown is invisible to `helm list` but still blocks `helmfile apply` from installing: the apply sees an existing release and silently no-ops. The EPP + InferencePool never deploy, and standup step 09 then hangs until timeout on "inference pool: no pods found yet". Detect this in step 08 before the router apply: query `helm status <release>-router`, and if the release is in a wedged state, delete its backing release secret(s) (`owner=helm,name=<release>`) so the apply performs a real install instead of a no-op. Best-effort and gated on non-dry-run; a missing release (the healthy case) is a no-op. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Angelo Ruocco <ang@zurich.ibm.com> * teardown: clean Helm releases stuck in a transitional state `step_01_uninstall_helm._uninstall_releases` listed releases with plain `helm list`, which HIDES releases in a transitional state (`uninstalling`, `pending-*`, `failed`). A release left `uninstalling` by an interrupted teardown was therefore never cleaned: it stayed invisible to teardown yet blocked the next standup's `helmfile apply` (which no-ops on an already-present release), so the EPP/InferencePool never redeployed and standup hung on "inference pool: no pods found yet". Switch to `helm list --all -o json` so wedged releases are seen, and for a matched release in a wedged state delete its backing release secret (`owner=helm,name=<release>`) directly -- `helm uninstall` does not reliably clear an already-`uninstalling` release. Healthy releases are still uninstalled via `helm uninstall` as before. Complements the standup-side guard in step 08 so both ends self-heal. Adds tests covering the wedged, healthy, and unrelated-release paths. --------- Signed-off-by: Angelo Ruocco <ang@zurich.ibm.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Signed off by: fossabot <badges@fossa.com>
…start (llm-d#1613) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: aavarghese <avarghese@us.ibm.com>
…llm-d#1614) Updates the requirements on [numpy](https://github.com/numpy/numpy) to permit the latest version. - [Release notes](https://github.com/numpy/numpy/releases) - [Changelog](https://github.com/numpy/numpy/blob/main/doc/RELEASE_WALKTHROUGH.rst) - [Commits](numpy/numpy@v2.5.0...v2.5.1) --- updated-dependencies: - dependency-name: numpy dependency-version: 2.5.1 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Fix teardown lingering secrets (llm-d#1605) * standup: clear wedged router Helm release before apply A router Helm release left in a transitional state (`uninstalling`, or `pending-*`) by a previously interrupted teardown is invisible to `helm list` but still blocks `helmfile apply` from installing: the apply sees an existing release and silently no-ops. The EPP + InferencePool never deploy, and standup step 09 then hangs until timeout on "inference pool: no pods found yet". Detect this in step 08 before the router apply: query `helm status <release>-router`, and if the release is in a wedged state, delete its backing release secret(s) (`owner=helm,name=<release>`) so the apply performs a real install instead of a no-op. Best-effort and gated on non-dry-run; a missing release (the healthy case) is a no-op. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Angelo Ruocco <ang@zurich.ibm.com> * teardown: clean Helm releases stuck in a transitional state `step_01_uninstall_helm._uninstall_releases` listed releases with plain `helm list`, which HIDES releases in a transitional state (`uninstalling`, `pending-*`, `failed`). A release left `uninstalling` by an interrupted teardown was therefore never cleaned: it stayed invisible to teardown yet blocked the next standup's `helmfile apply` (which no-ops on an already-present release), so the EPP/InferencePool never redeployed and standup hung on "inference pool: no pods found yet". Switch to `helm list --all -o json` so wedged releases are seen, and for a matched release in a wedged state delete its backing release secret (`owner=helm,name=<release>`) directly -- `helm uninstall` does not reliably clear an already-`uninstalling` release. Healthy releases are still uninstalled via `helm uninstall` as before. Complements the standup-side guard in step 08 so both ends self-heal. Adds tests covering the wedged, healthy, and unrelated-release paths. --------- Signed-off-by: Angelo Ruocco <ang@zurich.ibm.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Abhishek Malvankar <asm582@nyu.edu> * Remove dead VariantAutoscaling (VA) object references 27_wva-variantautoscaling.yaml.j2 is a permanently-empty template stub (just an HTML comment noting the modern KEDA path uses annotation-based ScaledObject discovery). The template never renders non-empty, so _has_yaml_content() always skips it — no VA object is ever actually applied today. This removes the dead code paths that still reference it: - _apply_wva_stack_resources(): drop 27_wva-variantautoscaling from loop - _log_wva_stack_state(): remove VA from resource-state query loop - _teardown_wva(): drop VA deletion from cleanup loop - Module/docstring updates across wva.py, step_09, teardown step - Delete the empty template file entirely Rationale: simplify the codebase by removing misleading code that implies VA creation is a live path when it isn't. The KEDA/ScaledObject path is now the only one, with nothing confusing left behind. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> Signed-off-by: Abhishek Malvankar <asm582@nyu.edu> --------- Signed-off-by: Angelo Ruocco <ang@zurich.ibm.com> Signed-off-by: Abhishek Malvankar <asm582@nyu.edu> Co-authored-by: Angelo Ruocco <ang@zurich.ibm.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…1611) * Implement direct KEDA autoscaling (EPP+KEDA) without WVA controller Add a new autoscaling path that queries EPP's pool metrics directly via KEDA, without installing the WVA controller. This enables simpler deployments for single-model scenarios. New features: - EPP+KEDA saturation autoscaling via dual Prometheus triggers (KV cache utilization and queue size) - Extracted shared Prometheus auth logic into keda_prometheus_auth.py - New guide scenario (guides/epp-keda-saturation) and spec - CLI flag --epp-keda-saturation for easy toggling - Per-stack and cluster-scoped setup in step_03 and step_09 - Teardown support in step_01 (delete ScaledObject and auth resources) - Smoketest validation mixin for EPP+KEDA health checks - Documentation and examples for tuning and troubleshooting - Mutual-exclusion validation between WVA and EPP+KEDA modes Implementation: - keda_prometheus_auth.py: Reusable bearer-token + CA cert extraction - keda_saturation.py: Stack discovery and per-namespace orchestration - Templates 29 (ServiceMonitor + RBAC) and 30 (ScaledObject) for EPP+KEDA - Generalized templates 21, 22, 23 to support both WVA and EPP+KEDA - step_09: Added _apply_epp_keda_stack_resources and _log_epp_keda_stack_state Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> Signed-off-by: Abhishek Malvankar <asm582@nyu.edu> * Fix linter issues: remove unused imports, break long lines - Remove unused imports (BaseSmoketest, verify_keda_installed, extract_prometheus_ca_cert, yaml) - Break long function signatures and string literals to comply with line-length limits - Simplify docstrings to fit within 88-character limit - Format multiline function parameters consistently All critical linting errors (F401, F811) resolved. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> Signed-off-by: Abhishek Malvankar <asm582@nyu.edu> * Remove EPP+KEDA documentation file Remove docs/epp-keda-saturation-autoscaling.md from the PR as requested. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> Signed-off-by: Abhishek Malvankar <asm582@nyu.edu> * Apply ruff formatting Fix formatting issues identified by ruff format check. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> Signed-off-by: Abhishek Malvankar <asm582@nyu.edu> * Apply ruff formatting to teardown module Fix formatting issues in step_01_uninstall_helm.py. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> Signed-off-by: Abhishek Malvankar <asm582@nyu.edu> * Fix missing function exports in keda_saturation module The step_03_workload_monitoring.py calls verify_keda_installed() and extract_prometheus_ca_cert() via the keda_saturation module. These functions are defined in keda_prometheus_auth.py but were not being re-exported from keda_saturation.py. Added imports with noqa flags to suppress unused-import warnings since these are intentionally re-exported for external use by step_03. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * Add EPP+KEDA scaling test results and deployment verification report Documents the successful standup and deployment of EPP+KEDA autoscaling on asmalvan-test-2 namespace with Qwen3-32B model. Includes: - Deployment summary (2 decode pods, 1-10 replica limits) - All deployed Kubernetes resources (ScaledObject, HPA, ServiceMonitor, TriggerAuthentication) - Scaling test with 300 concurrent inference requests - Full terminal output showing pod state throughout load test - Root cause analysis of Prometheus 401 authentication issue - Cluster configuration requirements for enabling full scaling - Testing commands and troubleshooting guide for developers Key Finding: Standup and infrastructure deployment successful. Scaling not activated due to cluster-level Prometheus RBAC configuration (not a harness defect). Once cluster admin configures Prometheus access for wva-prometheus-auth ServiceAccount, autoscaling will activate immediately. Verified: - ✅ Step 03 completes without errors (missing functions now exported) - ✅ All EPP+KEDA resources deployed successfully - ✅ Model serving inference requests - ✅ 300 requests processed during load test -⚠️ Scaling requires cluster Prometheus RBAC configuration Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * Fix EPP pool name label in KEDA ScaledObject queries The Prometheus metrics for EPP include the router service/pod name (e.g., 'qwen-qwe-05ee3d35-wen3-32b-router-epp') in their 'name' label, not the bare model ID. Updated the ScaledObject template to append '-router-epp' suffix to the pool name when querying metrics. This fixes the Prometheus queries to match actual metric labels: - Before: name='qwen-qwe-05ee3d35-wen3-32b' (no results) - After: name='qwen-qwe-05ee3d35-wen3-32b-router-epp' (matches metrics) Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * Add prometheus-proxy for EPP+KEDA with configurable URL and detailed documentation EPP+KEDA ScaledObjects now use a local prometheus-proxy deployment (running in openshift-keda namespace) that queries cluster Prometheus using the KEDA operator's projected ServiceAccount token. This avoids kube-rbac-proxy 401 Unauthorized errors that occur when external bearer tokens are used. Changes: - New template 24_prometheus-proxy.yaml.j2: HTTP proxy deployment forwarding to thanos-querier - Updated 30_epp-keda-saturation-scaledobject.yaml.j2: queries proxy instead of Prometheus directly - Added eppKedaSaturation.prometheus.proxyUrl config (default: http://prometheus-proxy.openshift-keda:8080) - Added detailed comments with smoke test commands to verify proxy connectivity - Fixed epp-keda-saturation scenario to use agentgateway (not epponly) for full modelservice deployment The proxy URL is now configurable to support different deployment topologies. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * Add detailed CMA and authentication documentation to ScaledObject template Explains why traditional bearer tokens fail on OpenShift 4.19+ (CMA's hardened kube-rbac-proxy only trusts specific system ServiceAccounts), and documents the solution: using pod-native projected tokens via a local proxy, which matches how other cluster components successfully authenticate to Prometheus. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * add gateway * Remove EPP_KEDA_SCALING_TEST_RESULTS.md from PR Testing results are documented in the PR comment instead. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * Fix markdown link check failures in PR - Fix broken link in README.md: removed reference to non-existent docs/epp-keda-saturation-autoscaling.md and consolidated both WVA and EPP+KEDA autoscaling references to point to the unified guide - Fix 3 broken links in docs/workload-variant-autoscaler.md: - 27_wva-variantautoscaling.yaml.j2 (doesn't exist) - 28_wva-hpa.yaml.j2 (correct file is 28_wva-scaledobject.yaml.j2) - 21_prometheus-adapter-values.yaml.j2 (doesn't exist) Combined these into a single entry pointing to the correct scaledobject template Fixes CI markdown-link-check failures: 1 broken link in README.md, 3 in workload-variant-autoscaler.md Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> --------- Signed-off-by: Abhishek Malvankar <asm582@nyu.edu> Signed-off-by: Abhishek Malvankar <asm582@users.noreply.github.com> Co-authored-by: Claude Haiku 4.5 <noreply@anthropic.com>
Signed-off-by: Mengmei Ye <mye@ibm.com>
…d#1616) Add a top-level `reset_caches` key to experiment YAML files. When set, step_07 POSTs /reset_prefix_cache, /reset_mm_cache, and /reset_encoder_cache to every vLLM serving pod before each treatment's run, so every treatment starts against cold caches without redeploying the stack. Default is false. Resets are non-fatal: failures (e.g. a 404 when the server lacks VLLM_SERVER_DEV_MODE=1) warn and continue. Signed-off-by: Angelo Ruocco <ang@zurich.ibm.com>
The gateway-enforced HTTPRoute request timeout was hardcoded in 08_httproute.yaml.j2 (300s on shared-mode rules, 0s on per-stack). Long agentic turns that exceed it are aborted with a 502, which surfaces as a "connection reset by peer" on the decode routing-proxy loopback hop. Expose it as httpRoute.requestTimeout / httpRoute.backendRequestTimeout, defaulting to 300s / 0s in defaults.yaml and applied to all three rule blocks. Guides carry an explicit httpRoute block; agentic-serving sets requestTimeout: 0s to disable the cap for long sessions. Signed-off-by: Angelo Ruocco <ang@zurich.ibm.com>
The template was incorrectly appending '-router-epp' to construct the metric label,
but Prometheus exports the metric with just the model name (e.g., 'qwen-qwe-debb0c3c-wen3-32b-router').
The '-epp' suffix is part of the service name, not the metric label itself.
KEDA was querying:
max(inference_pool_average_kv_cache_utilization{name="model-router-epp",...})
But Prometheus has:
max(inference_pool_average_kv_cache_utilization{name="model-router",...})
This mismatch caused KEDA to always return empty results, preventing autoscaling.
Fix: Change poolName default from 'model_id_label ~ "-router-epp"' to 'model_id_label ~ "-router"'.
llm-d#1610) * move sections like routing and gateway under the model service section * fix broken links in docs/workload-variant-autoscaler.md * fix one more broken link in docs/workload-variant-autoscaler.md * move httpRoute under modelservice section * enable prefix caching for predicted-latency-routing --------- Signed-off-by: Mengmei Ye <mye@ibm.com>
llm-d#1628) Signed-off-by: aavarghese <avarghese@us.ibm.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
9 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
skopeo list-tagsdirectly and sort parsed tag data safelyv0.20.1versusv0.9.2Verification
pytest -q tests/test_version_resolver.py(17 passed)ruff check llmdbenchmark/parser/version_resolver.py tests/test_version_resolver.pyFixes When automatically selecting the "most recent tag", sort by version number, not lexicographically llm-d/llm-d-benchmark#1060