fix(daemon): decouple vLLM Prometheus scraping from enable_docker#104
Open
michaelroy-amd wants to merge 2 commits into
Open
fix(daemon): decouple vLLM Prometheus scraping from enable_docker#104michaelroy-amd wants to merge 2 commits into
michaelroy-amd wants to merge 2 commits into
Conversation
VllmPrometheusCollector was constructed only when opts.enable_docker && !opts.disable_vllm_metrics, but the scraper is a plain HTTP GET against a vLLM instance's /metrics endpoint with no Docker dependency. Managed (native) instances are already discovered independently of Docker via services_dir. apps/rocm/src/dash.rs::maybe_spawn_embedded_daemon always calls runner_options(config, paths, false), hardcoding enable_docker=false, which made the scraper permanently dead for the embedded daemon - the common no-Docker / managed-vLLM path. Gate vllm construction on !opts.disable_vllm_metrics alone; enable_docker continues to gate only DockerDiscovery. Relates to EAI-7359 Signed-off-by: Michael Roy <michael.roy@amd.com>
… shipped flag Adversarial-review follow-up for EAI-7359 (wording only, no behavior change): disable_vllm_metrics is hardcoded false and not wired to any CLI flag or config field, so it is not user-reachable today. Soften the field doc and test comments to describe it as the internal seam a future opt-out would flip, rather than implying a live off-switch. Relates to EAI-7359 Signed-off-by: Michael Roy <michael.roy@amd.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
VllmPrometheusCollector(the vLLM/metricsHTTP scraper) was only constructed whenopts.enable_docker && !opts.disable_vllm_metrics, even though the scraper has zero Docker dependency — it's a plain HTTP GET against a vLLM instance's/metricsendpoint.apps/rocm/src/dash.rs::maybe_spawn_embedded_daemonalways callsrunner_options(config, paths, false), hardcodingenable_docker = false. Combined with the gate above, this made the vLLM Prometheus scraper permanently dead for the embedded daemon — the common no-Docker / managed-vLLM path.vllmconstruction inrun_loopon!opts.disable_vllm_metricsalone.enable_dockercontinues to gate onlyDockerDiscovery, which is the only genuinely Docker-dependent piece.Root cause
Managed (native) vLLM instances are already discovered independently of Docker via
services_dir, so the scraper never neededenable_dockeras a precondition. The original gate conflated "Docker discovery is enabled" with "scrape metrics from any known vLLM instance," silently disabling metrics for every embedded-daemon deployment.Relates to EAI-7359
Test Plan
cargo buildcleancargo clippy --all-targets --all-features -- -D warningscleancargo test -p rocm-dash-daemon runner::— 11 passed, including newvllm_metrics_enabled_is_independent_of_dockerregression testcargo test -p rocm dash:: -- --test-threads=1— 8 passed, including newrunner_options_keeps_vllm_metrics_enabled_without_dockerregression test