vLLM Dashboard

[gaby]

Would be nice to have a dashboard for vLLM which is heavily used to run LLM's

They provide Prometheus metrics: https://docs.vllm.ai/en/latest/design/metrics.html#v0-metrics

I tried replicating the Grafana dashboard, but couldnt figure out how to do rate() in OpenObserve.

[prabhatsharma]

Just use the same promql query in Grafana in O2. It should work without any change

[gaby]

@prabhatsharma I tried it but it complains about undefined values. These are the queries from the example grafana dashboard:

rate(vllm:prompt_tokens_total{model_name=\"$model_name\"}[$__rate_interval])"

Dashboard is here: https://github.com/vllm-project/vllm/blob/main/examples/online_serving/prometheus_grafana/grafana.json

[prabhatsharma]

they are variables in grafana. you will need to define same variables in O2 as well.

$__rate_interval is available in O2 just like in grafana. Howevere define model name or remove it.

To test you can change the query to:

rate(vllm:prompt_tokens_total{}[$__rate_interval])

[gaby]

@prabhatsharma I will try this week and report back