vllm V1 production metrics

[jjk-g]

What would you like to be added:
Support for all non-deprecated metrics in https://docs.vllm.ai/en/stable/usage/metrics/?h=production+metrics

Why is this needed:
Full metric parity for 1.0.0 milestone.

[jjk-g]

Thoughts on managing previous versions' supported metrics.

1. Version output reports so they align with expected metrics
2. Maintain a metrics library that can span deprecated metrics

Ideally we have visibility to when something is no longer supported, and clear guidance on what is supported.

[changminbark]

Are any of these vLLM metrics being recorded/saved currently?

[changminbark]

@jjk-g I looked through the codebase and was not able to find anything on vLLM metrics. I am guessing that this feature would look something similar to the PrometheusMetricsClient, but we would querying the vLLM metrics endpoint?

[jjk-g]

inference-perf/inference_perf/client/modelserver/vllm_client.py

Line 65 in e3e690b

"vllm:num_requests_waiting",

Is where the metrics definitions are for inference-perf

[changminbark]

@jjk-g I thought about this issue and I think a good way to approach this would be to allow users to define a version (add an optional version field to server config) for the model server. We would have a default version that is set to the latest version for that model server. This will be passed to the ModelServerClient constructor.

Then we can modify the vLLMModelServerClient (and other ModelServerClients) to have private methods like _get_v0_metrics(self), which will be called by the get_prometheus_metric_metadata(self) method by using the self.model_server_version.

That way, we can have a versioning system via private methods and make it backwards-compatible (by having a default version for each ModelServerClient).

If this looks good, I will go ahead and implement this strategy. I also have a question regarding the current metrics. What version of vLLM do they belong to?

[jjk-g]

I think proposing some compatability system would be great, let's continue that discussion in #279. It is complicated because the V0 metrics (vllm v0 engine) likely doesn't need to be supported. The challenge is supporting of the many minor version (eg 0.11 - 0.16).

For this issue and inference-perf v1.0.0 I think it is sufficient to just ensure all the production metrics are present and correct from https://docs.vllm.ai/en/stable/usage/metrics/?h=production+metrics latest