Question: How does WVA map a vLLM deployment to its EPP for SLO-based scaling?

[vivekk16]

Context

We are integrating WVA with KServe. We have seen mentions of SLO-based scaling being introduced in a future release and want to ensure KServe is ready for it when it lands.

Questions

1. How does WVA pick EPP metrics for SLO-based scaling? Which metrics are used and where are they fetched from?

2. Can you point us to the logic that maps a vLLM deployment (the scale target) to its corresponding EPP? We want to understand the full chain so we can verify it works correctly in a KServe environment.

[asm582]

@dumb0002 can you please help?

[dumb0002]

@vivekk16 (cc: @asm582) - see below my answers in the context of the implementation of the scaleFromZero feature:

How does WVA pick EPP metrics for SLO-based scaling? Which metrics are used and where are they fetched from?

For scaleFromZero WVA fetch the metric inference_extension_flow_control_queue_size from the EPP flowcontrol layer - see an overview of the architecture flow in the doc here.

Can you point us to the logic that maps a vLLM deployment (the scale target) to its corresponding EPP?

The logic for the mapping a vLLM deployment to the corresponding EPP is here.

Please, let us know if you have any further question. Thanks!

[vivekk16]

Thanks @dumb0002 . Closing this issue for now.
Will reopen if there are further queries.