Problem Statement
Currently saturation signals can be configured based on external metric which takes time to update because of the scrape interval this can lead to llm-d-async overshooting saturation. llm-d-async can also send a surge of requests before the saturation metric updates, and the async processor fetches it. This was seen in #304
Additionally not all inference requests have equal saturation impact on the server. For instance an inference request with (100000 input/1000 output tokens) will saturate the inference server's capacity signicantly more than a (500 input/100 output tokens) request. Currently in llm-d-async, both these request occupy a single queue and pool worker slot.
Proposed Solution
There are a few solutions that need to be experimented:
- See if latency-prediction sidecar can be utilized to predict impact on saturation
- add plugin support for request parsers to estimate token throughput.
Alternatives Considered
No response
Willingness to Contribute
Yes, I can submit a PR
Additional Context
No response
Problem Statement
Currently saturation signals can be configured based on external metric which takes time to update because of the scrape interval this can lead to llm-d-async overshooting saturation. llm-d-async can also send a surge of requests before the saturation metric updates, and the async processor fetches it. This was seen in #304
Additionally not all inference requests have equal saturation impact on the server. For instance an inference request with (100000 input/1000 output tokens) will saturate the inference server's capacity signicantly more than a (500 input/100 output tokens) request. Currently in llm-d-async, both these request occupy a single queue and pool worker slot.
Proposed Solution
There are a few solutions that need to be experimented:
Alternatives Considered
No response
Willingness to Contribute
Yes, I can submit a PR
Additional Context
No response