-
Notifications
You must be signed in to change notification settings - Fork 33
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Endpoint Picker Protocol Proposal #164
Conversation
This is awesome! /hold |
✅ Deploy Preview for gateway-api-inference-extension ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
| TimePerPrefillToken | Histogram | The prefill latency per token in the last W seconds. W will be decided by simulation/benchmarking. In time series metric the latency is typically reported as Histogram and we can derive the average from the Histogram. | `vllm:time_to_first_token_seconds` | | ||
| TimePerDecodeToken | Histogram | The decode latency per token in the last W seconds. W will be decided by simulation/benchmarking. | `vllm:time_per_output_token_seconds` | | ||
|
||
## LoRA Adapter Serving |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should consider making the model server protocol pluggable with LoRA being a reference plugin implementation.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Some nits and thoughts, but overall LGTM! This is great!
/lgtm |
I suggest the following:
|
Also, part of this protocol is how to communicate the picked endpoint, the request header |
/retitle "Add Endpoint Picker Protocol Proposal" |
request, provided the requested adapter is valid. | ||
|
||
The model server MUST expose the following LoRA adapter information via a RESTful API with response | ||
in JSON : |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does the current EPP implementation support this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No it does not. It uses the current vLLM metrics implementation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The protocol is part of the release, and so i think we should remove this until the next release when we actually implement it and only spec the metrics based approach.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I can document the current metrics approach for vllm, but I don't want to make it a "protocol", because it's really awkward. It's not something we should recommend for the next model server to implement.
The purpose of the protocol is to set up a contract for any new model server integration to follow. So I think we should document this here, and with a note that the current vllm workaround should converge too.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
But that is why we have versioning, it doesn't make sense to document a contract that the EPP doesn't implement. Once we have EPP implements it, we update the protocol and create a new release. @robscott @smarterclayton for opinions.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
IMO this is just a short-term workaround and we don't need a protocol to document it - the code speaks for itself. I think the protocol should be something reasonably stable based on our best knowledge, and we are willing to take to integrate with another model server.
One option is to just not document this at all, until we have some implementation to support it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not documenting anything is an option, but documenting a protocol and make it part of a release that doesn't implement it is I think not what we should do here. Remember that we are versioning the protocol with the EPP image.
I would still lean towards documenting what we implemented and released, and we can change that in the next release, that is what versioning allows us to do. It is expected that the initial iterations will include more frequent changes, and the protocol will stabilize after that.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would probably align to abdullah here - the goal is to spec what is currently required, so that you don't read things that don't exist. There will be people using 0.1 for quite a while, so be accurate to 0.1. Anything that is to be removed can simply be moved to a separate PR which is "draft for 0.2 proposed changes" and not lost.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the feedback, I updated the doc
/lgtm Thanks @liu-cong ! |
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: ahg-g, liu-cong The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
/hold cancel |
This is adapted from the initial doc
I didn't include the ORCA load reporting section as it's not currently required by the inference extension, though parallel efforts are happening. The intention is to keep the scope of this protocol small and expand in the future if needed.
I see this as a very initial effort to define the contract, and is an evolving process to monitor industry trends and drive more unification.