Define a standard approach for plugins to identify prefill and decode endpoints

[ahg-g]

What would you like to be added:

Define a standard approach for plugins to identify prefill and decode endpoints

Why is this needed:

Some plugins need to be p/d aware, one example is #2021

One suggestion is to have the SchedulingResult define optional PrefillProfileName and DecodeProfileName in addition to the PrimaryProfileName we currently have. The argument for this is that p/d is a core feature of LLM serving, and so having such explicit definitions is reasonable especially when defined as optional fields.

Please suggest other options to address this requirement.

[kfswain]

Questions/Thoughts

I have some thoughts here:

• Up until this point, we have been putting context and opinions into plugins, and keeping the framework very context free. I think this is something we should preserver.
• Making an exception here sets the precedent that we make an exception to our system whenever something is in vogue. Which can cause for a lot of optional fields down the line, leading to a more confusing framework to interact with.
• If we did choose to add these fields, how do we bind the specific profiles to these fields? Is that config on the user end?
  I think we have smaller steps we can make before we determine if this is necessary:
• Before this, or other steps are considered, we need to have an IGW implementation of P/D to reference to.
• llm-d PD plugins kinda solve this already: https://github.com/llm-d/llm-d-inference-scheduler/blob/d608568b50f308699cda5351f57c71bc415a5398/pkg/plugins/profile/pd_profile_handler.go#L48

Ideas

• We could provide a metadata field as part of the profile results that can provide labeling, or have ProfileHandlers implement a Config() function that gives readonly access to scorers to gather this information.
• Can we make a smaller, more measured approach of just using a couples const defined in the P/D package and use those as the canonical profile names? Example here

[ahg-g]

On preserving generality: I think this is a gray area with lots of room for interpretation of what that exactly entail. The whole IGW is a specialization of general load balancing, IGW also has a fair amount of code specific to LLMs, so I don't see this necessarily as an "exception". My personal opinion is that p/d is such a core feature of llm serving that warrants making it "framework aware" if done cleanly.

On proposed alternatives: I think defining constants across plugins is not ideal, the goal of the framework is to offer coordination such that plugins can to the extent possible be independent, and when they have a dependency it is managed via well defined interfaces. I am not against that if we can make it work in a standard way, but at the same time I don't see this as a major improvement with respect to preserving "generalizability" because whatever approach we come up with will become an "API" in some sense, albeit less explicitly defined (which in my opinion is a con)

[ahg-g]

I agree we need to get p/d to upstream first, we have llm-d/llm-d-inference-scheduler#356 to track this work. So I suggest that we do a quick a dirty fix for #2021 (a hard coded prefill profile name) and as part of llm-d/llm-d-inference-scheduler#356 we address the problem discussed in this issue.

[ahg-g]

/assign @yangligt2

[nirrozenbaum]

I tend to agree with @kfswain.

I was trying to think about this note a lot:

IGW also has a fair amount of code specific to LLMs, so I don't see this necessarily as an "exception".

This thing is - IGW is handling standards and not specifics. the LLMs handing we have is aligned with OpenAI, e.g., we support chat/completions, we rely on some output fields on the response. not because we decided arbitrarily, but because we aligned with a standard.
I think the same goes here - as long as P/D is not even defined as a protocol in IGW, there is no reason treating it specially.
adding the profile names of P/D to scheduling result while P/D is not even standardized in IGW sounds to me too specific and is very not ideal.
I believe the general problem that we're trying to address here can be easily solved by adding parameters of prefill profile and decode profile (as Kellen pointed in the example) with default values (so if nothing is specified the default is a const in the code).

I definitely agree that we should keep the framework generalized while putting context and opinions into plugins.
also, I'd be happy to understand - which plugins are expected to use the profile names and what are their usage?
is it only the pre-request and post response or others as well?

[ahg-g]

As noted in #2080 (comment), there is a plan to bring P/D into IGW and I agree that this issue should be addressed as part of that effort.

[nirrozenbaum]

which plugins are expected to use the profile names and what are their usage?
is it only the pre-request and post response or others as well?

The reason I’m asking is - if any of the scheduling plugins needs this information, it would not help adding p/d profile names to SchedulingResult, cause none of these plugins have access to this object. only rquestcontrol layer plugins.

but setting a common parameter would work.