Should we enhance prefill-filter/decode-filter with configurable labels, or encourage direct use of by-label?

[googs1025]

What would you like to be added:

When integrating other PD (Prefill/Decode)workloads—such as Aibrix’s StormService— we use different Pod labels like:

role=prefill
role=decode

However, the current prefill-filter and decode-filter plugins hardcode:

Label key: "llm-d.ai/role"
Valid values: ["prefill"] and ["decode", "both"]
This makes it difficult to adopt the scheduler with workloads that use different labeling schemes (e.g., role=decode) without modifying their manifests or forking the code.

We see two possible paths forward and would like the community’s input on which is preferred:

✅ Option 1: Enhance prefill-filter / decode-filter with optional parameters
Allow these plugins to accept configuration like:

- type: decode-filter
  parameters:
    label: "role"
    validValues: ["decode", "both"]

If no parameters are provided, fall back to the current hardcoded defaults (llm-d.ai/role, etc.). Internally, they would delegate to the existing ByLabel filter to avoid duplication.

🔁 Option 2: Encourage direct use of the generic by-label plugin
Instead of extending the specialized filters, guide users to define their own by-label filters:

- name: prefill-role-filter
  type: by-label
  parameters:
    label: "role"
    validValues: ["prefill"]

- name: decode-role-filter
  type: by-label
  parameters:
    label: "role"
    validValues: ["decode", "both"]

Practical note: This requires each by-label instance to have a unique name, and assumes the framework supports multiple instances of the same plugin type. In practice, this works if the plugin system is instance-based—but it does make configs slightly more verbose and less self-explanatory compared to named semantic filters like prefill-filter.

Why is this needed:

[elevran]

As you note above, the set of filters/scorers is configured in a file and one can use either by-label or the specific decode/prefill-filter types. The default P/D configuration file uses the specific filters.

Unfortunately, they're not 100% interchangeable in our case. A Pod is considered decode if set to decode, both or the lable is not set at all (to accomodate the case of existing pools without explicit labels, and require only Prefill to be marked so they do not receive token generation requests).
This last case can not be expressed by a k8s LabelSelector cleanly.

So the options might be

• do nothing in code, users can configure by-label filter in the file if they know labels are mandatory and set their values for the filter;
• configure use by-label for all and require role labels to always exist (users configure the label name and values); or
• allow decode nodes without any role label and extend the decode/prefill-filter to accept name and value.

See #188 for additional context.

Practical note: This requires each by-label instance to have a unique name, and assumes the framework supports multiple instances of the same plugin type. In practice, this works if the plugin system is instance-based—but it does make configs slightly more verbose and less self-explanatory compared to named semantic filters like prefill-filter.

The configuration framework supports multiple instances of the same plugin type. Each is required to have a unique name which is slightly more verbose than the explicit prefill-filter and decode-filter types.

[googs1025]

Thanks for the clarification, @elevran ! 😄

To confirm: we should proceed with enhancing prefill-filter and decode-filter to support configurable label keys and valid values, while preserving the current default behavior — especially the decode semantics where pods without the label are still considered decode-capable.

This would allow integrations like Aibrix StormService to use their own labeling scheme (e.g., role=prefill) without breaking backward compatibility or losing the “unlabeled = decode” fallback.

If this aligns with your recommendation, I’m happy to draft a PR implementing this enhancement.

[googs1025]

/assign

[elevran]

@googs1025 if AiBrix StormService always sets specific labels and values for prefill and decode (that's my understanding based on the available doc), you could use the existing by-label filter in the config. The possible downsides are (1) specific configuration, not pre-canned with llm-d, (2) slightly more verbose by adding filter names.
Are there other shortcomings you are aware of? If not, then this would be my recommendation. Requesting additionl comments from others, below.

To confirm: we should proceed with enhancing prefill-filter and decode-filter to support configurable label keys and valid values, while preserving the current default behavior — especially the decode semantics where pods without the label are still considered decode-capable.

@mayabar @shmuelk @nilig WDYT?

[shmuelk]

Are there other shortcomings you are aware of? If not, then this would be my recommendation. Requesting additionl comments from others, below.

I don't see any shortcomings.

I will further point out that both the prefill-filter and the decode-filter are simply syntactic sugar for by-label filter mentioned by @elevran. Adding parameters to the prefill-filter and the decode-filter makes no sense. Just use the existing underlying by-label filter.

You could, if you wanted, use the by-label-selector filter as well. It would depend on how you want your configuration to read.

[googs1025]

Thanks for the suggestion, 😄 @elevran @shmuelk

I’m exploring the option of using by-label directly for AiBrix StormService (which always sets role=prefill or role=decode).

However, I’m not sure whether the current plugin system supports defining multiple instances of the same plugin type—for example, two by-label filters with different parameters.

In particular, does the schema allow a name (or equivalent identifier) to distinguish them? If not, a configuration like the one below might not be valid:

plugins:
...
- type: by-label
  parameters:
    label: "role"
    validValues: ["prefill"]
- type: by-label  # ← Is this allowed without a unique name?
  parameters:
    label: "role"
    validValues: ["decode", "both"]

schedulingProfiles:
- name: prefill
  plugins:
  - pluginRef: ???  # How would we reference the first by-label?
  ...
- name: decode
  plugins:
  - pluginRef: ???  # And the second?
  ...

Could you clarify whether the framework currently allows this pattern? Happy to go either way once we confirm the capabilities!

[shmuelk]

Try the following config instead:

plugins:
...
- type: by-label
  name: "prefill"
  parameters:
    label: "role"
    validValues: ["prefill"]
- type: by-label
  name: "decode"
  parameters:
    label: "role"
    validValues: ["decode", "both"]

schedulingProfiles:
- name: prefill
  plugins:
  - pluginRef: "prefill"
  ...
- name: decode
  plugins:
  - pluginRef: "decode"

pluginRef is actually by name, not by type. If you don't give a plugin a name, the name is automatically set to it's type. In the config above, I simply added names to the two plugin instances and referenced them accordingly.

[googs1025]

Try the following config instead:

plugins:
...

• type: by-label
  name: "prefill"
  parameters:
  label: "role"
  validValues: ["prefill"]
• type: by-label
  name: "decode"
  parameters:
  label: "role"
  validValues: ["decode", "both"]

schedulingProfiles:

• name: prefill
  plugins:
  • pluginRef: "prefill"
    ...
• name: decode
  plugins:
  • pluginRef: "decode"
    pluginRef is actually by name, not by type. If you don't give a plugin a name, the name is automatically set to it's type. In the config above, I simply added names to the two plugin instances and referenced them accordingly.

Thanks, I checked the code and it supports names. https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/79457c792a2b482b59e583253c27e6532906141c/apix/config/v1alpha1/endpointpickerconfig_types.go#L54-L70

I will use the by-label plugin to test it. Thank you for your answer. 😄