Add optional inference objective #1995

Gregory-Pereira · 2025-12-13T19:03:41Z

What type of PR is this?
/kind cleanup
/kind feature

What this PR does / why we need it:

Enable utilization of the InferenceObjective CR we already have

Does this PR introduce a user-facing change?:
NONE, simply exposes the inferencepool objective in the helm charts

Signed-off-by: greg pereira <[email protected]>

netlify · 2025-12-13T19:03:47Z

✅ Deploy Preview for gateway-api-inference-extension ready!

Name	Link
🔨 Latest commit	`db76251`
🔍 Latest deploy log	https://app.netlify.com/projects/gateway-api-inference-extension/deploys/6940304a6ff8670008660d30
😎 Deploy Preview	https://deploy-preview-1995--gateway-api-inference-extension.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

k8s-ci-robot · 2025-12-13T19:03:47Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Gregory-Pereira
Once this PR has been reviewed and has the lgtm label, please assign danehans for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot · 2025-12-13T19:03:51Z

Hi @Gregory-Pereira. Thanks for your PR.

I'm waiting for a github.com member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

shmuelk · 2025-12-14T09:31:45Z

This PR looks ok, but somehow I think it's missing something.

It is creating a single InferenceObjective with a name that matches the Helm Release Name.

As I understand things the InferenceObjective is referenced by the header x-gateway-inference-objective sent with the request. This is a request related thing. I would expect the ability to create several InferenceObjectives each with a different name and different priority.

Gregory-Pereira · 2025-12-14T20:20:03Z

Good point, I will update the implementation so that users could define all the inference objectives they wish to relate to the inference pool

Signed-off-by: greg pereira <[email protected]>

config/charts/inferencepool/values.yaml

config/charts/inferencepool/templates/inferenceobjective.yaml

nirrozenbaum · 2025-12-15T08:45:21Z

/ok-to-test

Signed-off-by: greg pereira <[email protected]>

config/charts/inferencepool/templates/inferenceobjectives.yaml

ahg-g · 2025-12-15T15:33:05Z

Can you please discuss the motivation for this? I see some value, but infObj are a resource that will be created/updated/deleted after creating the infPool; meaning likely new objectives will be added/deleted later.

config/charts/inferencepool/values.yaml

… over inference objectives Signed-off-by: greg pereira <[email protected]>

Gregory-Pereira · 2025-12-15T16:35:19Z

Can you please discuss the motivation for this? I see some value, but infObj are a resource that will be created/updated/deleted after creating the infPool; meaning likely new objectives will be added/deleted later.

I saw the value as automating the creation / deletion of them. In this way they get created and cleaned up with the helm chart. Not to say that others cannot add more out of band. I started on this in preparation for the Flow Control integration work with regard to an LLM-D guide that could showcase the work.

nirrozenbaum · 2025-12-15T16:38:44Z

config/charts/inferencepool/templates/inferenceobjectives.yaml

+  priority: {{ .priority }}
+  poolRef:
+    group: {{ .Values.inferenceExtension.apiVersion }}
+    name: {{ .name }}


another miss?

Suggested change

name: {{ .name }}

name: {{ .Release.Name }}

ahg-g · 2025-12-15T16:39:12Z

ok, I can see value in cases where for the most part the objectives are known in advance and mostly static

nirrozenbaum · 2025-12-15T16:39:43Z

config/charts/inferencepool/templates/inferenceobjectives.yaml

+kind: InferenceObjective
+metadata:
+  name: {{ .name }}
+  namespace: {{ $.Release.Namespace }}


OOC - why do we need here the $?
shouldn't we use

Suggested change

namespace: {{ $.Release.Namespace }}

namespace: {{ .Release.Namespace }}

?

ahg-g · 2025-12-15T16:41:33Z

config/charts/inferencepool/README.md

+| `inferenceExtension.sidecar.volumeMounts`                  | List of volume mounts for the sidecar container. Optional.                                                                                                                                                                                         |
+| `inferenceExtension.sidecar.volumes`                       | List of volumes for the sidecar container. Optional.                                                                                                                                                                                               |
+| `inferenceExtension.sidecar.configMapData`                 | Custom key-value pairs to be included in a ConfigMap created for the sidecar container. Only used when `inferenceExtension.sidecar.enabled` is `true`. Optional.                                                                                   |
+| `inferenceObjectives`                                      | A list of names and priorities to create InferenceObjectives from that will be assigned to the inference pool                                                                                                                                      |


Recommend documenting that this is for the case where the objectives are known in advance and mostly static, and that the user can still add/update/delete objectives later.

kfswain · 2025-12-15T23:06:21Z

config/charts/inferencepool/values.yaml

      #     maxRequestsPerConnection: 256000
+
+
+# Optional: Define multiple InferenceObjectives for this InferencePool.


I think: https://github.com/kubernetes-sigs/gateway-api-inference-extension/pull/1995/changes#r2620118777 would apply here also

kfswain · 2025-12-15T23:07:16Z

Agreed with the other comments here. As long as we communicate clearly that there isn't a need to correlate the infObjectives at Pool creation, this all seems reasonable to me

Gregory-Pereira added 2 commits December 13, 2025 10:59

enable creating inferenceObjective via the inferencepool helm chart

857ed6b

Signed-off-by: greg pereira <[email protected]>

updating readme + linting

d876b40

Signed-off-by: greg pereira <[email protected]>

k8s-ci-robot requested review from nirrozenbaum and shmuelk December 13, 2025 19:03

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Dec 13, 2025

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Dec 13, 2025

allow array of inferencepools

17a159c

Signed-off-by: greg pereira <[email protected]>

nirrozenbaum reviewed Dec 14, 2025

View reviewed changes

config/charts/inferencepool/values.yaml Outdated Show resolved Hide resolved

nirrozenbaum reviewed Dec 14, 2025

View reviewed changes

config/charts/inferencepool/templates/inferenceobjective.yaml Outdated Show resolved Hide resolved

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Dec 15, 2025

move inferenceObjective to top level and cleanup template

ff97818

Signed-off-by: greg pereira <[email protected]>

Gregory-Pereira force-pushed the add-optional-inference-objective branch from ca76a99 to ff97818 Compare December 15, 2025 15:12

nirrozenbaum reviewed Dec 15, 2025

View reviewed changes

config/charts/inferencepool/templates/inferenceobjectives.yaml Outdated Show resolved Hide resolved

nirrozenbaum reviewed Dec 15, 2025

View reviewed changes

config/charts/inferencepool/values.yaml Outdated Show resolved Hide resolved

Gregory-Pereira force-pushed the add-optional-inference-objective branch from e000098 to 6751dd6 Compare December 15, 2025 15:57

remaining cleanup removing the checking of apiVersion when itterating…

db76251

… over inference objectives Signed-off-by: greg pereira <[email protected]>

Gregory-Pereira force-pushed the add-optional-inference-objective branch from 6751dd6 to db76251 Compare December 15, 2025 15:59

nirrozenbaum reviewed Dec 15, 2025

View reviewed changes

ahg-g reviewed Dec 15, 2025

View reviewed changes

kfswain reviewed Dec 15, 2025

View reviewed changes

	namespace: {{ $.Release.Namespace }}
	namespace: {{ .Release.Namespace }}

		# maxRequestsPerConnection: 256000


		# Optional: Define multiple InferenceObjectives for this InferencePool.

Add optional inference objective #1995

Are you sure you want to change the base?

Add optional inference objective #1995

Conversation

Gregory-Pereira commented Dec 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

netlify bot commented Dec 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for gateway-api-inference-extension ready!

Uh oh!

k8s-ci-robot commented Dec 13, 2025

Uh oh!

k8s-ci-robot commented Dec 13, 2025

Uh oh!

shmuelk commented Dec 14, 2025

Uh oh!

Gregory-Pereira commented Dec 14, 2025

Uh oh!

Uh oh!

Uh oh!

nirrozenbaum commented Dec 15, 2025

Uh oh!

Uh oh!

ahg-g commented Dec 15, 2025

Uh oh!

Uh oh!

Gregory-Pereira commented Dec 15, 2025

Uh oh!

nirrozenbaum Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

ahg-g commented Dec 15, 2025

Uh oh!

nirrozenbaum Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

ahg-g Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

kfswain Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

kfswain commented Dec 15, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Gregory-Pereira commented Dec 13, 2025 •

edited

Loading

netlify bot commented Dec 13, 2025 •

edited

Loading