Add StrictFIFOPerFlavor queueing strategy to avoid cross-flavor head-of-line blocking

[rooty0]

What type of PR is this?

/kind feature
/kind api-change

What this PR does / why we need it:

This PR introduces a new queueing strategy StrictFIFOPerFlavor that combines strict FIFO ordering with per-flavor independence to prevent cross-flavor head-of-line blocking.

Problem:
With the existing StrictFIFO strategy, a workload waiting for quota on one resource flavor (e.g., expensive GPUs) blocks all workloads behind it in the queue, even those that could run immediately on different resource flavors (e.g., cheaper CPUs or different GPU types).

Solution:
StrictFIFOPerFlavor maintains strict FIFO ordering within each resource flavor but allows workloads targeting different flavors to be admitted out of order. This enables better resource utilization in multi-flavor clusters while preserving fairness guarantees within each flavor.

Key behaviors:

• Workloads are ordered by creation timestamp (strict FIFO)
• A workload only blocks subsequent workloads that need the same resource flavor(s)
• Workloads needing different flavors can be admitted out of order
• Within a flavor, FIFO order is strictly maintained (prevents starvation)

Implementation:

• Tracks attempted flavors for each inadmissible workload
• During QueueInadmissibleWorkloads(), checks for flavor conflicts before moving to active queue
• Minimal performance impact: O(I × F) where I = inadmissible workloads, F = flavors per workload

Which issue(s) this PR fixes:

Fixes #8309

Special notes for your reviewer:

• Backward compatible: No changes to existing StrictFIFO and BestEffortFIFO behavior
• New API field: Added StrictFIFOPerFlavor to the QueueingStrategy enum in ClusterQueue CRD
• Testing: Unit tests added in cluster_queue_strictfifo_perflavor_test.go
• Debug logging: Added V(3) logs to help troubleshoot flavor blocking decisions

Suggested review order:

1. API changes: apis/kueue/v1beta2/clusterqueue_types.go
2. Core logic: pkg/cache/queue/cluster_queue.go (RequeueIfNotPresent, QueueInadmissibleWorkloads)
3. Flavor tracking: pkg/scheduler/scheduler.go (requeueAndUpdate)
4. Tests: pkg/cache/queue/cluster_queue_strictfifo_perflavor_test.go

Working example: https://asciinema.org/a/BMWnC6mY5OvWf2yjHyJQpV38d

Does this PR introduce a user-facing change?

Add new `StrictFIFOPerFlavor` queueing strategy for ClusterQueues. This strategy maintains strict FIFO ordering but only blocks workloads that compete for the same resource flavor(s), allowing workloads targeting different flavors to be admitted out of order. This prevents head-of-line blocking in multi-flavor clusters while preserving fairness within each flavor.

[k8s-ci-robot]

Welcome @rooty0!

It looks like this is your first PR to kubernetes-sigs/kueue 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/kueue has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: rooty0
Once this PR has been reviewed and has the lgtm label, please assign mimowo for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[k8s-ci-robot]

Hi @rooty0. Thanks for your PR.

I'm waiting for a github.com member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[netlify]

✅ Deploy Preview for kubernetes-sigs-kueue ready!

Name	Link
🔨 Latest commit	320fb6b
🔍 Latest deploy log	https://app.netlify.com/projects/kubernetes-sigs-kueue/deploys/694a067be10427000819a185
😎 Deploy Preview	https://deploy-preview-8393--kubernetes-sigs-kueue.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[tenzen-y]

/ok-to-test

[tenzen-y]

I think such changes require the KEP process.
/hold

https://github.com/kubernetes-sigs/kueue/tree/main/keps

[k8s-ci-robot]

@rooty0: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-kueue-verify-main	320fb6b	link	true	/test pull-kueue-verify-main

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[rooty0]

I think such changes require the KEP process.
/hold

https://github.com/kubernetes-sigs/kueue/tree/main/keps

I'm totally open to improvements - I just wanted to get a sense from others whether this PR looks good to proceed

[k8s-ci-robot]

PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.