Enhance PodGrouper plugin matching to fallback through owner chain

[singh1203]

What you would like to be added?

Enhance the PodGrouper plugin matching logic so that, after resolving the full owner reference chain for a pod (for example, Pod → Job → JobSet → Kubeflow trainer), the plugin is not selected solely based on the topmost owner type with an immediate fallback to the default plugin when no match is found. Instead, PodGrouper should iterate over the owner chain, starting from the topmost owner and moving down towards the pod, attempting to match a plugin for each owner type in order, using the existing plugin hub implementation (for example, the Job plugin registration in hub.go). Only if no plugin is found for any owner in the chain should the default plugin be used, so that existing workload-specific plugins continue to drive PodGroup creation even when new intermediate owner types (such as JobSet) are introduced without a dedicated plugin.

Useful references:

PodGrouper main flow and owner resolution: pkg/podgrouper/podgrouper/podgrouper.go.

Plugin hub: pkg/podgrouper/podgrouper/hub/hub.go

PodGrouper developer docs: docs/developer/pod-grouper.md

Why is this needed?

With the current behavior, extending RBAC and ownership chains to include new types (e.g., JobSet) can unintentionally degrade grouping semantics, because PodGrouper climbs to the new topmost owner, fails to match a plugin, and immediately falls back to the default per-pod grouping instead of leveraging an already available, more suitable plugin for an intermediate owner (such as batch/v1 Job). An owner-chain–aware fallback mechanism makes PodGrouper more robust to incremental integrations, prevents regressions in gang scheduling behavior when new workload types are added, and ensures that existing plugins continue to be applied wherever they can accurately model the desired scheduling semantics until dedicated plugins for new owners are implemented.

[singh1203]

Please assign.
Thank you!

[davidLif]

Hello @singh1203 ,

I know that there is currently an effort to propagate annotation+labels (mostly for kai features) and priority calculations from skipped top owners to the actual used owner.
I would consider adding something similar as part of this design as well.

[enoodle]

@singh1203
This makes sense, but can you explain the use case? In what scenario will this be used? Why would someone add permission to a CRD that is not actually supported by the pod grouper?
If someone has a new CRD they want to use with KAI but it is not supported with the PodGrouper, why can either add support or create the PodGroup object on their own. This enhanced error handling means that they manually added their CRD to the pod grouper cluster role and then submitted it, wouldn't it be easier for them to specify the correct PodGroup and point all the pods to it so that they can get the best fit for their case?

Anyway if this can be implemented without adding complexity to the code then I am for it because it makes sense.

[romanbaron]

It is exactly the issue we discussed in the bi-weekly meeting on Jan 5th. When someone adds a new PodGrouper plugin they need to add all the relevant types (GVK) in the OwnerReference chain to PodGrouper's ClusterRole, from the pod all the way to the top owner. It doesn't always mean that all these types have a dedicated plugin and these permissions are not scoped to this specific workload/top-owner/ownerref-chain, it also affects the PodGrouper's ability to search for the top owner when other workloads use one or more of these new types.

For example, when adding a plugin for Kubeflow trainer v2 the owner reference chain looks like this:
Kubeflow trainer -> JobSet -> Job -> Pod

The PodGrouper doesn't have a dedicated plugin for JobSet yet (WIP here) so if someone schedules JobSet with KAI, the PodGrouper groups the pods by jobs (not perfect but probably better than grouping as single pods).

So, if we simply add this Kubeflow trainer plugin and add JobSet to PodGrouper's ClusterRole, we unintentionally affect the way JobSet is grouped. From this point and on, the PodGrouper will be able to read the JobSet object and it will unsuccessfully attempt to match a specific plugin for it, and will not find one. The fallback is the default grouper plugin (scheduled as separate pods).