[Umbrella] Workload Affinity Feature - phase I

[RainbowMango]

Summary:

This issue is used to track the iteration tasks of feature Workload Affinity described at #4968.
See proposal(#6685) for more details.

Iteration Tasks -- Part-1: workload anti-affinity:

• Finalize the proposal. (@mszacillo, Adding proposal for workload affinity and anti-affinity support #6685)
• Decouple placement(.spec.placement) in ResourceBinding from PropagationPolicy
  • Refactor the ResourceBinding API to define its own independent placement structure instead of reusing the one from PropagationPolicy, as their placement semantics and fields are not identical.
• Introduce anti-affinity field to PropagationPolicy.
• Grab the affinity group from the resource template and populate it in ResourceBinding.
• Introduce a new feature gate TBD for this feature, started with alpha and disabled by default.
  • Feature gate name candidates: WorkloadAffinity, SingleWorkloadAffinityGroup.
• karmda-scheduler: build a global index based on the affinity group identifier declared in ResourceBinding.
• Introduce a filter plugin for filter clusters by taking anti-affinity group into account.
  • A tricky part of this, we need to make sure the ResourceBinding the plugin is looking at is the latest one. (cache might not get a chance to update)
• Setup E2E tests
  • TBD
• Add documentation

Iteration Tasks -- Part-2: workload affinity:

[mszacillo]

Thanks @RainbowMango for populating the iteration tasks!

Overall I think the tasks make sense, though I suspect building a global index in the scheduler could probably be broken down. But that can be done once we start working on it.

Out of curiosity, could you elaborate on:

Refactor the ResourceBinding API to define its own independent placement structure instead of reusing the one from PropagationPolicy, as their placement semantics and fields are not identical.

Was this a change that we've been meaning to do for a while, and this feature provides a good motivation for prioritizing it? I wasn't aware that this was an ongoing concern.

[RainbowMango]

I was thinking about how and where to store the resolved affinity group in the ResourceBinding. As I commented on your branch at mszacillo@752f5d9#r173001777.

Currently, the .spec.placement of ResourceBinding shares the Go structure with PropagationPolicy, we still need to find a place to store the resolved affinity group based on label.
I'm thinking the following approaches:

1. Decoping ResourceBinding and PropagationPolicy: that means let ResourceBinding have its own placement.
   It involves a lot of change, I'm not sure if it is worth or necessary to do it.
2. 👍 Introduce another field into the ResourceBinding spec, like .spec.appfiltyGroup string.
3. Introduce another field into the ObjectReference in ResourceBinding, exactly as you have done in your demo.

[edit]:
I tend to the second approach, and leave the API design idea at #6685 (comment).