[Feature] FederatedResourceQuota Enhancement

[mszacillo]

Summary
This feature aims to enhance the existing FederatedResourceQuota so it can impose namespaced resource limits directly on the Karmada control-plane level. This feature enhancement has several benefits, including:

1. Track resource consumption and limits from a unified place
2. Prevent users from over-consuming aggregate resources across clusters
3. Easier for users of application and cluster failover to ensure that fallback clusters have enough resources to house all the user's workflows

Proposal:

• FederatedResourceQuota Enhancement: Proposal to enhance FederatedResourceQuota to enforce resource limits directly on Karmada level #6181 (@mszacillo)

Iteration Tasks -- Part-1: FederatedResourceQuota Enhancement MVP

• Feature Flag addition: FederatedQuotaEnforcement add FederatedQuotaEnforcement feature flag #6366 (@liwang0513)
• [API Change] Clarify intent and usage of staticAssignments in the FRQ API Twist the behavior of staticAssignments in FederatedResourceQuota #6351 (@RainbowMango)
• [Controller change] Adjust the behavior of the legacy staticAssignments to skip creating redundant ResourceQuotas to member clusters FederatedResourceQuota: skip creating redundant ResourceQuotas to member clusters #6363 (@zhzhuang-zju)
• Update federated-resource-quota-status-controller so that staticAssignment only updates AggregatedStatus staticAssignment only updates AggregatedStatus #6382 (@liwang0513)
• Create federated-resource-quota-enforcement controller: Create federated-resource-quota-enforcement-controller #6367 (@mszacillo)
  - Watch create/update events for FRQ, and update federatedresourcequota.status.overall
  - Watch deletion of RB -> find related FRQ -> recalculate quota usage
  - Periodically recalculate FRQ status
• Add ResourceBinding validation webhook to monitor creates + updates to resourcebinding.spec and resourcebinding.spec.replicaRequirements after scheduling result Create RB ValidatingWebhook for FederatedResourceQuota check #6377 (@seanlaii)
  - Check delta in quota usage and deny request if above limits
  - Otherwise update federatedresourcequota.status.overallUsed and then approve update request
• Periodically reconcile resource quota enforcement controller: Periodically reconcile FederatedResourceQuota to update the usage state #6407 (@zhzhuang-zju)
• Validate FederatedResourceQuota length to no more than 63 chars (feature: add name length check for fedresourcequota #2168)
• Add new feature documentation #835 (@mszacillo)
• Functional verification and add e2e tests add e2e test to test FederatedResourceQuotaEnforcement #6408 (@zhzhuang-zju)

[mszacillo]

@RainbowMango Let me know if this looks alright!

Also, @zhzhuang-zju, do you have any preference on how to subdivide this work between us?

[RainbowMango]

Generally looks great! Thanks.
I will leave comments if I want to twist or add something.
For now, I'm thinking of introducing a feature gate, like what we did for the StatefulFailoverInjection, do you have any suggestion for the feature name?

[RainbowMango]

@mszacillo The origin behavior of staticAssignments might be a distraction for this feature as with this field unset, the controller still creates an empty ResourceQuota to all clusters, that doesn't make sense, because empty ResourceQuota can not do anything.

Sent #6351 for updating the comments, please take a look, and proposing two iterated tasks:

• (API) Adjust the behavior of the legacy staticAssignments to skip creating redundant ResourceQuotas to member clusters.
• (Controller) Adjust the behavior of the legacy staticAssignments to skip creating redundant ResourceQuotas to member clusters.

[mszacillo]

Great! I've gone ahead and updated the iteration tasks to include the two other staticAssignments tasks.

Just to double check, do these iteration tasks make sense as well:

• Update federated-resource-quota-status-controller to skip reconcile if staticAssignment is not set
• FederatedResourceQuota validating webhook should block updates to enable staticAssignment on Update requests

The thought it is that the staticAssignment controller and new federated resource quota controller will both attempt to update the status of the federated resource quota resource. So we want to prevent users from toggling between the two features. If someone wants to use staticAssignment, they can delete their quota and reapply using staticAssignment.

For now, I'm thinking of introducing a feature gate, like what we did for the StatefulFailoverInjection, do you have any suggestion for the feature name?

Perhaps FederatedQuotaEnforcement? Or GlobalQuotaEnforcement?

[RainbowMango]

The thought it is that the staticAssignment controller and new federated resource quota controller will both attempt to update the status of the federated resource quota resource.

This is a technical issue and can be resolved. Concurrent updates to the same resource object by multiple controllers are a common scenario. By clearly defining the responsibilities of each controller to ensure they do not modify the same fields simultaneously, we can resolve this problem.

In this case, we can let the federated-resource-quota-status-controller only update the .status.aggregatedStatus:

karmada/pkg/apis/policy/v1alpha1/federatedresourcequota_types.go

Lines 103 to 105 in 32a4076

	// AggregatedStatus is the observed quota usage of each cluster.
	// +optional
	AggregatedStatus []ClusterQuotaStatus `json:"aggregatedStatus,omitempty"`

, and the .status.overallUsed will be managed by the new controller.

FederatedResourceQuota validating webhook should block updates to enable staticAssignment on Update requests

So, we might not need to do this. The staticAssignments field is still an optional field. Once it is set, the origin controller will create ResourceQuota for member clusters.

Update federated-resource-quota-status-controller to skip reconcile if staticAssignment is not set

Basically yes, both the federated-resource-quota-status-controller and federated-resource-quota-sync-controller are for reconciling staticAssignment (I'm thinking of renaming it, by the way).

But there is a corner case that we probably can not just skip reconciliation even staticAssignment is unset, consider the following case: what if the staticAssignment is removed during an update operation, if we skip the reconciliation, the status(.status.aggregatedStatus) will not be cleaned up.

[RainbowMango]

Perhaps FederatedQuotaEnforcement? Or GlobalQuotaEnforcement?

I like the FederatedQuotaEnforcement as the feature name and feature gate name.

[mszacillo]

In this case, we can let the federated-resource-quota-status-controller only update the .status.aggregatedStatus:

Gotcha, I should have been more clear. The concern wasn't that two controllers are editing the status, but rather that they will be editing the same field OverallUsed. I'll edit the iteration tasks to account for this!

So, we might not need to do this. The staticAssignments field is still an optional field. Once it is set, the origin controller will create ResourceQuota for member clusters.

But there is a corner case that we probably can not just skip reconciliation even staticAssignment is unset, consider the following case: what if the staticAssignment is removed during an update operation, if we skip the reconciliation, the status(.status.aggregatedStatus) will not be cleaned up.

Makes sense! I'll edit this iteration task as well. Should I include the renaming of staticAssignment as part of this umbrella list, or should it be standalone?

[RainbowMango]

I'll edit this iteration task as well. Should I include the renaming of staticAssignment as part of this umbrella list, or should it be standalone?

OK, we can have one. Refactoring/Twisting the legacy feature is also part of the new feature.

[edit]:
Renaming the legacy controller may not be mandatory. Given that the new controller might be named federated-resource-quota-enforcement-controller, it seems unlikely to cause much misunderstanding. Then we will have 3 controllers:

• (legacy) federated-resource-quota-sync-controller: synchronizes Kubernetes ResourceQuota to member clusters based on staticAssignment field
• (legacy) federated-resource-quota-status-controller: aggregates and reports the usage status of ResourceQuota across all member clusters.
• (new) federated-resource-quota-enforcement-controller: Calculates and reports the usage based on the enforcement at the Federation side.

What do you think? Do you have any preference for the new controller name? @mszacillo @zhzhuang-zju

[RainbowMango]

By the way, please assign the task to yourself and @liwang0513 once you have selected it. PS: Tasks without an assignment are open for anyone to pick.