What would you like to be added:
Native support for running karmada-agent components out of cluster for managed platforms.
Why is this needed:
At Bloomberg, we run a fully-managed multi-cluster platform built atop Karmada to support platform teams who want to leverage multi-cluster orchestration to adhere to the firm's high availability and disaster recovery compliance mandates.
Part of the managed service offering is fully automated member cluster registration for managed Karmada instances. To achieve that, as part of each managed Karmada control plane, we run a member cluster registration sub-system as shown below.
In that sub-system, we have three main components:
- cluster-provider: This component integrates with our cluster inventory API to dynamically sync cluster state for the managed Karmada control plane.
- cluster-credentials-provider: This component integrates with our token exchange service to provide the credentials needed to access the registered member clusters.
- cluster-agent-provider: For clusters registered in
Pull mode, we also need to dynamically provision karmada-agent instances for each Pull mode member cluster. Those agents need to run along side the other control plane components (e.g., karmada-controler-manager) on the host cluster, not on the member cluster. To make that work, we just provide the agent with two kube configs. One to connect to the Karmada API server, and the other to connect to the member cluster.
At a high level, the flow is as follows:
cluster-provider, using our cluster inventory API as the authoritative data source, registers clusters with the Karmada API server.cluster-credentials-provider, watching event sources from the Karmada API server and integrated with our token exchange service, provides the credentials needed to connect to the member clusters.- For clusters registered in
Pull mode, cluster-agent-provider, also watching event sources from the Karmada API server, will provision a karmada-agent instance for each cluster.
One of the responsibilities of the karmada-agent component is to do all the plumbing required to register a member cluster with the Karmada control plane. Since we have controllers in our cluster registration sub-system doing that work, we need a way (e.g., using a flag) to turn off that functionality to avoid conflicts or weird edge cases that might arise for our use case.
We have a tenant running at very large scale who wants their member clusters registered in Pull mode. At that scale, we have to do all we can to maximize performance. One of the conclusions from the most recently published Karmada scalability test is a better performance profile when member clusters are registered in Pull mode. We plan to also conduct a scalability test for their specific use case using both Push mode and Pull mode registration strategies to gather more data on this.
What would you like to be added:
Native support for running
karmada-agentcomponents out of cluster for managed platforms.Why is this needed:
At Bloomberg, we run a fully-managed multi-cluster platform built atop Karmada to support platform teams who want to leverage multi-cluster orchestration to adhere to the firm's high availability and disaster recovery compliance mandates.
Part of the managed service offering is fully automated member cluster registration for managed Karmada instances. To achieve that, as part of each managed Karmada control plane, we run a member cluster registration sub-system as shown below.
In that sub-system, we have three main components:
Pullmode, we also need to dynamically provisionkarmada-agentinstances for eachPullmode member cluster. Those agents need to run along side the other control plane components (e.g.,karmada-controler-manager) on the host cluster, not on the member cluster. To make that work, we just provide the agent with two kube configs. One to connect to the Karmada API server, and the other to connect to the member cluster.At a high level, the flow is as follows:
cluster-provider, using our cluster inventory API as the authoritative data source, registers clusters with the Karmada API server.cluster-credentials-provider, watching event sources from the Karmada API server and integrated with our token exchange service, provides the credentials needed to connect to the member clusters.Pullmode,cluster-agent-provider, also watching event sources from the Karmada API server, will provision akarmada-agentinstance for each cluster.One of the responsibilities of the
karmada-agentcomponent is to do all the plumbing required to register a member cluster with the Karmada control plane. Since we have controllers in our cluster registration sub-system doing that work, we need a way (e.g., using a flag) to turn off that functionality to avoid conflicts or weird edge cases that might arise for our use case.We have a tenant running at very large scale who wants their member clusters registered in
Pullmode. At that scale, we have to do all we can to maximize performance. One of the conclusions from the most recently published Karmada scalability test is a better performance profile when member clusters are registered inPullmode. We plan to also conduct a scalability test for their specific use case using bothPushmode andPullmode registration strategies to gather more data on this.