[Question] Best practices for implementing FederatedHPA with external metrics (KEDA + Redis) across member clusters

[krax1337]

I am looking for architectural guidance on how to implement a Federated Horizontal Pod Autoscaler (FederatedHPA) that relies on external metrics (specifically utilizing KEDA and Redis).

Current Setup:

• We have a multi-cluster setup managed by Karmada.
• In each member cluster, we currently run KEDA which monitors a Redis instance (list length) to drive the scaling of local workloads.
• Scaling decisions are currently isolated per cluster.

Desired Goal: I want to promote this scaling logic to the Karmada Control Plane using FederatedHPA.

• The FederatedHPA should monitor the external metrics (Redis list length).
• Based on these metrics, Karmada should calculate the total desired replicas.
• Karmada should then distribute these replicas across the member clusters according to our propagation policies.

The Question: What is the recommended approach to expose these external metrics to the Karmada Control Plane?

Should KEDA be installed on the Karmada Host Cluster to scale the FederatedDeployment directly?
Or, is there a way for FederatedHPA to aggregate/listen to the external metrics generated by the KEDA instances running inside the Member Clusters?

Now this logic works with Duplicate Scheduling Type, and every deployment scaled duplicated on other cluster. This is not working with Weighted.

[zhzhuang-zju]

@krax1337 Thank you for your feedback. Currently, FederatedHPA only supports resource metrics and custom metrics, while external metrics are still in the implementation phase.

karmada/pkg/metricsadapter/provider/externalmetrics.go

Lines 37 to 40 in a056fbe

	// GetExternalMetric will query metrics by selector from member clusters and return the result
	func (c *ExternalMetricsProvider) GetExternalMetric(_ context.Context, _ string, _ labels.Selector, _ provider.ExternalMetricInfo) (*external_metrics.ExternalMetricValueList, error) {
	return nil, fmt.Errorf("karmada-metrics-adapter still not implement it")
	}

From the existing architecture, Karmada collects metrics from member clusters through the metrics-adapter component. Therefore, for external metrics, I believe we can reference this architecture, so KEDA should be installed in the member clusters, and the metrics-adapter would collect KEDA's metrics (if possible, not yet verified) for subsequent calculations.

[krax1337]

@zhzhuang-zju Thank you for clarification! Do you know when this feature would be implemented?

And from our case? Maybe you could suggest something how can we implement this pattern KEDA-like but multicluster with Federated Deployment. Maybe there is a way to install KEDA directly on Karmada to scale deployments that later will be propagated?

[zhzhuang-zju]

Thank you for clarification! Do you know when this feature would be implemented?

@krax1337 Currently, there is no definite schedule. At least in release-1.17, there are no implementation plans yet. Of course, if there is demand, we would be happy to push for it.

And from our case? Maybe you could suggest something how can we implement this pattern KEDA-like but multicluster with Federated Deployment. Maybe there is a way to install KEDA directly on Karmada to scale deployments that later will be propagated?

Installing KEDA directly on Karmada might not work. First, KEDA needs to start pods, such as keda-operator, but the Karmada control plane does not support starting pods (configurable). Second, the business information that KEDA needs, such as Redis list length, is generated in member clusters and cannot be directly used by the control plane.

If my understanding of KEDA is correct, KEDA exposes external metrics through external.metrics.k8s.io to cooperate with HPA. Extending this pattern to FHPA, if FHPA also supports external metrics, it could achieve similar things as HPA.