Disable CC network resource goals when resource capacities are not set.

[kyguy]

Type of change

• Enhancement / new feature

Description

This PR adds logic to the Strimzi Cluster Operator to remove network resource related Cruise Control goals from the default.goals and hard.goals lists by default if the capacity configurations for those resources are not explicitly set in the .spec.kafka.resources or .spec.cruiseControl.brokerCapacity sections of the Kafka custom resource. The following Cruise Control goals are affected:

com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkInboundCapacityGoal
com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkOutboundCapacityGoal
com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkInboundUsageDistributionGoal
com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkOutboundUsageDistributionGoal
com.linkedin.kafka.cruisecontrol.analyzer.goals.PotentialNwOutGoal

This will prevent users from balancing partitions based on network resources without defining capacities for those network resources in their Kafka custom resource.

Addresses the issue raised here: #11409

(EDITED): Updated to reduce scope to focus on network-related goals.

Checklist

Please go through this checklist and make sure all applicable tasks have been done

• Write tests
• Make sure all tests pass
• Update documentation
• Check RBAC rights for Kubernetes / OpenShift roles
• Try your changes from Pod inside your Kubernetes and OpenShift cluster, not just locally
• Reference relevant issue(s) and close them after merging
• Update CHANGELOG.md
• Supply screenshots for visual changes, such as Grafana dashboards

[scholzj]

As I suggested in one of the comments. Resources are not configured in the Kafka CR anymore. It would be also good to understand what is the plan when some nodes have resources defined and some not. What will you do then?

[kyguy]

It would be also good to understand what is the plan when some nodes have resources defined and some not. What will you do then?

Thanks for the catch (and review), we only want to consider the CPU capacity for a cluster to be considered to be properly configured for CPU-dependent Cruise Control goals if there is a capacity value explicitly configured for every broker whether through the general capacity or broker specific overrides, or the resource requirements. So if none of these are defined for a broker, including the resource requirement, we want to disable the CPU-dependent Cruise Control goals.

I had wrongly assumed that all the node pools had their resources defined in the kafkaBrokerResources map, I have updated the logic to disable the CPU-dependent Cruise Control goals if there is a broker in the kafkaBrokerNodes set that doesn't have it's node pool (and resource requirements) defined in the kafkaBrokerResources map.

[scholzj]

It seems to have quite a lot of changes given this now should be a pretty basic if-else thing. Should we split this into a refactoring PR and network goals PR? TBH, it is pretty hard to review what changes are really related to what.

You should probably also update the documentation to clarify the default goals?

[kyguy]

Should we split this into a refactoring PR and network goals PR? TBH, it is pretty hard to review what changes are really related to what.

Yeah that makes sense, just isolated the refactoring work into a different PR here [1] Once that is sorted, this PR will be a lot easier to review.

[1] #11615

[scholzj]

Just some quick comments.

[scholzj]

Few more nits, but mostly looking good.

[scholzj]

One more nit. LGTM otherwise. Thanks.

[scholzj]

/azp run regression

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[ppatierno]

LGTM. Just a couple of nits.

[scholzj]

@tinaselenge Do you have anything more on this? You had some comments on this PR in the past. Thanks.

[tinaselenge]

Thanks for the changes. LGTM!