Reconcile PodClique TopologyConstraints (#302)

* Changes for setting Topology aware scheduling constraints in PodGangs.
* Renamed ClusterTopologyConfiguration to TopologyAwareSchedulingConfiguration in operator config.
* Introduced a new condition TopologyLevelsUnavailable on PCS
* PackDomain field in corev1alpha1 TopologyConstraint is now required.
* When creating ClusterTopology, if host topology level is not defined in TopologyAwareSchedulingConfiguration then the operator will set this level in ClusterTopology as this is a required level.
* Adapted PodGang component to set pack constraints at all hierarchy levels.
* Introduced Conditions in PCS status.
* Introduced a new condition TopologyLevelsUnavailable in PCS status.
* Added reconciliation code to update the PCS status condition.
* feat: implement topology-aware scheduling and add annotations for PodGangs
* Added constants for TopologyLevelsUnavailable condition reason
* Added code to update or remove the condition on PCS.
* Create utility function for cluster topology with unit test
* Made PodCliqueSetStatus.Conditions optional
* Upgraded KAI scheduler version dependency for e2e test to v0.12.0
* Changed polling timeout for e2e tests to 4 mins due to repeated timeouts on GHA
* Removed NVIDIA GPU operator to be installed as its not required.
* Fixed constant values for condition reasons and reduced the no of control plane servers for e2e from 3 to 1
* Moved `synchronizeTopology` in main to clustertopology package.
* Added restartPolicy to Always for Grove operator deployment.
* Removed defaulting preferred constraint to Host topology domain. This will be set later after requirements are clear.
* Moved GetClusterTopologyLevels to clustertopology package.
* Added docstrings to computeTopologyLevelsUnavailableCondition and
  mutateTopologyLevelUnavailableConditions functions.
* Changed the docstring for TopologyPackConstraint Preferred and
  Required fields.
* Reworded docstring for PodGang TopologyConstraint.

Signed-off-by: Madhav Bhargava <madhav.bhargava@sap.com>

---------

Signed-off-by: Madhav Bhargava <madhav.bhargava@sap.com>
Signed-off-by: Ron Kahn <rkahn@nvidia.com>
Co-authored-by: Ron Kahn <rkahn@nvidia.com>

Author: unmarshall

docs/api-reference/operator-api.md

@@ -424,6 +424,7 @@ _Appears in:_
 | Field | Description | Default | Validation |
 | --- | --- | --- | --- |
 | `observedGeneration` _integer_ | ObservedGeneration is the most recent generation observed by the controller. |  |  |
+| `conditions` _[Condition](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.33/#condition-v1-meta) array_ | Conditions represents the latest available observations of the PodCliqueSet by its controller. |  |  |
 | `lastErrors` _[LastError](#lasterror) array_ | LastErrors captures the last errors observed by the controller when reconciling the PodCliqueSet. |  |  |
 | `replicas` _integer_ | Replicas is the total number of PodCliqueSet replicas created. |  |  |
 | `updatedReplicas` _integer_ | UpdatedReplicas is the number of replicas that have been updated to the desired revision of the PodCliqueSet. | 0 |  |
@@ -638,8 +639,8 @@ allowing workload operators a consistent way to reference topology levels when d
 
 
 _Appears in:_
-- [ClusterTopologyConfiguration](#clustertopologyconfiguration)
 - [ClusterTopologySpec](#clustertopologyspec)
+- [TopologyAwareSchedulingConfiguration](#topologyawareschedulingconfiguration)
 
 | Field | Description | Default | Validation |
 | --- | --- | --- | --- |
@@ -689,23 +690,6 @@ _Appears in:_
 | `acceptContentTypes` _string_ | AcceptContentTypes defines the Accept header sent by clients when connecting to the server,<br />overriding the default value of 'application/json'. This field will control all connections<br />to the server used by a particular client. |  |  |
 
 
-#### ClusterTopologyConfiguration
-
-
-
-ClusterTopologyConfiguration defines the configuration for topology-aware scheduling.
-
-
-
-_Appears in:_
-- [OperatorConfiguration](#operatorconfiguration)
-
-| Field | Description | Default | Validation |
-| --- | --- | --- | --- |
-| `enabled` _boolean_ | Enabled indicates whether topology-aware scheduling is enabled. |  |  |
-| `levels` _[TopologyLevel](#topologylevel) array_ | Levels is an ordered list of topology levels from broadest to narrowest scope.<br />Used to create/update the ClusterTopology CR at operator startup. |  |  |
-
-
 #### ControllerConfiguration
 
 
@@ -883,6 +867,23 @@ _Appears in:_
 | `metrics` _[Server](#server)_ | Metrics is the configuration for serving the metrics endpoint. |  |  |
 
 
+#### TopologyAwareSchedulingConfiguration
+
+
+
+TopologyAwareSchedulingConfiguration defines the configuration for topology-aware scheduling.
+
+
+
+_Appears in:_
+- [OperatorConfiguration](#operatorconfiguration)
+
+| Field | Description | Default | Validation |
+| --- | --- | --- | --- |
+| `enabled` _boolean_ | Enabled indicates whether topology-aware scheduling is enabled. |  |  |
+| `levels` _[TopologyLevel](#topologylevel) array_ | Levels is an ordered list of topology levels from broadest to narrowest scope.<br />Used to create/update the TopologyAwareScheduling CR at operator startup. |  |  |
+
+
 #### WebhookServer
 
 

docs/api-reference/scheduler-api.md

@@ -86,7 +86,7 @@ _Appears in:_
 | --- | --- | --- | --- |
 | `podgroups` _[PodGroup](#podgroup) array_ | PodGroups is a list of member pod groups in the PodGang. |  |  |
 | `topologyConstraint` _[TopologyConstraint](#topologyconstraint)_ | TopologyConstraint defines topology packing constraints for entire pod gang.<br />Translated from PodCliqueSet.TopologyConstraint.<br />Updated by operator on each reconciliation when PodCliqueSet topology constraints change. |  |  |
-| `topologyConstraintGroupConfigs` _[TopologyConstraintGroupConfig](#topologyconstraintgroupconfig) array_ | TopologyConstraintGroupConfigs defines groups of PodGroups for topology-aware placement.<br />Enhanced with topology constraints for PCSG-level packing.<br />Updated by operator on each reconciliation when PCSG topology constraints change. |  |  |
+| `topologyConstraintGroupConfigs` _[TopologyConstraintGroupConfig](#topologyconstraintgroupconfig) array_ | TopologyConstraintGroupConfigs defines TopologyConstraints for a group of PodGroups when it is a strict subset<br />of total number of PodGroups for topology-aware placement. |  |  |
 | `priorityClassName` _string_ | PriorityClassName is the name of the PriorityClass for the PodGang. |  |  |
 | `reuseReservationRef` _[NamespacedName](#namespacedname)_ | ReuseReservationRef holds the reference to another PodGang resource scheduled previously.<br />During updates, an operator can suggest to reuse the reservation of the previous PodGang for a newer version of the<br />PodGang resource. This is a suggestion for the scheduler and not a requirement that must be met. If the scheduler plugin<br />finds that the reservation done previously was network optimised and there are no better alternatives available, then it<br />will reuse the reservation. If there are better alternatives available, then the scheduler will ignore this suggestion. |  |  |
 
@@ -159,16 +159,17 @@ _Appears in:_
 
 | Field | Description | Default | Validation |
 | --- | --- | --- | --- |
-| `name` _string_ | Name is the name of the topology constraint group.<br />It will drive from the corresponding PCSG name. |  |  |
+| `name` _string_ | Name is the name of the topology constraint group. |  |  |
 | `podGroupNames` _string array_ | PodGroupNames is the list of PodGroup names in the topology constraint group. |  |  |
-| `topologyConstraint` _[TopologyConstraint](#topologyconstraint)_ | TopologyConstraint defines topology packing constraints for this group.<br />Enables PCSG-level topology constraints.<br />Updated by operator when PodCliqueScalingGroup topology constraints change. |  |  |
+| `topologyConstraint` _[TopologyConstraint](#topologyconstraint)_ | TopologyConstraint defines topology packing constraints for this group. |  |  |
 
 
 #### TopologyPackConstraint
 
 
 
 TopologyPackConstraint defines a topology packing constraint.
+Each of Required and Preferred fields hold a topologyKey, e.g. "kubernetes.io/hostname" ( these are key of labels added on nodes).
 
 
 
@@ -177,7 +178,7 @@ _Appears in:_
 
 | Field | Description | Default | Validation |
 | --- | --- | --- | --- |
-| `required` _string_ | Required defines topology constraint that must be satisfied.<br />Holds topologyKey (not level name) translated from user's packLevel specification.<br />Example: "topology.kubernetes.io/rack" |  |  |
-| `preferred` _string_ | Preferred defines best-effort topology constraint.<br />Auto-generated by operator using strictest level topologyKey for optimization.<br />Scheduler can fallback to less strict levels if preferred cannot be satisfied.<br />Example: "kubernetes.io/hostname" |  |  |
+| `required` _string_ | Required defines a topology constraint that must be satisfied as a hard requirement. The workload will not be<br />scheduled if this constraint cannot be satisfied. Generally, it is easier for the scheduler to satisfy constraints<br />on topology domains with larger compute capacity, (e.g. zone or datacenter), than smaller domains, (e.g. host or<br />numa). Holds topologyKey (not level name) translated from user's packLevel specification.<br />Example: "topology.kubernetes.io/rack" |  |  |
+| `preferred` _string_ | Preferred defines best-effort topology constraint. Topology domains that provide the most optimized performance<br />with dense packing (e.g. host or numa) are typically used as preferred constraints for topology packing. It might be<br />harder to satisfy these constraints if the topology domains are limited in compute capacity. Since it is preferred<br />constraint, it is therefore not binding on the scheduler to mandatorily satisfy this packing constraint. Scheduler<br />can fall back to higher topology levels (upto Required constraint) if preferred cannot be satisfied.<br />Example: "kubernetes.io/hostname" |  |  |
 
 

operator/api/common/constants/constants.go

@@ -43,6 +43,8 @@ const (
 	// AnnotationDisableManagedResourceProtection is an annotation set by an operator on a PodCliqueSet to explicitly
 	// disable protection of managed resources for a PodCliqueSet.
 	AnnotationDisableManagedResourceProtection = "grove.io/disable-managed-resource-protection"
+	// AnnotationTopologyName is an annotation set on PodGang to allow KAI scheduler to discover which topology to use.
+	AnnotationTopologyName = "grove.io/topology-name"
 )
 
 // Constants for Grove environment variables
@@ -87,6 +89,9 @@ const (
 	// ConditionTypePodCliqueScheduled indicates that the PodClique has been successfully scheduled.
 	// This condition is set to true when number of scheduled pods in the PodClique is greater than or equal to PodCliqueSpec.MinAvailable.
 	ConditionTypePodCliqueScheduled = "PodCliqueScheduled"
+	// ConditionTopologyLevelsUnavailable indicates that the required topology levels defined on a PodCliqueSet for topology-aware scheduling are no longer available.
+	// This can happen when the ClusterTopology resource is modified which removes one or more levels required by the PodCliqueSet.
+	ConditionTopologyLevelsUnavailable = "TopologyLevelsUnavailable"
 )
 
 // Constants for Condition Reasons.
@@ -107,6 +112,14 @@ const (
 	ConditionReasonSufficientAvailablePCSGReplicas = "SufficientAvailablePodCliqueScalingGroupReplicas"
 	// ConditionReasonUpdateInProgress indicates that the resource is undergoing rolling update.
 	ConditionReasonUpdateInProgress = "UpdateInProgress"
+	// ConditionReasonClusterTopologyNotFound indicates that the ClusterTopology resource required for topology-aware scheduling was not found.
+	ConditionReasonClusterTopologyNotFound = "ClusterTopologyNotFound"
+	// ConditionReasonTopologyLevelsUnavailable indicates that the one or more required topology levels defined on a
+	// PodCliqueSet for topology-aware scheduling are no longer defined in the ClusterTopology resource.
+	ConditionReasonTopologyLevelsUnavailable = "ClusterTopologyLevelsUnavailable"
+	// ConditionReasonAllTopologyLevelsAvailable indicates that all required topology levels defined on a
+	// PodCliqueSet for topology-aware scheduling are defined in the ClusterTopology resource.
+	ConditionReasonAllTopologyLevelsAvailable = "AllClusterTopologyLevelsAvailable"
 )
 
 const (

operator/api/config/v1alpha1/types.go

@@ -55,16 +55,16 @@ var (
 
 // OperatorConfiguration defines the configuration for the Grove operator.
 type OperatorConfiguration struct {
-	metav1.TypeMeta  `json:",inline"`
-	ClientConnection ClientConnectionConfiguration `json:"runtimeClientConnection"`
-	LeaderElection   LeaderElectionConfiguration   `json:"leaderElection"`
-	Server           ServerConfiguration           `json:"server"`
-	Debugging        *DebuggingConfiguration       `json:"debugging,omitempty"`
-	Controllers      ControllerConfiguration       `json:"controllers"`
-	LogLevel         LogLevel                      `json:"logLevel"`
-	LogFormat        LogFormat                     `json:"logFormat"`
-	Authorizer       AuthorizerConfig              `json:"authorizer"`
-	ClusterTopology  ClusterTopologyConfiguration  `json:"clusterTopology"`
+	metav1.TypeMeta         `json:",inline"`
+	ClientConnection        ClientConnectionConfiguration        `json:"runtimeClientConnection"`
+	LeaderElection          LeaderElectionConfiguration          `json:"leaderElection"`
+	Server                  ServerConfiguration                  `json:"server"`
+	Debugging               *DebuggingConfiguration              `json:"debugging,omitempty"`
+	Controllers             ControllerConfiguration              `json:"controllers"`
+	LogLevel                LogLevel                             `json:"logLevel"`
+	LogFormat               LogFormat                            `json:"logFormat"`
+	Authorizer              AuthorizerConfig                     `json:"authorizer"`
+	TopologyAwareScheduling TopologyAwareSchedulingConfiguration `json:"topologyAwareScheduling"`
 }
 
 // LeaderElectionConfiguration defines the configuration for the leader election.
@@ -191,12 +191,12 @@ type AuthorizerConfig struct {
 	ExemptServiceAccountUserNames []string `json:"exemptServiceAccountUserNames,omitempty"`
 }
 
-// ClusterTopologyConfiguration defines the configuration for topology-aware scheduling.
-type ClusterTopologyConfiguration struct {
+// TopologyAwareSchedulingConfiguration defines the configuration for topology-aware scheduling.
+type TopologyAwareSchedulingConfiguration struct {
 	// Enabled indicates whether topology-aware scheduling is enabled.
 	Enabled bool `json:"enabled"`
 	// Levels is an ordered list of topology levels from broadest to narrowest scope.
-	// Used to create/update the ClusterTopology CR at operator startup.
+	// Used to create/update the TopologyAwareScheduling CR at operator startup.
 	// +optional
 	Levels []corev1alpha1.TopologyLevel `json:"levels,omitempty"`
 }

operator/api/config/v1alpha1/zz_generated.deepcopy.go

@@ -37,7 +37,7 @@ func ValidateOperatorConfiguration(config *configv1alpha1.OperatorConfiguration)
 	allErrs = append(allErrs, validateLeaderElectionConfiguration(config.LeaderElection, field.NewPath("leaderElection"))...)
 	allErrs = append(allErrs, validateClientConnectionConfiguration(config.ClientConnection, field.NewPath("clientConnection"))...)
 	allErrs = append(allErrs, validateControllerConfiguration(config.Controllers, field.NewPath("controllers"))...)
-	allErrs = append(allErrs, validateClusterTopologyConfiguration(config.ClusterTopology, field.NewPath("clusterTopology"))...)
+	allErrs = append(allErrs, validateTopologyAwareSchedulingConfig(config.TopologyAwareScheduling, field.NewPath("topologyAwareScheduling"))...)
 	return allErrs
 }
 
@@ -116,10 +116,10 @@ func mustBeGreaterThanZeroDuration(duration metav1.Duration, fldPath *field.Path
 	return allErrs
 }
 
-// validateClusterTopologyConfiguration validates the cluster topology configuration.
+// validateTopologyAwareSchedulingConfig validates the cluster topology configuration.
 // When cluster topology is enabled, it ensures the topology name and levels are provided,
 // and validates domain and key uniqueness.
-func validateClusterTopologyConfiguration(clusterTopologyCfg configv1alpha1.ClusterTopologyConfiguration, fldPath *field.Path) field.ErrorList {
+func validateTopologyAwareSchedulingConfig(clusterTopologyCfg configv1alpha1.TopologyAwareSchedulingConfiguration, fldPath *field.Path) field.ErrorList {
 	allErrs := field.ErrorList{}
 	if !clusterTopologyCfg.Enabled {
 		return allErrs