Skip to content

Inconsistent pod distrbution when using minDomains of Topology Spread Constraints when testing between 3 AZs and 2 AZs #7585

Open
@vchintal

Description

@vchintal

Description

Observed Behavior:

When the Karpenter CR is limited to two zones (us-west-2a and us-west-2b), an application deployed with Topology Spread Constraints with maxSkew: 1 and minDomains: 1 is resulting in a distribution as show below:

AZ            Instance Type          Host                                          # of app pods                                                 
us-west-2b    t3.small               ip-10-0-29-165.us-west-2.compute.internal     1
us-west-2a    t3.small               ip-10-0-6-0.us-west-2.compute.internal        1

Expected Behavior:

AZ            Instance Type          Host                                          # of app pods                                                 
us-west-2b    t3.small               ip-10-0-18-102.us-west-2.compute.internal     5
us-west-2a    t3.small               ip-10-0-8-14.us-west-2.compute.internal       5

Few things to note:

  1. When the Karpenter CR has all three Availability Zones (us-west-2a,us-west-2b, us-west-2c) then the same settings on the application's Deployment work just fine
  2. Also, ironically enough, this result (show under expected behavior with Karpenter CR having two Availability Zones) can be achieved by commenting out minDomains and setting whenUnsatisfiable: ScheduleAnyway

Reproduction Steps (Please include YAML):

Karpenter CR

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: default
spec:
  template:
    spec:
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
      requirements:
        - key: "topology.kubernetes.io/zone"
          operator: In
          values: ["us-west-2a", "us-west-2b"]    
        - key: "karpenter.k8s.aws/instance-hypervisor"
          operator: In
          values: ["nitro"]
        - key: "karpenter.k8s.aws/instance-generation"
          operator: Gt
          values: ["2"]
  limits:
    cpu: 1000
  disruption:    
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 5s

Inflate Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: inflate
spec:
  replicas: 10
  selector:
    matchLabels:
      app: inflate
  template:
    metadata:
      labels:
        app: inflate
    spec:
      terminationGracePeriodSeconds: 0
      topologySpreadConstraints:
      - maxSkew: 1
        minDomains: 1
        whenUnsatisfiable: DoNotSchedule
        topologyKey: topology.kubernetes.io/zone
        labelSelector:
          matchLabels:
            app: inflate
      containers:
        - name: inflate
          image: public.ecr.aws/eks-distro/kubernetes/pause:3.7
          resources:
            requests:
              cpu: "1m"
            limits:
              cpu: "1"
              memory: "250Mi"

Versions:

  • Chart Version: 1.1.1
  • Kubernetes Version (kubectl version): v1.31.3
  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingtriage/needs-investigationIssues that need to be investigated before triaging

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions