Option for KWOK additional `AllowUndefined` labels

[alec-rabold]

Description

What problem are you trying to solve?

When using KWOK as a simulation environment for other cloudproviders like AWS, I think there's some friction in syncing up the NodeSelector Labels that are allowed to be undefined for each; for example:

• karpenter.k8s.aws/instance-cpu --> karpenter.kwok.sh/instance-cpu
• karpenter.k8s.aws/instance-memory --> karpenter.kwok.sh/instance-memory
• karpenter.k8s.aws/instance-local-nvme --> custom requirement in Offering
  etc.

Most of the labels I'm interested in have a KWOK counterpart (like those above) and it's not too difficult to translate those from AWS NodePools into KWOK NodePools, but for certain labels like karpenter.k8s.aws/instance-generation it's a bit harder since IIUC specifying that in the instance offering would mean that a Pod needs to explicitly set it.

I'm wondering if it makes sense to expose an option in the KWOK provider to allow additional requirements/labels to be undefined?

(I'm assuming the AllowUndefined labels between KWOK<-->AWS is the issue but not 100% sure)

This is the NodePool and Instance Type/Offering I was testing with:

Instance Type

[
  {
    "name": "m6a.12xlarge",
    "offerings": [
      {
        "Price": 2.0736,
        "Available": true,
        "Requirements": [
          {
            "key": "karpenter.sh/capacity-type",
            "operator": "In",
            "values": ["on-demand"]
          },
          {
            "key": "topology.kubernetes.io/zone",
            "operator": "In",
            "values": ["us-west-2a", "us-west-2b", "us-west-2c"]
          },
          {
            "key": "kubernetes.io/arch",
            "operator": "In",
            "values": ["amd64"]
          },
          { "key": "kubernetes.io/os", "operator": "In", "values": ["linux"] },
          {
            "key": "karpenter.k8s.aws/instance-family",
            "operator": "In",
            "values": ["m6"]
          },
          {
            "key": "karpenter.k8s.aws/instance-category",
            "operator": "In",
            "values": ["m"]
          },
          {
            "key": "karpenter.k8s.aws/instance-size",
            "operator": "In",
            "values": ["12xlarge"]
          },
          {
            "key": "karpenter.k8s.aws/instance-generation",
            "operator": "In",
            "values": ["6"]
          },
          {
            "key": "karpenter.k8s.aws/instance-type",
            "operator": "In",
            "values": ["m6a.12xlarge"]
          },
          {
            "key": "karpenter.k8s.aws/instance-cpu",
            "operator": "In",
            "values": ["48"]
          },
          {
            "key": "karpenter.k8s.aws/instance-memory",
            "operator": "In",
            "values": ["206"]
          }
        ]
      }
    ],
    "architecture": "amd64",
    "operatingSystems": ["linux"],
    "resources": {
      "cpu": "48",
      "ephemeral-storage": "100Gi",
      "memory": "192Gi",
      "pods": "200"
    }
  }
]

NodePool

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: default
spec:
  disruption:
    budgets:
      - nodes: 10%
    consolidateAfter: 0s
    consolidationPolicy: WhenEmptyOrUnderutilized
  template:
    metadata: {}
    spec:
      expireAfter: Never
      nodeClassRef:
        group: karpenter.kwok.sh
        kind: KWOKNodeClass
        name: default
      requirements:
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values:
            - c
            - m
            - r
        - key: karpenter.k8s.aws/instance-size
          operator: In
          values:
            - 12xlarge
            - 16xlarge
            - 24xlarge
            - 32xlarge
            - 48xlarge
        - key: karpenter.k8s.aws/instance-generation
          operator: Gt
          values:
            - "5"
        - key: karpenter.k8s.aws/instance-cpu
          operator: Gt
          values:
            - "2"
        - key: karpenter.sh/capacity-type
          operator: In
          values:
            - on-demand
        - key: kubernetes.io/os
          operator: In
          values:
            - linux
        - key: kubernetes.io/arch
          operator: In
          values:
            - amd64
      terminationGracePeriod: 25h

---

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[jonathan-innis]

Just to make sure I understand what you mean, you're talking about the well-known label set, right? There's some issue with the scheduler because it doesn't understand that something can be undefined because it hasn't been injected into the well-known label set?

Is this something that we could solve by auto-injecting all of the label keys that we find in the provided instance-types.json?

[jonathan-innis]

/triage needs-information
/priority important-soon

[alec-rabold]

Yep that sounds accurate; although I'm not super familiar with the scheduling mechanics so maybe explaining what I'm trying to accomplish would be easier:

I'm hoping to use the KWOK provider and give it a list of instance-types that have the same labels that the AWS provider uses (e.g. matching a m6.12xlarge instance). With that, I'd then use the same NodePool definition that I'm using in an AWS environment (with the only thing changed being the nodeClassRef to point to a KWOKNodeClass).

My understanding of the problem is that when I add a requirement to my instance_types.json like this:

          {
            "key": "karpenter.k8s.aws/instance-generation",
            "operator": "In",
            "values": ["6"]
          },

then it's treated like a custom-label instead of a well-known label since it's the KWOK provider instead of AWS provider.

Is this something that we could solve by auto-injecting all of the label keys that we find in the provided instance-types.json?

IIUC that sounds reasonable!

[alec-rabold]

Seemed to get something functional with this small PR: #2053