Can karpenter provide something like PodDisruptionBudget?

[zwwhdls]

Description

What problem are you trying to solve?
JuiceFS users using JuiceFS CSI Driver in Kubernetes with karpenter have encountered the following problem:

I have pods on the same node that are depend on each other.

For example pod B depends on pod A and they are on the same node. And when B is deleted, A will be deleted by its controller(kubelet will trigger A's controller). But when A is deleted before B, B will not run well and result in Error.

But when I did a disruption in node. A and B are evicted in the same time, and B will cause Error which can not be accepted by consumers.

I tried two methods:

1. setup PodDisruptionBudget for A (which is proven to be effective in Kubernetes without karpenter), but karpenter ignored it, and evict both A and B.
2. set annotation karpenter.sh/do-not-disrupt in A , but karpenter do not trigger disruption anymore.

PDB will prevent A from be evicted during drain, and A will be deleted by its controller after B is deleted, so the entire drain can be successful. But it seems not work in karpenter.

Can karpenter provide something like PDB? It will help a lot in JuiceFS's case. Thanks!

How important is this feature to you?

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[jonathan-innis]

setup PodDisruptionBudget for A (which is proven to be effective in Kubernetes without karpenter), but karpenter ignored it, and evict both A and B

Can you show the PodDisruptionBudget that you are setting up? That statement leads me to believe that Karpenter doesn't respect PDBs but they should be considered during graceful disruption and in general during eviction

[zwwhdls]

@jonathan-innis The PodDisruptionBudget:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: jfs-pdb
  namespace: kube-system
spec:
  minAvailable: "100%"
  selector:
    matchLabels:
      app.kubernetes.io/name: juicefs-mount

[jonathan-innis]

And can you also share the pod manifests?

[elijah-rou]

Here's an example of one of the pods. We're looking to enforce an exit order, since JuiceFS creates a pod for an application's PVC mount. Since the application needs this pod to use the PVC mount, any disruption to the PVC mount can cause the application to crash if it attempts to use the PVC (read/write a file). This wouldn't be an issue by itself, but it creates a problem when Karpenter wants to do consolidation/disruption. We essentially want Karpenter to consider the node for disruption, but keep the mount pod alive until all other non-daemonset pods have finished first.

At the moment, there are only 2 extremes. Either we remove karpenter.sh/do-not-disrupt or not use a PDB, but the mount pod gets killed prematurely on disruption, or we use use one of them and Karpenter is never able to consolidate.

apiVersion: v1
kind: Pod
metadata:
  annotations:
    juicefs-delete-delay: 1m
    karpenter.sh/do-not-disrupt: "true"
  finalizers:
    - juicefs.com/finalizer
  labels:
    app.kubernetes.io/name: juicefs-mount
    volume-id: pvc-abde
  name: juicefs-ip-10-0-13-148.internal-pvc-abde
  namespace: juicefs
spec:
  containers:
    - command:
        - sh
        - -c
        - |-
          cp /etc/juicefs/juice-prod.conf /root/.juicefs
          exec /sbin/mount.juicefs juice-prod /jfs/pvc-abde-wtcyil -o foreground,no-update,buffer-size=1024,cache-dir=/mnt/juicefs-cache,cache-group=jfscache,cache-size=204800,max-downloads=200,max-uploads=200,no-sharing
      env:
        - name: JFS_INSIDE_CONTAINER
          value: "1"
        - name: secretkey
          valueFrom:
            secretKeyRef:
              key: secretkey
              name: juicefs-pvc-abde-secret
        - name: token
          valueFrom:
            secretKeyRef:
              key: token
              name: juicefs-pvc-abde-secret
        - name: JFS_FOREGROUND
          value: "1"
        - name: JFS_SUPER_COMM
          value: /tmp/fuse_fd_csi_comm.sock
      image: juicedata/mount:ee-5.1.9-d809773
      imagePullPolicy: IfNotPresent
      lifecycle:
        preStop:
          exec:
            command:
              - sh
              - -c
              - +e
              - umount -l /jfs/pvc-abde-wtcyil;
                rmdir /jfs/pvc-abde-wtcyil; exit
                0
      name: jfs-mount
      resources:
        limits:
          cpu: "2"
          memory: 6Gi
        requests:
          cpu: 100m
          memory: 2Gi
      securityContext:
        privileged: true
        runAsUser: 0
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
        - mountPath: /etc/juicefs
          name: init-config
        - mountPath: /jfs
          mountPropagation: Bidirectional
          name: jfs-dir
        - mountPath: /tmp
          name: jfs-fuse-fd
        - mountPath: /etc/updatedb.conf
          name: updatedb
        - mountPath: /mnt/juicefs-cache
          name: cachedir-0
        - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
          name: kube-api-access-jnzqs
          readOnly: true
  dnsPolicy: ClusterFirstWithHostNet
  enableServiceLinks: true
  hostname: pvc-abde
  nodeName: ip-10-0-13-148.internal
  preemptionPolicy: Never
  priority: 1000000000
  priorityClassName: juicefs-mount-priority-nonpreempting
  restartPolicy: OnFailure
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: juicefs-csi-node-sa
  serviceAccountName: juicefs-csi-node-sa
  terminationGracePeriodSeconds: 43200
  tolerations:
    - key: CriticalAddonsOnly
      operator: Exists
    - effect: NoSchedule
      key: node.kubernetes.io/disk-pressure
      operator: Exists
    - effect: NoSchedule
      key: node.kubernetes.io/memory-pressure
      operator: Exists
    - effect: NoExecute
      key: node.kubernetes.io/not-ready
      operator: Exists
    - effect: NoSchedule
      key: node.kubernetes.io/pid-pressure
      operator: Exists
    - effect: NoExecute
      key: node.kubernetes.io/unreachable
      operator: Exists
    - effect: NoSchedule
      key: node.kubernetes.io/unschedulable
      operator: Exists
  volumes:
    - name: init-config
      secret:
        defaultMode: 420
        items:
          - key: initconfig
            path: juice-prod.conf
        secretName: juicefs-pvc-abde-secret
    - hostPath:
        path: /var/lib/juicefs/volume
        type: DirectoryOrCreate
      name: jfs-dir
    - hostPath:
        path: /var/run/juicefs-csi/xyz
        type: DirectoryOrCreate
      name: jfs-fuse-fd
    - hostPath:
        path: /etc/updatedb.conf
        type: FileOrCreate
      name: updatedb
    - hostPath:
        path: /mnt/juicefs-cache
        type: DirectoryOrCreate
      name: cachedir-0
    - name: kube-api-access-jnzqs
      projected:
        defaultMode: 420
        sources:
          - serviceAccountToken:
              expirationSeconds: 3607
              path: token
          - configMap:
              items:
                - key: ca.crt
                  path: ca.crt
              name: kube-root-ca.crt
          - downwardAPI:
              items:
                - fieldRef:
                    apiVersion: v1
                    fieldPath: metadata.namespace
                  path: namespace

[jonathan-innis]

keep the mount pod alive until all other non-daemonset pods have finished first

We do have a pod ordering today. The order of eviction works in the following way:

1. Non-critical non-daemon pods
2. Non-critical daemon pods
3. Critical non-daemon pods
4. Critical daemon pods

We define "critical" to mean system-cluster-critical or system-node-critical. You can see that logic implemented here. It sounds to me that you could assign a critical priority class to the pod that you want pulled off the node last and that should solve your eviction ordering problem.

[jonathan-innis]

I think I get what you're saying at the top of the original issue finally: You're saying that Karpenter won't allow the pod to be disrupted because one of the PDBs is fully blocking. Karpenter obviously doesn't know that that dependency will be removed when the other pods are torn down.

Can you describe a little bit how you orchestrate the unblocking of the PDB once the application pods are torn down?

[elijah-rou]

@jonathan-innis I will try either of the priority classes you suggested. It will likely roundabout solve this issue then.

There is currently not a mechanism on the JuiceFS side (as far as I know, I'm not a core team member just a user) that would allow unblocking the PDB once reliant nodes are taken down. I think saw an issue a while back relating to exit order of pods to evict when Karpenter decides to disrupt. Might be useful for the Karpenter team to consider either the PriorityClass priority or some annotation on the pod (something along the lines of karpenter.sh/disrupt-order: "1", where all lower orders are evicted first) which specifies order for eviction. Happy to start authoring this should the fix you mentioned above not be suitable.

[zwwhdls]

Hi @jonathan-innis , JuiceFS mount pod uses system-cluster-critical by default. By in some situation, users will use customs priorityClass to prevent evict the application pod on the node. In this case(raised by @elijah-rou ), will Karpenter provide some mechanism to control the order of non-critical pods? Or just something like pdb to prevent pod to be evicted?

[elijah-rou]

Worth noting as well, I have observed that the logic for taking down system-node-critical and system-cluster-critical after "non-critical" nodes, does not apply under certain situations. For example, if I remove a node pool to disable the node type, Karpenter disrupts nodes from the pool, but attempts to evict every pod without respecting the order.

[jonathan-innis]

but attempts to evict every pod without respecting the order.

Which version are you talking about? The drain logic should work irrespective of the way that the node was disrupted. If it doesn't work this way, then we have a bug and we should open an issue in upstream here: https://github.com/kubernetes-sigs/karpenter/issues