Mountpoint pods are sometimes unschedulable on Auto Mode

[simonfogliato]

/kind bug

NOTE: If this is a filesystem related bug, please take a look at the Mountpoint repo to submit a bug report

What happened?

Using the aws-mountpoint-s3-csi-driver as an EKS managed add-on on an EKS Auto Mode cluster, the per-volume mountpoint pods stay stuck in Pending and never reach Running. As a result, S3-backed PVCs never finish mounting and application pods are stuck in ContainerCreating with FailedMount.

The workload pod events show:

MountVolume.SetUp failed for volume "ws47d38456-d132-11f0-9f88-333856a768da-d2029-pv" :
rpc error: code = Internal desc = Could not mount "workspace-precisely-actively-faithful-mayfly" at
"/var/lib/kubelet/pods/7e37581b-ab99-4e90-9992-4e5f6c720626/volumes/kubernetes.io~csi/ws47d38456-d132-11f0-9f88-333856a768da-d2029-pv/mount":
Failed to wait for Mountpoint Pod "mp-p2khv" to be ready: mppod/watcher: mountpoint pod not found.
Seems like Mountpoint Pod is not in 'Running' status.

Pod Event:

MountVolume.SetUp failed for volume "ws47d38456-d132-11f0-9f88-333856a768da-d2029-pv" : rpc error: code = Internal desc = Could not mount "workspace-precisely-actively-faithful-mayfly" at "/var/lib/kubelet/pods/7e37581b-ab99-4e90-9992-4e5f6c720626/volumes/kubernetes.io~csi/ws47d38456-d132-11f0-9f88-333856a768da-d2029-pv/mount": Failed to wait for Mountpoint Pod "mp-p2khv" to be ready: mppod/watcher: mountpoint pod not found. Seems like Mountpoint Pod is not in 'Running' status. You can see it's status and any potential failures by running: `kubectl describe pods -n mount-s3 mp-p2khv`

Pod Mount:

kubectl describe pods -n mount-s3 mp-p2khv
Name:                 mp-p2khv
Namespace:            mount-s3
Priority:             1000000000
Priority Class Name:  mount-s3-critical
Service Account:      default
Node:                 <none>
Labels:               s3.csi.aws.com/mounted-by-csi-driver-version=2.2.0
                      s3.csi.aws.com/mountpoint-version=1.21.0
Annotations:          s3.csi.aws.com/volume-id: ws47d38456-d132-11f0-9f88-333856a768da-d2029-s3
                      s3.csi.aws.com/volume-name: ws47d38456-d132-11f0-9f88-333856a768da-d2029-pv
Status:               Pending
IP:
IPs:                  <none>
Containers:
  mountpoint:
    Image:           602401143452.dkr.ecr.ca-central-1.amazonaws.com/eks/aws-s3-csi-driver:v2.2.0
    Port:            <none>
    Host Port:       <none>
    SeccompProfile:  RuntimeDefault
    Command:
      /bin/aws-s3-csi-mounter
    Environment:  <none>
    Mounts:
      /comm from comm (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-vcss8 (ro)
Conditions:
  Type           Status
  PodScheduled   False
Volumes:
  comm:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  10Mi
  kube-api-access-vcss8:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    Optional:                false
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 op=Exists
Events:
  Type     Reason            Age                    From                   Message
  ----     ------            ----                   ----                   -------
  Warning  FailedScheduling  5m37s                  default-scheduler      0/10 nodes are available: 1 Too many pods, 9 node(s) didn't satisfy plugin(s) [NodeAffinity]. no new claims to deallocate, preemption: not eligible due to preemptionPolicy=Never.
  Warning  FailedScheduling  5m21s (x2 over 5m21s)  default-scheduler      0/10 nodes are available: 1 Too many pods, 9 node(s) didn't satisfy plugin(s) [NodeAffinity]. no new claims to deallocate, preemption: not eligible due to preemptionPolicy=Never.
  Warning  FailedScheduling  5m20s (x2 over 5m21s)  default-scheduler      0/11 nodes are available: 1 Too many pods, 10 node(s) didn't satisfy plugin(s) [NodeAffinity]. no new claims to deallocate, preemption: not eligible due to preemptionPolicy=Never.
  Warning  FailedScheduling  4m56s (x2 over 5m2s)   default-scheduler      0/10 nodes are available: 1 Too many pods, 9 node(s) didn't satisfy plugin(s) [NodeAffinity]. no new claims to deallocate, preemption: not eligible due to preemptionPolicy=Never.
  Warning  FailedScheduling  3m51s                  default-scheduler      0/10 nodes are available: 1 Too many pods, 9 node(s) didn't satisfy plugin(s) [NodeAffinity]. no new claims to deallocate, preemption: not eligible due to preemptionPolicy=Never.
  Warning  FailedScheduling  3m51s (x2 over 3m51s)  default-scheduler      0/10 nodes are available: 1 Too many pods, 9 node(s) didn't satisfy plugin(s) [NodeAffinity]. no new claims to deallocate, preemption: not eligible due to preemptionPolicy=Never.
  Warning  FailedScheduling  3m22s (x2 over 3m23s)  default-scheduler      0/9 nodes are available: 1 Too many pods, 8 node(s) didn't satisfy plugin(s) [NodeAffinity]. no new claims to deallocate, preemption: not eligible due to preemptionPolicy=Never.
  Warning  FailedScheduling  3m1s (x2 over 3m2s)    default-scheduler      0/10 nodes are available: 1 Too many pods, 9 node(s) didn't satisfy plugin(s) [NodeAffinity]. no new claims to deallocate, preemption: not eligible due to preemptionPolicy=Never.
  Warning  FailedScheduling  2m20s (x2 over 2m21s)  default-scheduler      0/11 nodes are available: 1 Too many pods, 10 node(s) didn't satisfy plugin(s) [NodeAffinity]. no new claims to deallocate, preemption: not eligible due to preemptionPolicy=Never.
  Warning  FailedScheduling  78s (x2 over 102s)     default-scheduler      0/10 nodes are available: 1 Too many pods, 9 node(s) didn't satisfy plugin(s) [NodeAffinity]. no new claims to deallocate, preemption: not eligible due to preemptionPolicy=Never.
  Warning  FailedScheduling  77s (x2 over 77s)      default-scheduler      0/9 nodes are available: 1 Too many pods, 8 node(s) didn't satisfy plugin(s) [NodeAffinity]. no new claims to deallocate, preemption: not eligible due to preemptionPolicy=Never.
  Warning  FailedScheduling  37s (x2 over 5m37s)    eks-auto-mode/compute  Failed to schedule pod, node selector term with matchFields is not supported

Workaround:

kubectl get pods | grep -v Running
NAME                                                              READY   STATUS              RESTARTS   AGE
ws47d38456-d132-11f0-9f88-333856a768da-d2029-0                    0/1     ContainerCreating   0          23m

kubectl delete pod ws47d38456-d132-11f0-9f88-333856a768da-d2029-0
pod "ws47d38456-d132-11f0-9f88-333856a768da-d2029-0" deleted from default namespace

kubectl get pods | grep -v Running
NAME                                                              READY   STATUS              RESTARTS   AGE
ws47d38456-d132-11f0-9f88-333856a768da-d2029-0                    0/1     ContainerCreating   0          2s

kubectl get pods | grep -v Running
NAME                                                              READY   STATUS    RESTARTS   AGE

[simonfogliato]

The following is how I have been able to reproduce the problem on an EKS cluster using "Auto Mode" with the "Mountpoint for Amazon S3 CSI Driver"...

Helper script:

#!/bin/bash
set -ex
bucket=your-aws-bucket-name-in-ca-central-1
case $1 in
        d)
                kubectl delete statefulsets --all
                kubectl delete pods --all
                kubectl delete pvc --all
                kubectl delete pv --all
                ;;
        a)
                for i in {1..100}; do
                        sed "s/bucket_name/$bucket/g; s/uuid/$(uuidgen | tr -d '-' | head -c 16)/g" ./statefulset.yaml | kubectl apply -f -
                done
                ;;
esac

Main statefulset.yaml:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: s3-uuid-pv
spec:
  capacity:
    storage: 1200Gi # Ignored, required
  accessModes:
    - ReadWriteMany # Supported options: ReadWriteMany / ReadOnlyMany
  storageClassName: "" # Required for static provisioning
  claimRef: # To ensure no other PVCs can claim this PV
    namespace: default # Namespace is required even though it's in "default" namespace.
    name: s3-uuid-pvc # Name of your PVC
  mountOptions:
    - allow-delete
    - region ca-central-1
    - prefix s3-uuid/
  csi:
    driver: s3.csi.aws.com # Required
    volumeHandle: s3-uuid-csi # Must be unique
    volumeAttributes:
      bucketName: bucket_name
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: s3-uuid-pvc
spec:
  accessModes:
    - ReadWriteMany # Supported options: ReadWriteMany / ReadOnlyMany
  storageClassName: "" # Required for static provisioning
  resources:
    requests:
      storage: 1200Gi # Ignored, required
  volumeName: s3-uuid-pv # Name of your PV
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    app.kubernetes.io/name: s3-uuid
  name: s3-uuid
spec:
  replicas: 1
  selector:
    matchLabels:
      app: s3-uuid
  serviceName: s-s3-uuid
  template:
    metadata:
      labels:
        app: s3-uuid
    spec:
      containers:
      - name: app-uuid-s3
        image: ubuntu
        command: ["/bin/sh"]
        args: ["-c", "echo 'Hello from the container!' >> /data/$(date -u).txt; tail -f /dev/null"]
        resources:
          limits:
            ephemeral-storage: 1Gi
            memory: 128Mi
          requests:
            ephemeral-storage: 256Mi
            cpu: 32m
            memory: 128Mi
        volumeMounts:
        - name: persistent-storage
          mountPath: /data
      volumes:
      - name: persistent-storage
        persistentVolumeClaim:
          claimName: s3-uuid-pvc

Result:

kubectl get pods | grep -v Running
NAME                    READY   STATUS              RESTARTS   AGE
s3-14c567049ff549b5-0   0/1     ContainerCreating   0          14m
s3-aaf8a5160c96420d-0   0/1     ContainerCreating   0          15m
s3-fb1d83f4d864423e-0   0/1     ContainerCreating   0          14m

[simonfogliato]

The issue appears to stem from a timing/sequencing problem during rapid scale-up rather than Karpenter itself. Mountpoint pods are being created with node-specific scheduling constraints for nodes that do not yet exist, causing the kube-scheduler to fail placement and Karpenter to subsequently reject the pods. This makes Karpenter a symptom, not the root cause. The behavior is consistent with known scheduling challenges in the Mountpoint CSI v2 driver during scale events, as discussed in #543.

[youwalther65]

@simonfogliato The "Too many pods" message make me think, if it could be related to node already reached maxPods in your particular case? Have you checked this?

[simonfogliato]

I did notice the “Too many pods” line and in my failing case one node was at its pod capacity (hence the 1 Too many pods), so that may be part of what the scheduler is reporting.

That said, the mountpoint pod events also include:
eks-auto-mode/compute Failed to schedule pod, node selector term with matchFields is not supported

So I’m not fully confident maxPods is the only/primary cause here, it looks like a combination of capacity/placement constraints and an Auto Mode scheduling limitation (and possibly a timing issue during scale-up).

[youwalther65]

@simonfogliato From my experience I can say, that matchFields is not supported event is "just" an after-effect of kube-scheduler not able to place this pod on a node (due to missing resources, in this case maxPods exhausted) which leads to Auto Mode Karpenter controller to kick in based on the pending pod signal in and try to provision an additional node, which doesn't make sense in this case because MP S3 pod has to be placed on an already provisioned node side by side to application workload pod.

I would suggest to open an AWS support case for this if the issue still persists.

[simonfogliato]

Thanks for the info! I do have an active AWS support case but I'm always open to suggestions. I wonder if I should open a Kubernetes scheduler or a Karpenter github issue in place of this one?

[youwalther65]

If my assumption is correct, then everything works as expected and it makes no sense to open an upstream GitHub issue in KSH or Karpenter repo.
You can verify this with control plane Audit and kube-scheduler logs or you can reach out to me on K8s Slack directly and provide the AWS support case number, so I can check internally with the support folks after the holiday season.

[youwalther65]

@simonfogliato I can confirm that I was able to reproduce the error. Will start debugging it now.

[youwalther65]

Here is what I found, which is inline with my assumption above

Using the script above, lots of workload pods are created by statefulsets and bin-packed to EKS Auto Mode nodes, which uses a Karpenter controller under the hood.
Due to the setup used above a dedicated mountpoint S3 pod is required for each workload pod and because those mountpoint pods are created later , they can't fit on the node anymore because it already runs the maximum number of pods. In this case 27 (c6a.large) leaving the mountpod pods pending (because they have to fit on same node, but maximum number of pods is already reached).

# workload pod created by statefulset
$ kubectl describe po s3-f7a3d7fdb3c64132-0
Name:             s3-f7a3d7fdb3c64132-0
Namespace:        default
...
  Normal   Nominated         2m29s                  eks-auto-mode/compute  Pod should schedule on: nodeclaim/general-purpose-m9gf7
...
  Warning  FailedMount       41s (x2 over 88s)      kubelet                MountVolume.SetUp failed for volume "s3-f7a3d7fdb3c64132-pv" : rpc error: code = Internal desc = Could not mount "nma-logs" at "/var/lib/kubelet/pods/2126619a-5b9d-41bd-944e-8e4eb1ea03d1/volumes/kubernetes.io~csi/s3-f7a3d7fdb3c64132-pv/mount": Failed to wait for Mountpoint Pod "mp-qdsdw" to be ready: mppod/watcher: mountpoint pod not found. Seems like Mountpoint Pod is not in 'Running' status. You can see it's status and any potential failures by running: `kubectl describe pods -n mount-s3 mp-qdsdw`
c889f3ee30f1:tmp waltju$ kubectl describe pods -n mount-s3 mp-qdsdw

$ kubectl get po s3-f7a3d7fdb3c64132-0 -o wide
NAME                    READY   STATUS              RESTARTS   AGE   IP       NODE                  NOMINATED NODE   READINESS GATES
s3-f7a3d7fdb3c64132-0   0/1     ContainerCreating   0          12m   <none>   i-0c02b560276319df3   <none>           <none>

# mountpoint S3 pod, which is in pending because not nominated to any node
$ kubectl get pod -n mount-s3 mp-qdsdw -o wide
NAME       READY   STATUS    RESTARTS   AGE   IP       NODE     NOMINATED NODE   READINESS GATES
mp-qdsdw   0/1     Pending   0          19m   <none>   <none>   <none>           <none>

$ kubectl describe pods -n mount-s3 mp-qdsdw
Name:                 mp-qdsdw
Namespace:            mount-s3
...
Events:
  Type     Reason            Age                    From                   Message
  ----     ------            ----                   ----                   -------
  Warning  FailedScheduling  2m39s                  default-scheduler      0/4 nodes are available: 1 Too many pods, 3 node(s) didn't satisfy plugin(s) [NodeAffinity]. no new claims to deallocate, preemption: not eligible due to preemptionPolicy=Never.
  Warning  FailedScheduling  2m31s                  default-scheduler      0/5 nodes are available: 1 Too many pods, 4 node(s) didn't satisfy plugin(s) [NodeAffinity]. no new claims to deallocate, preemption: not eligible due to preemptionPolicy=Never.
  Warning  FailedScheduling  2m30s (x2 over 2m31s)  default-scheduler      0/5 nodes are available: 1 Too many pods, 4 node(s) didn't satisfy plugin(s) [NodeAffinity]. no new claims to deallocate, preemption: not eligible due to preemptionPolicy=Never.
  Warning  FailedScheduling  2m10s (x2 over 2m11s)  default-scheduler      0/6 nodes are available: 1 Too many pods, 5 node(s) didn't satisfy plugin(s) [NodeAffinity]. no new claims to deallocate, preemption: not eligible due to preemptionPolicy=Never.
  Warning  FailedScheduling  114s (x2 over 114s)    default-scheduler      0/7 nodes are available: 1 Too many pods, 6 node(s) didn't satisfy plugin(s) [NodeAffinity]. no new claims to deallocate, preemption: not eligible due to preemptionPolicy=Never.
  Warning  FailedScheduling  94s                    default-scheduler      0/8 nodes are available: 1 Too many pods, 7 node(s) didn't satisfy plugin(s) [NodeAffinity]. no new claims to deallocate, preemption: not eligible due to preemptionPolicy=Never.
  Warning  FailedScheduling  2m34s                  eks-auto-mode/compute  Failed to schedule pod, node selector term with matchFields is not supported

# node on which workload pod is running advertises a capacity of 27 pods
$ kubectl get no i-0c02b560276319df3 -o yaml 
apiVersion: v1
kind: Node
...
    node.kubernetes.io/instance-type: c6a.large
...
  allocatable:
    cpu: 1780m
    ephemeral-storage: "76173383962"
    hugepages-1Gi: "0"
    hugepages-2Mi: "0"
    memory: 3136652Ki
    pods: "27"
  capacity:
    cpu: "2"
    ephemeral-storage: 81854Mi
    hugepages-1Gi: "0"
    hugepages-2Mi: "0"
    memory: 3881100Ki
    pods: "27"
...

# there are 27 pods running on the node , leaving no place for additional mountpoint S3 pods
$ kubectl describe no i-0c02b560276319df3
Name:               i-0c02b560276319df3
...
Non-terminated Pods:          (27 in total)
  Namespace                   Name                              CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                   ----                              ------------  ----------  ---------------  -------------  ---
  amazon-cloudwatch           cloudwatch-agent-r8b9q            250m (14%)    500m (28%)  128Mi (4%)       512Mi (16%)    17m
  amazon-cloudwatch           fluent-bit-5rwbd                  50m (2%)      500m (28%)  25Mi (0%)        250Mi (8%)     17m
  default                     s3-0d8774b7b4aa4299-0             32m (1%)      0 (0%)      128Mi (4%)       128Mi (4%)     17m
...

# pod status on node
$ kubectl get pod -A -o wide | grep i-0c02b560276319df3
amazon-cloudwatch             cloudwatch-agent-r8b9q                                            1/1     Running             2 (30m ago)   30m    192.168.104.21    i-0c02b560276319df3   <none>           <none>
amazon-cloudwatch             fluent-bit-5rwbd                                                  1/1     Running             0             30m    192.168.104.21    i-0c02b560276319df3   <none>           <none>
default                       s3-0d8774b7b4aa4299-0                                             0/1     ContainerCreating   0             30m    <none>            i-0c02b560276319df3   <none>           <none>
default                       s3-148435a59ff04062-0                                             0/1     ContainerCreating   0             30m    <none>            i-0c02b560276319df3   <none>           <none>
default                       s3-2aa720ec5f994f15-0                                             0/1     ContainerCreating   0             30m    <none>            i-0c02b560276319df3   <none>           <none>
default                       s3-36081d57c2ee4297-0                                             0/1     ContainerCreating   0             30m    <none>            i-0c02b560276319df3   <none>           <none>
default                       s3-42b80187b03643a9-0                                             0/1     ContainerCreating   0             30m    <none>            i-0c02b560276319df3   <none>           <none>
default                       s3-443e1952e9274b58-0                                             0/1     ContainerCreating   0             30m    <none>            i-0c02b560276319df3   <none>           <none>
default                       s3-495957bd3f4e4ffe-0                                             0/1     ContainerCreating   0             30m    <none>            i-0c02b560276319df3   <none>           <none>
default                       s3-4bbbc21672994bc9-0                                             1/1     Running             0             30m    192.168.98.115    i-0c02b560276319df3   <none>           <none>
default                       s3-4dce54f6f7a84ca1-0                                             0/1     ContainerCreating   0             30m    <none>            i-0c02b560276319df3   <none>           <none>
default                       s3-561dcbbe41a94d95-0                                             0/1     ContainerCreating   0             30m    <none>            i-0c02b560276319df3   <none>           <none>
default                       s3-56f53605b0284cba-0                                             0/1     ContainerCreating   0             30m    <none>            i-0c02b560276319df3   <none>           <none>
default                       s3-5a35ec0f99324f35-0                                             0/1     ContainerCreating   0             30m    <none>            i-0c02b560276319df3   <none>           <none>
default                       s3-78df7a44797f4156-0                                             1/1     Running             0             30m    192.168.98.116    i-0c02b560276319df3   <none>           <none>
default                       s3-8877bdde201743f3-0                                             0/1     ContainerCreating   0             30m    <none>            i-0c02b560276319df3   <none>           <none>
default                       s3-b643ab7a995e405f-0                                             0/1     ContainerCreating   0             30m    <none>            i-0c02b560276319df3   <none>           <none>
default                       s3-c6f71a3ea26046a6-0                                             0/1     ContainerCreating   0             30m    <none>            i-0c02b560276319df3   <none>           <none>
default                       s3-ce9f408cc51e4ebc-0                                             0/1     ContainerCreating   0             30m    <none>            i-0c02b560276319df3   <none>           <none>
default                       s3-d5e956e6ec214ac2-0                                             0/1     ContainerCreating   0             30m    <none>            i-0c02b560276319df3   <none>           <none>
default                       s3-db7e28f0c5eb440b-0                                             0/1     ContainerCreating   0             30m    <none>            i-0c02b560276319df3   <none>           <none>
default                       s3-dc29a96da1444891-0                                             0/1     ContainerCreating   0             30m    <none>            i-0c02b560276319df3   <none>           <none>
default                       s3-f7a3d7fdb3c64132-0                                             0/1     ContainerCreating   0             30m    <none>            i-0c02b560276319df3   <none>           <none>
kube-system                   s3-csi-node-zk5kr                                                 3/3     Running             0             30m    192.168.98.113    i-0c02b560276319df3   <none>           <none>
mount-s3                      mp-rqt89                                                          1/1     Running             0             30m    192.168.98.112    i-0c02b560276319df3   <none>           <none>
mount-s3                      mp-tqzdb                                                          1/1     Running             0             30m    192.168.98.114    i-0c02b560276319df3   <none>           <none>
prometheus-node-exporter      prometheus-node-exporter-p7jqf                                    1/1     Running             0             30m    192.168.104.21    i-0c02b560276319df3   <none>           <none>

$ kubectl get pod -A -o wide | grep -c i-0c02b560276319df3
27

A solution would be to use shared mountpoint S3 pods for all workload pods on a node, so the likelihood of pending mountpoint pod could be minimised.