Skip to content

S3CSI-213: Fix mounter pod FSGroup for workload pods with fsGroup#328

Draft
anurag4DSB wants to merge 8 commits intomainfrom
bugfix/S3CSI-213-fsgroup-mounter-pods
Draft

S3CSI-213: Fix mounter pod FSGroup for workload pods with fsGroup#328
anurag4DSB wants to merge 8 commits intomainfrom
bugfix/S3CSI-213-fsgroup-mounter-pods

Conversation

@anurag4DSB
Copy link
Copy Markdown
Collaborator

@anurag4DSB anurag4DSB commented Feb 16, 2026 •
edited
Loading

Summary

  • Fix volume mount failure when workload pods specify fsGroup in their securityContext. The mounter pod's communication socket (/comm/mount.sock) timed out because the emptyDir volume lacked proper group ownership. The fix adds FSGroup to the mounter pod's PodSecurityContext.
  • Add E2E tests verifying mounter pod security context and the customer-reported fsGroup scenario.
  • Bump version to v2.1.1 with release notes and documentation updates.

Test plan

  • Unit tests pass (make unit-test)
  • Linter passes (make lint)
  • E2E tests compile (go build ./... in tests/e2e/)
  • E2E test: mounter pod FSGroup is set correctly in PodSecurityContext
  • E2E test: workload pod with fsGroup can mount and write/read data
  • Verify Helm chart installs with updated version references

@anurag4DSB
S3CSI-213: Add FSGroup to mounter pod SecurityContext
7013bf9 
Set FSGroup in the mounter pod's PodSecurityContext to the
MountpointPodUserID (1000 on vanilla K8s). Without this, the
emptyDir communication directory (/comm/) does not get the
correct group ownership, causing the non-root mount-s3 process
to fail reading/writing the unix socket when the workload pod
specifies its own fsGroup.

Issue: S3CSI-213
@anurag4DSB
S3CSI-213: Add E2E tests for mounter pod FSGroup
4d5ef26 
Add a new mounterpod test suite with two tests:
1. Verifies the mounter pod's PodSecurityContext.FSGroup,
   RunAsUser, and RunAsNonRoot are set correctly.
2. Reproduces the customer scenario where a workload pod with
   fsGroup set in its SecurityContext failed to mount S3 volumes,
   and verifies that the mount succeeds and data can be read/written.

Issue: S3CSI-213
@anurag4DSB
S3CSI-213: Release version v2.1.1
19e7b1e 
Update code version and documentation references from 2.1.0
to 2.1.1. Add release notes for v2.1.1.

Issue: S3CSI-213
@anurag4DSB anurag4DSB mentioned this pull request Feb 16, 2026
Closed
4 tasks
@codecov
Copy link
Copy Markdown

codecov bot commented Feb 16, 2026 •
edited
Loading

❌ 6 Tests Failed:

Tests completed Failed Passed Skipped
135 6 129 58
View the top 3 failed test(s) by shortest run time 
Scality CSI Driver for S3 E2E Suite::[It] [sig-storage] CSI Volumes [Driver: s3.csi.scality.com] [Testpattern: Pre-provisioned PV (default fs)] mounterpod should mount volume when workload pod has fsGroup set [sig-storage]
Stack Traces | 302s run time 
[FAILED] workload pod with fsGroup should reach Running state — mount must not time out
Unexpected error:
    <*fmt.wrapError | 0xc000e473c0>: 
    pod "fsgroup-pod-10f266ce" not Running: Timed out after 300.001s.
    Expected Pod to be in <v1.PodPhase>: "Running"
    Got instead:
        <*v1.Pod | 0xc000d11208>: 
            metadata:
              creationTimestamp: "2026-02-16T10:40:28Z"
              generateName: pvc-tester-
              managedFields:
              - apiVersion: v1
                fieldsType: FieldsV1
                fieldsV1:
                  f:metadata:
                    f:generateName: {}
                  f:spec:
                    f:containers:
                      k:{"name":"write-pod"}:
                        .: {}
                        f:command: {}
                        f:image: {}
                        f:imagePullPolicy: {}
                        f:name: {}
                        f:resources: {}
                        f:securityContext:
                          .: {}
                          f:allowPrivilegeEscalation: {}
                          f:capabilities:
                            .: {}
                            f:drop: {}
                          f:runAsGroup: {}
                          f:runAsNonRoot: {}
                          f:runAsUser: {}
                        f:terminationMessagePath: {}
                        f:terminationMessagePolicy: {}
                        f:volumeMounts:
                          .: {}
                          k:{"mountPath":"/mnt/volume1"}:
                            .: {}
                            f:mountPath: {}
                            f:name: {}
                    f:dnsPolicy: {}
                    f:enableServiceLinks: {}
                    f:restartPolicy: {}
                    f:schedulerName: {}
                    f:securityContext:
                      .: {}
                      f:fsGroup: {}
                      f:runAsGroup: {}
                      f:runAsNonRoot: {}
                      f:runAsUser: {}
                      f:seccompProfile:
                        .: {}
                        f:type: {}
                    f:terminationGracePeriodSeconds: {}
                    f:volumes:
                      .: {}
                      k:{"name":"volume1"}:
                        .: {}
                        f:name: {}
                        f:persistentVolumeClaim:
                          .: {}
                          f:claimName: {}
                manager: e2e.test
                operation: Update
                time: "2026-02-16T10:40:28Z"
              - apiVersion: v1
                fieldsType: FieldsV1
                fieldsV1:
                  f:status:
                    f:conditions:
                      k:{"type":"ContainersReady"}:
                        .: {}
                        f:lastProbeTime: {}
                        f:lastTransitionTime: {}
                        f:message: {}
                        f:reason: {}
                        f:status: {}
                        f:type: {}
                      k:{"type":"Initialized"}:
                        .: {}
                        f:lastProbeTime: {}
                        f:lastTransitionTime: {}
                        f:status: {}
                        f:type: {}
                      k:{"type":"PodReadyToStartContainers"}:
                        .: {}
                        f:lastProbeTime: {}
                        f:lastTransitionTime: {}
                        f:status: {}
                        f:type: {}
                      k:{"type":"Ready"}:
                        .: {}
                        f:lastProbeTime: {}
                        f:lastTransitionTime: {}
                        f:message: {}
                        f:reason: {}
                        f:status: {}
                        f:type: {}
                    f:containerStatuses: {}
                    f:hostIP: {}
                    f:hostIPs: {}
                    f:startTime: {}
                manager: kubelet
                operation: Update
                subresource: status
                time: "2026-02-16T10:40:28Z"
              name: fsgroup-pod-10f266ce
              namespace: mounterpod-8206
              resourceVersion: "5514"
              uid: 2f2799af-3d17-44ca-b95d-4a619c682d85
            spec:
              containers:
              - command:
                - /bin/sh
                - -c
                - trap exit TERM; while true; do sleep 1; done
                image: registry.k8s.io/e2e-test-images/busybox:1.36.1-1
                imagePullPolicy: IfNotPresent
                name: write-pod
                resources: {}
                securityContext:
                  allowPrivilegeEscalation: false
                  capabilities:
                    drop:
                    - ALL
                  runAsGroup: 2000
                  runAsNonRoot: true
                  runAsUser: 1001
                terminationMessagePath: /dev/termination-log
                terminationMessagePolicy: File
                volumeMounts:
                - mountPath: /mnt/volume1
                  name: volume1
                - mountPath: .../run/secrets/kubernetes.io/serviceaccount
                  name: kube-api-access-gc2w5
                  readOnly: true
              dnsPolicy: ClusterFirst
              enableServiceLinks: true
              nodeName: helm-test-cluster-control-plane
              preemptionPolicy: PreemptLowerPriority
              priority: 0
              restartPolicy: OnFailure
              schedulerName: default-scheduler
              securityContext:
                fsGroup: 1001
                runAsGroup: 2000
                runAsNonRoot: true
                runAsUser: 1001
                seccompProfile:
                  type: RuntimeDefault
              serviceAccount: default
              serviceAccountName: default
              terminationGracePeriodSeconds: 30
              tolerations:
              - effect: NoExecute
                key: node.kubernetes.io/not-ready
                operator: Exists
                tolerationSeconds: 300
              - effect: NoExecute
                key: node.kubernetes.io/unreachable
                operator: Exists
                tolerationSeconds: 300
              volumes:
              - name: volume1
                persistentVolumeClaim:
                  claimName: s3-e2e-pvc-31ba228e-8ad6-4b45-b87f-d34144cd76d0
              - name: kube-api-access-gc2w5
                projected:
                  defaultMode: 420
                  sources:
                  - serviceAccountToken:
                      expirationSeconds: 3607
                      path: token
                  - configMap:
                      items:
                      - key: ca.crt
                        path: ca.crt
                      name: kube-root-ca.crt
                  - downwardAPI:
                      items:
                      - fieldRef:
                          apiVersion: v1
                          fieldPath: metadata.namespace
                        path: namespace
            status:
              conditions:
              - lastProbeTime: null
                lastTransitionTime: "2026-02-16T10:40:28Z"
                status: "False"
                type: PodReadyToStartContainers
              - lastProbeTime: null
                lastTransitionTime: "2026-02-16T10:40:28Z"
                status: "True"
                type: Initialized
              - lastProbeTime: null
                lastTransitionTime: "2026-02-16T10:40:28Z"
                message: 'containers with unready status: [write-pod]'
                reason: ContainersNotReady
                status: "False"
                type: Ready
              - lastProbeTime: null
                lastTransitionTime: "2026-02-16T10:40:28Z"
                message: 'containers with unready status: [write-pod]'
                reason: ContainersNotReady
                status: "False"
                type: ContainersReady
              - lastProbeTime: null
                lastTransitionTime: "2026-02-16T10:40:28Z"
                status: "True"
                type: PodScheduled
              containerStatuses:
              - image: registry.k8s.io/e2e-test-images/busybox:1.36.1-1
                imageID: ""
                lastState: {}
                name: write-pod
                ready: false
                restartCount: 0
                started: false
                state:
                  waiting:
                    reason: ContainerCreating
              hostIP: 172.18.0.2
              hostIPs:
              - ip: 172.18.0.2
              phase: Pending
              qosClass: BestEffort
              startTime: "2026-02-16T10:40:28Z"
    {
        msg: "pod \"fsgroup-pod-10f266ce\" not Running: Timed out after 300.001s.\nExpected Pod to be in <v1.PodPhase>: \"Running\"\nGot instead:\n    <*v1.Pod | 0xc000d11208>: \n        metadata:\n          creationTimestamp: \"2026-02-16T10:40:28Z\"\n          generateName: pvc-tester-\n          managedFields:\n          - apiVersion: v1\n            fieldsType: FieldsV1\n            fieldsV1:\n              f:metadata:\n                f:generateName: {}\n              f:spec:\n                f:containers:\n                  k:{\"name\":\"write-pod\"}:\n                    .: {}\n                    f:command: {}\n                    f:image: {}\n                    f:imagePullPolicy: {}\n                    f:name: {}\n                    f:resources: {}\n                    f:securityContext:\n                      .: {}\n                      f:allowPrivilegeEscalation: {}\n                      f:capabilities:\n                        .: {}\n                        f:drop: {}\n                      f:runAsGroup: {}\n                      f:runAsNonRoot: {}\n                      f:runAsUser: {}\n                    f:terminationMessagePath: {}\n                    f:terminationMessagePolicy: {}\n                    f:volumeMounts:\n                      .: {}\n                      k:{\"mountPath\":\"/mnt/volume1\"}:\n                        .: {}\n                        f:mountPath: {}\n                        f:name: {}\n                f:dnsPolicy: {}\n                f:enableServiceLinks: {}\n                f:restartPolicy: {}\n                f:schedulerName: {}\n                f:securityContext:\n                  .: {}\n                  f:fsGroup: {}\n                  f:runAsGroup: {}\n                  f:runAsNonRoot: {}\n                  f:runAsUser: {}\n                  f:seccompProfile:\n                    .: {}\n                    f:type: {}\n                f:terminationGracePeriodSeconds: {}\n                f:volumes:\n                  .: {}\n                  k:{\"name\":\"volume1\"}:\n                    .: {}\n                    f:name: {}\n                    f:persistentVolumeClaim:\n                      .: {}\n                      f:claimName: {}\n            manager: e2e.test\n            operation: Update\n            time: \"2026-02-16T10:40:28Z\"\n          - apiVersion: v1\n            fieldsType: FieldsV1\n            fieldsV1:\n              f:status:\n                f:conditions:\n                  k:{\"type\":\"ContainersReady\"}:\n                    .: {}\n                    f:lastProbeTime: {}\n                    f:lastTransitionTime: {}\n                    f:message: {}\n                    f:reason: {}\n                    f:status: {}\n                    f:type: {}\n                  k:{\"type\":\"Initialized\"}:\n                    .: {}\n                    f:lastProbeTime: {}\n                    f:lastTransitionTime: {}\n                    f:status: {}\n                    f:type: {}\n                  k:{\"type\":\"PodReadyToStartContainers\"}:\n                    .: {}\n                    f:lastProbeTime: {}\n                    f:lastTransitionTime: {}\n                    f:status: {}\n                    f:type: {}\n                  k:{\"type\":\"Ready\"}:\n                    .: {}\n                    f:lastProbeTime: {}\n                    f:lastTransitionTime: {}\n                    f:message: {}\n                    f:reason: {}\n                    f:status: {}\n                    f:type: {}\n                f:containerStatuses: {}\n                f:hostIP: {}\n                f:hostIPs: {}\n                f:startTime: {}\n            manager: kubelet\n            operation: Update\n            subresource: status\n            time: \"2026-02-16T10:40:28Z\"\n          name: fsgroup-pod-10f266ce\n          namespace: mounterpod-8206\n          resourceVersion: \"5514\"\n          uid: 2f2799af-3d17-44ca-b95d-4a619c682d85\n        spec:\n          containers:\n          - command:\n            - /bin/sh\n            - -c\n            - trap exit TERM; while true; do sleep 1; done\n            image: registry.k8s.io/e2e-test-images/busybox:1.36.1-1\n            imagePullPolicy: IfNotPresent\n            name: write-pod\n            resources: {}\n            securityContext:\n              allowPrivilegeEscalation: false\n              capabilities:\n                drop:\n                - ALL\n              runAsGroup: 2000\n              runAsNonRoot: true\n              runAsUser: 1001\n            terminationMessagePath: /dev/termination-log\n            terminationMessagePolicy: File\n            volumeMounts:\n            - mountPath: /mnt/volume1\n              name: volume1\n            - mountPath: .../run/secrets/kubernetes.io/serviceaccount\n              name: kube-api-access-gc2w5\n              readOnly: true\n          dnsPolicy: ClusterFirst\n          enableServiceLinks: true\n          nodeName: helm-test-cluster-control-plane\n          preemptionPolicy: PreemptLowerPriority\n          priority: 0\n          restartPolicy: OnFailure\n          schedulerName: default-scheduler\n          securityContext:\n            fsGroup: 1001\n            runAsGroup: 2000\n            runAsNonRoot: true\n            runAsUser: 1001\n            seccompProfile:\n              type: RuntimeDefault\n          serviceAccount: default\n          serviceAccountName: default\n          terminationGracePeriodSeconds: 30\n          tolerations:\n          - effect: NoExecute\n            key: node.kubernetes.io/not-ready\n            operator: Exists\n            tolerationSeconds: 300\n          - effect: NoExecute\n            key: node.kubernetes.io/unreachable\n            operator: Exists\n            tolerationSeconds: 300\n          volumes:\n          - name: volume1\n            persistentVolumeClaim:\n              claimName: s3-e2e-pvc-31ba228e-8ad6-4b45-b87f-d34144cd76d0\n          - name: kube-api-access-gc2w5\n            projected:\n              defaultMode: 420\n              sources:\n              - serviceAccountToken:\n                  expirationSeconds: 3607\n                  path: token\n              - configMap:\n                  items:\n                  - key: ca.crt\n                    path: ca.crt\n                  name: kube-root-ca.crt\n              - downwardAPI:\n                  items:\n                  - fieldRef:\n                      apiVersion: v1\n                      fieldPath: metadata.namespace\n                    path: namespace\n        status:\n          conditions:\n          - lastProbeTime: null\n            lastTransitionTime: \"2026-02-16T10:40:28Z\"\n            status: \"False\"\n            type: PodReadyToStartContainers\n          - lastProbeTime: null\n            lastTransitionTime: \"2026-02-16T10:40:28Z\"\n            status: \"True\"\n            type: Initialized\n          - lastProbeTime: null\n            lastTransitionTime: \"2026-02-16T10:40:28Z\"\n            message: 'containers with unready status: [write-pod]'\n            reason: ContainersNotReady\n            status: \"False\"\n            type: Ready\n          - lastProbeTime: null\n            lastTransitionTime: \"2026-02-16T10:40:28Z\"\n            message: 'containers with unready status: [write-pod]'\n            reason: ContainersNotReady\n            status: \"False\"\n            type: ContainersReady\n          - lastProbeTime: null\n            lastTransitionTime: \"2026-02-16T10:40:28Z\"\n            status: \"True\"\n            type: PodScheduled\n          containerStatuses:\n          - image: registry.k8s.io/e2e-test-images/busybox:1.36.1-1\n            imageID: \"\"\n            lastState: {}\n            name: write-pod\n            ready: false\n            restartCount: 0\n            started: false\n            state:\n              waiting:\n                reason: ContainerCreating\n          hostIP: 172.18.0.2\n          hostIPs:\n          - ip: 172.18.0.2\n          phase: Pending\n          qosClass: BestEffort\n          startTime: \"2026-02-16T10:40:28Z\"",
        err: <framework.FailureError>{
            msg: "Timed out after 300.001s.\nExpected Pod to be in <v1.PodPhase>: \"Running\"\nGot instead:\n    <*v1.Pod | 0xc000d11208>: \n        metadata:\n          creationTimestamp: \"2026-02-16T10:40:28Z\"\n          generateName: pvc-tester-\n          managedFields:\n          - apiVersion: v1\n            fieldsType: FieldsV1\n            fieldsV1:\n              f:metadata:\n                f:generateName: {}\n              f:spec:\n                f:containers:\n                  k:{\"name\":\"write-pod\"}:\n                    .: {}\n                    f:command: {}\n                    f:image: {}\n                    f:imagePullPolicy: {}\n                    f:name: {}\n                    f:resources: {}\n                    f:securityContext:\n                      .: {}\n                      f:allowPrivilegeEscalation: {}\n                      f:capabilities:\n                        .: {}\n                        f:drop: {}\n                      f:runAsGroup: {}\n                      f:runAsNonRoot: {}\n                      f:runAsUser: {}\n                    f:terminationMessagePath: {}\n                    f:terminationMessagePolicy: {}\n                    f:volumeMounts:\n                      .: {}\n                      k:{\"mountPath\":\"/mnt/volume1\"}:\n                        .: {}\n                        f:mountPath: {}\n                        f:name: {}\n                f:dnsPolicy: {}\n                f:enableServiceLinks: {}\n                f:restartPolicy: {}\n                f:schedulerName: {}\n                f:securityContext:\n                  .: {}\n                  f:fsGroup: {}\n                  f:runAsGroup: {}\n                  f:runAsNonRoot: {}\n                  f:runAsUser: {}\n                  f:seccompProfile:\n                    .: {}\n                    f:type: {}\n                f:terminationGracePeriodSeconds: {}\n                f:volumes:\n                  .: {}\n                  k:{\"name\":\"volume1\"}:\n                    .: {}\n                    f:name: {}\n                    f:persistentVolumeClaim:\n                      .: {}\n                      f:claimName: {}\n            manager: e2e.test\n            operation: Update\n            time: \"2026-02-16T10:40:28Z\"\n          - apiVersion: v1\n            fieldsType: FieldsV1\n            fieldsV1:\n              f:status:\n                f:conditions:\n                  k:{\"type\":\"ContainersReady\"}:\n                    .: {}\n                    f:lastProbeTime: {}\n                    f:lastTransitionTime: {}\n                    f:message: {}\n                    f:reason: {}\n                    f:status: {}\n                    f:type: {}\n                  k:{\"type\":\"Initialized\"}:\n                    .: {}\n                    f:lastProbeTime: {}\n                    f:lastTransitionTime: {}\n                    f:status: {}\n                    f:type: {}\n                  k:{\"type\":\"PodReadyToStartContainers\"}:\n                    .: {}\n                    f:lastProbeTime: {}\n                    f:lastTransitionTime: {}\n                    f:status: {}\n                    f:type: {}\n                  k:{\"type\":\"Ready\"}:\n                    .: {}\n                    f:lastProbeTime: {}\n                    f:lastTransitionTime: {}\n                    f:message: {}\n                    f:reason: {}\n                    f:status: {}\n                    f:type: {}\n                f:containerStatuses: {}\n                f:hostIP: {}\n                f:hostIPs: {}\n                f:startTime: {}\n            manager: kubelet\n            operation: Update\n            subresource: status\n            time: \"2026-02-16T10:40:28Z\"\n          name: fsgroup-pod-10f266ce\n          namespace: mounterpod-8206\n          resourceVersion: \"5514\"\n          uid: 2f2799af-3d17-44ca-b95d-4a619c682d85\n        spec:\n          containers:\n          - command:\n            - /bin/sh\n            - -c\n            - trap exit TERM; while true; do sleep 1; done\n            image: registry.k8s.io/e2e-test-images/busybox:1.36.1-1\n            imagePullPolicy: IfNotPresent\n            name: write-pod\n            resources: {}\n            securityContext:\n              allowPrivilegeEscalation: false\n              capabilities:\n                drop:\n                - ALL\n              runAsGroup: 2000\n              runAsNonRoot: true\n              runAsUser: 1001\n            terminationMessagePath: /dev/termination-log\n            terminationMessagePolicy: File\n            volumeMounts:\n            - mountPath: /mnt/volume1\n              name: volume1\n            - mountPath: .../run/secrets/kubernetes.io/serviceaccount\n              name: kube-api-access-gc2w5\n              readOnly: true\n          dnsPolicy: ClusterFirst\n          enableServiceLinks: true\n          nodeName: helm-test-cluster-control-plane\n          preemptionPolicy: PreemptLowerPriority\n          priority: 0\n          restartPolicy: OnFailure\n          schedulerName: default-scheduler\n          securityContext:\n            fsGroup: 1001\n            runAsGroup: 2000\n            runAsNonRoot: true\n            runAsUser: 1001\n            seccompProfile:\n              type: RuntimeDefault\n          serviceAccount: default\n          serviceAccountName: default\n          terminationGracePeriodSeconds: 30\n          tolerations:\n          - effect: NoExecute\n            key: node.kubernetes.io/not-ready\n            operator: Exists\n            tolerationSeconds: 300\n          - effect: NoExecute\n            key: node.kubernetes.io/unreachable\n            operator: Exists\n            tolerationSeconds: 300\n          volumes:\n          - name: volume1\n            persistentVolumeClaim:\n              claimName: s3-e2e-pvc-31ba228e-8ad6-4b45-b87f-d34144cd76d0\n          - name: kube-api-access-gc2w5\n            projected:\n              defaultMode: 420\n              sources:\n              - serviceAccountToken:\n                  expirationSeconds: 3607\n                  path: token\n              - configMap:\n                  items:\n                  - key: ca.crt\n                    path: ca.crt\n                  name: kube-root-ca.crt\n              - downwardAPI:\n                  items:\n                  - fieldRef:\n                      apiVersion: v1\n                      fieldPath: metadata.namespace\n                    path: namespace\n        status:\n          conditions:\n          - lastProbeTime: null\n            lastTransitionTime: \"2026-02-16T10:40:28Z\"\n            status: \"False\"\n            type: PodReadyToStartContainers\n          - lastProbeTime: null\n            lastTransitionTime: \"2026-02-16T10:40:28Z\"\n            status: \"True\"\n            type: Initialized\n          - lastProbeTime: null\n            lastTransitionTime: \"2026-02-16T10:40:28Z\"\n            message: 'containers with unready status: [write-pod]'\n            reason: ContainersNotReady\n            status: \"False\"\n            type: Ready\n          - lastProbeTime: null\n            lastTransitionTime: \"2026-02-16T10:40:28Z\"\n            message: 'containers with unready status: [write-pod]'\n            reason: ContainersNotReady\n            status: \"False\"\n            type: ContainersReady\n          - lastProbeTime: null\n            lastTransitionTime: \"2026-02-16T10:40:28Z\"\n            status: \"True\"\n            type: PodScheduled\n          containerStatuses:\n          - image: registry.k8s.io/e2e-test-images/busybox:1.36.1-1\n            imageID: \"\"\n            lastState: {}\n            name: write-pod\n            ready: false\n            restartCount: 0\n            started: false\n            state:\n              waiting:\n                reason: ContainerCreating\n          hostIP: 172.18.0.2\n          hostIPs:\n          - ip: 172.18.0.2\n          phase: Pending\n          qosClass: BestEffort\n          startTime: \"2026-02-16T10:40:28Z\"",
            fullStackTrace: "k8s..../e2e/framework/pod.WaitTimeoutForPodRunningInNamespace({0x7f3cb0d51f28, 0xc000de3b30}, {0x2e74af0?, 0xc0003faea0?}, {0xc000b62840, 0x14}, {0xc000688a00, 0xf}, 0x45d964b800)\n\.../pkg/mod/k8s.io/kubernetes@v1.29..../framework/pod/wait.go:459 +0x529\nk8s..../e2e/framework/pod.WaitForPodNameRunningInNamespace(...)\n\.../pkg/mod/k8s.io/kubernetes@v1.29..../framework/pod/wait.go:443\ngithub..../tests/e2e/customsuites.createPod({0x7f3cb0d51f28, 0xc000de3b30}, {0x2e74af0, 0xc0003faea0}, {0xc000688a00, 0xf}, 0xc0008a1208)\n\.../e2e/customsuites/util.go:218 +0x205\ngithub..../tests/e2e/customsuites.(*s3CSIMounterPodTestSuite).DefineTests.func4({0x7f3cb0d51f28, 0xc000de3b30})\n\.../e2e/customsuites/mounterpod.go:262 +0x378",
        },
    }
occurred
In [It] at: .../e2e/customsuites/mounterpod.go:263 @ 02/16/26 10:45:28.065

Scality CSI Driver for S3 E2E Suite::[It] [sig-storage] CSI Volumes [Driver: s3.csi.scality.com] [Testpattern: Pre-provisioned PV (default fs)] mounterpod should create separate mounter pods for dynamically provisioned volume with different fsGroup values [sig-storage]
Stack Traces | 305s run time 
[FAILED] first workload pod should reach Running state
Unexpected error:
    <*fmt.wrapError | 0xc0000b1840>: 
    pod "fsgroup-dyn-share-1-f562d8df" not Running: Timed out after 300.000s.
    Expected Pod to be in <v1.PodPhase>: "Running"
    Got instead:
        <*v1.Pod | 0xc000ce3208>: 
            metadata:
              creationTimestamp: "2026-02-16T10:40:42Z"
              generateName: pvc-tester-
              managedFields:
              - apiVersion: v1
                fieldsType: FieldsV1
                fieldsV1:
                  f:metadata:
                    f:generateName: {}
                  f:spec:
                    f:containers:
                      k:{"name":"write-pod"}:
                        .: {}
                        f:command: {}
                        f:image: {}
                        f:imagePullPolicy: {}
                        f:name: {}
                        f:resources: {}
                        f:securityContext:
                          .: {}
                          f:allowPrivilegeEscalation: {}
                          f:capabilities:
                            .: {}
                            f:drop: {}
                          f:runAsGroup: {}
                          f:runAsNonRoot: {}
                          f:runAsUser: {}
                        f:terminationMessagePath: {}
                        f:terminationMessagePolicy: {}
                        f:volumeMounts:
                          .: {}
                          k:{"mountPath":"/mnt/volume1"}:
                            .: {}
                            f:mountPath: {}
                            f:name: {}
                    f:dnsPolicy: {}
                    f:enableServiceLinks: {}
                    f:restartPolicy: {}
                    f:schedulerName: {}
                    f:securityContext:
                      .: {}
                      f:fsGroup: {}
                      f:runAsGroup: {}
                      f:runAsNonRoot: {}
                      f:runAsUser: {}
                      f:seccompProfile:
                        .: {}
                        f:type: {}
                    f:terminationGracePeriodSeconds: {}
                    f:volumes:
                      .: {}
                      k:{"name":"volume1"}:
                        .: {}
                        f:name: {}
                        f:persistentVolumeClaim:
                          .: {}
                          f:claimName: {}
                manager: e2e.test
                operation: Update
                time: "2026-02-16T10:40:42Z"
              - apiVersion: v1
                fieldsType: FieldsV1
                fieldsV1:
                  f:status:
                    f:conditions:
                      k:{"type":"ContainersReady"}:
                        .: {}
                        f:lastProbeTime: {}
                        f:lastTransitionTime: {}
                        f:message: {}
                        f:reason: {}
                        f:status: {}
                        f:type: {}
                      k:{"type":"Initialized"}:
                        .: {}
                        f:lastProbeTime: {}
                        f:lastTransitionTime: {}
                        f:status: {}
                        f:type: {}
                      k:{"type":"PodReadyToStartContainers"}:
                        .: {}
                        f:lastProbeTime: {}
                        f:lastTransitionTime: {}
                        f:status: {}
                        f:type: {}
                      k:{"type":"Ready"}:
                        .: {}
                        f:lastProbeTime: {}
                        f:lastTransitionTime: {}
                        f:message: {}
                        f:reason: {}
                        f:status: {}
                        f:type: {}
                    f:containerStatuses: {}
                    f:hostIP: {}
                    f:hostIPs: {}
                    f:startTime: {}
                manager: kubelet
                operation: Update
                subresource: status
                time: "2026-02-16T10:40:42Z"
              name: fsgroup-dyn-share-1-f562d8df
              namespace: mounterpod-3034
              resourceVersion: "6000"
              uid: 71d6474f-c7cd-4580-b92d-5d1bf396fb47
            spec:
              containers:
              - command:
                - /bin/sh
                - -c
                - trap exit TERM; while true; do sleep 1; done
                image: registry.k8s.io/e2e-test-images/busybox:1.36.1-1
                imagePullPolicy: IfNotPresent
                name: write-pod
                resources: {}
                securityContext:
                  allowPrivilegeEscalation: false
                  capabilities:
                    drop:
                    - ALL
                  runAsGroup: 2000
                  runAsNonRoot: true
                  runAsUser: 1001
                terminationMessagePath: /dev/termination-log
                terminationMessagePolicy: File
                volumeMounts:
                - mountPath: /mnt/volume1
                  name: volume1
                - mountPath: .../run/secrets/kubernetes.io/serviceaccount
                  name: kube-api-access-fl8vd
                  readOnly: true
              dnsPolicy: ClusterFirst
              enableServiceLinks: true
              nodeName: helm-test-cluster-control-plane
              preemptionPolicy: PreemptLowerPriority
              priority: 0
              restartPolicy: OnFailure
              schedulerName: default-scheduler
              securityContext:
                fsGroup: 1001
                runAsGroup: 2000
                runAsNonRoot: true
                runAsUser: 1001
                seccompProfile:
                  type: RuntimeDefault
              serviceAccount: default
              serviceAccountName: default
              terminationGracePeriodSeconds: 30
              tolerations:
              - effect: NoExecute
                key: node.kubernetes.io/not-ready
                operator: Exists
                tolerationSeconds: 300
              - effect: NoExecute
                key: node.kubernetes.io/unreachable
                operator: Exists
                tolerationSeconds: 300
              volumes:
              - name: volume1
                persistentVolumeClaim:
                  claimName: fsgroup-share-pvc-f9d12b78
              - name: kube-api-access-fl8vd
                projected:
                  defaultMode: 420
                  sources:
                  - serviceAccountToken:
                      expirationSeconds: 3607
                      path: token
                  - configMap:
                      items:
                      - key: ca.crt
                        path: ca.crt
                      name: kube-root-ca.crt
                  - downwardAPI:
                      items:
                      - fieldRef:
                          apiVersion: v1
                          fieldPath: metadata.namespace
                        path: namespace
            status:
              conditions:
              - lastProbeTime: null
                lastTransitionTime: "2026-02-16T10:40:42Z"
                status: "False"
                type: PodReadyToStartContainers
              - lastProbeTime: null
                lastTransitionTime: "2026-02-16T10:40:42Z"
                status: "True"
                type: Initialized
              - lastProbeTime: null
                lastTransitionTime: "2026-02-16T10:40:42Z"
                message: 'containers with unready status: [write-pod]'
                reason: ContainersNotReady
                status: "False"
                type: Ready
              - lastProbeTime: null
                lastTransitionTime: "2026-02-16T10:40:42Z"
                message: 'containers with unready status: [write-pod]'
                reason: ContainersNotReady
                status: "False"
                type: ContainersReady
              - lastProbeTime: null
                lastTransitionTime: "2026-02-16T10:40:42Z"
                status: "True"
                type: PodScheduled
              containerStatuses:
              - image: registry.k8s.io/e2e-test-images/busybox:1.36.1-1
                imageID: ""
                lastState: {}
                name: write-pod
                ready: false
                restartCount: 0
                started: false
                state:
                  waiting:
                    reason: ContainerCreating
              hostIP: 172.18.0.2
              hostIPs:
              - ip: 172.18.0.2
              phase: Pending
              qosClass: BestEffort
              startTime: "2026-02-16T10:40:42Z"
    {
        msg: "pod \"fsgroup-dyn-share-1-f562d8df\" not Running: Timed out after 300.000s.\nExpected Pod to be in <v1.PodPhase>: \"Running\"\nGot instead:\n    <*v1.Pod | 0xc000ce3208>: \n        metadata:\n          creationTimestamp: \"2026-02-16T10:40:42Z\"\n          generateName: pvc-tester-\n          managedFields:\n          - apiVersion: v1\n            fieldsType: FieldsV1\n            fieldsV1:\n              f:metadata:\n                f:generateName: {}\n              f:spec:\n                f:containers:\n                  k:{\"name\":\"write-pod\"}:\n                    .: {}\n                    f:command: {}\n                    f:image: {}\n                    f:imagePullPolicy: {}\n                    f:name: {}\n                    f:resources: {}\n                    f:securityContext:\n                      .: {}\n                      f:allowPrivilegeEscalation: {}\n                      f:capabilities:\n                        .: {}\n                        f:drop: {}\n                      f:runAsGroup: {}\n                      f:runAsNonRoot: {}\n                      f:runAsUser: {}\n                    f:terminationMessagePath: {}\n                    f:terminationMessagePolicy: {}\n                    f:volumeMounts:\n                      .: {}\n                      k:{\"mountPath\":\"/mnt/volume1\"}:\n                        .: {}\n                        f:mountPath: {}\n                        f:name: {}\n                f:dnsPolicy: {}\n                f:enableServiceLinks: {}\n                f:restartPolicy: {}\n                f:schedulerName: {}\n                f:securityContext:\n                  .: {}\n                  f:fsGroup: {}\n                  f:runAsGroup: {}\n                  f:runAsNonRoot: {}\n                  f:runAsUser: {}\n                  f:seccompProfile:\n                    .: {}\n                    f:type: {}\n                f:terminationGracePeriodSeconds: {}\n                f:volumes:\n                  .: {}\n                  k:{\"name\":\"volume1\"}:\n                    .: {}\n                    f:name: {}\n                    f:persistentVolumeClaim:\n                      .: {}\n                      f:claimName: {}\n            manager: e2e.test\n            operation: Update\n            time: \"2026-02-16T10:40:42Z\"\n          - apiVersion: v1\n            fieldsType: FieldsV1\n            fieldsV1:\n              f:status:\n                f:conditions:\n                  k:{\"type\":\"ContainersReady\"}:\n                    .: {}\n                    f:lastProbeTime: {}\n                    f:lastTransitionTime: {}\n                    f:message: {}\n                    f:reason: {}\n                    f:status: {}\n                    f:type: {}\n                  k:{\"type\":\"Initialized\"}:\n                    .: {}\n                    f:lastProbeTime: {}\n                    f:lastTransitionTime: {}\n                    f:status: {}\n                    f:type: {}\n                  k:{\"type\":\"PodReadyToStartContainers\"}:\n                    .: {}\n                    f:lastProbeTime: {}\n                    f:lastTransitionTime: {}\n                    f:status: {}\n                    f:type: {}\n                  k:{\"type\":\"Ready\"}:\n                    .: {}\n                    f:lastProbeTime: {}\n                    f:lastTransitionTime: {}\n                    f:message: {}\n                    f:reason: {}\n                    f:status: {}\n                    f:type: {}\n                f:containerStatuses: {}\n                f:hostIP: {}\n                f:hostIPs: {}\n                f:startTime: {}\n            manager: kubelet\n            operation: Update\n            subresource: status\n            time: \"2026-02-16T10:40:42Z\"\n          name: fsgroup-dyn-share-1-f562d8df\n          namespace: mounterpod-3034\n          resourceVersion: \"6000\"\n          uid: 71d6474f-c7cd-4580-b92d-5d1bf396fb47\n        spec:\n          containers:\n          - command:\n            - /bin/sh\n            - -c\n            - trap exit TERM; while true; do sleep 1; done\n            image: registry.k8s.io/e2e-test-images/busybox:1.36.1-1\n            imagePullPolicy: IfNotPresent\n            name: write-pod\n            resources: {}\n            securityContext:\n              allowPrivilegeEscalation: false\n              capabilities:\n                drop:\n                - ALL\n              runAsGroup: 2000\n              runAsNonRoot: true\n              runAsUser: 1001\n            terminationMessagePath: /dev/termination-log\n            terminationMessagePolicy: File\n            volumeMounts:\n            - mountPath: /mnt/volume1\n              name: volume1\n            - mountPath: .../run/secrets/kubernetes.io/serviceaccount\n              name: kube-api-access-fl8vd\n              readOnly: true\n          dnsPolicy: ClusterFirst\n          enableServiceLinks: true\n          nodeName: helm-test-cluster-control-plane\n          preemptionPolicy: PreemptLowerPriority\n          priority: 0\n          restartPolicy: OnFailure\n          schedulerName: default-scheduler\n          securityContext:\n            fsGroup: 1001\n            runAsGroup: 2000\n            runAsNonRoot: true\n            runAsUser: 1001\n            seccompProfile:\n              type: RuntimeDefault\n          serviceAccount: default\n          serviceAccountName: default\n          terminationGracePeriodSeconds: 30\n          tolerations:\n          - effect: NoExecute\n            key: node.kubernetes.io/not-ready\n            operator: Exists\n            tolerationSeconds: 300\n          - effect: NoExecute\n            key: node.kubernetes.io/unreachable\n            operator: Exists\n            tolerationSeconds: 300\n          volumes:\n          - name: volume1\n            persistentVolumeClaim:\n              claimName: fsgroup-share-pvc-f9d12b78\n          - name: kube-api-access-fl8vd\n            projected:\n              defaultMode: 420\n              sources:\n              - serviceAccountToken:\n                  expirationSeconds: 3607\n                  path: token\n              - configMap:\n                  items:\n                  - key: ca.crt\n                    path: ca.crt\n                  name: kube-root-ca.crt\n              - downwardAPI:\n                  items:\n                  - fieldRef:\n                      apiVersion: v1\n                      fieldPath: metadata.namespace\n                    path: namespace\n        status:\n          conditions:\n          - lastProbeTime: null\n            lastTransitionTime: \"2026-02-16T10:40:42Z\"\n            status: \"False\"\n            type: PodReadyToStartContainers\n          - lastProbeTime: null\n            lastTransitionTime: \"2026-02-16T10:40:42Z\"\n            status: \"True\"\n            type: Initialized\n          - lastProbeTime: null\n            lastTransitionTime: \"2026-02-16T10:40:42Z\"\n            message: 'containers with unready status: [write-pod]'\n            reason: ContainersNotReady\n            status: \"False\"\n            type: Ready\n          - lastProbeTime: null\n            lastTransitionTime: \"2026-02-16T10:40:42Z\"\n            message: 'containers with unready status: [write-pod]'\n            reason: ContainersNotReady\n            status: \"False\"\n            type: ContainersReady\n          - lastProbeTime: null\n            lastTransitionTime: \"2026-02-16T10:40:42Z\"\n            status: \"True\"\n            type: PodScheduled\n          containerStatuses:\n          - image: registry.k8s.io/e2e-test-images/busybox:1.36.1-1\n            imageID: \"\"\n            lastState: {}\n            name: write-pod\n            ready: false\n            restartCount: 0\n            started: false\n            state:\n              waiting:\n                reason: ContainerCreating\n          hostIP: 172.18.0.2\n          hostIPs:\n          - ip: 172.18.0.2\n          phase: Pending\n          qosClass: BestEffort\n          startTime: \"2026-02-16T10:40:42Z\"",
        err: <framework.FailureError>{
            msg: "Timed out after 300.000s.\nExpected Pod to be in <v1.PodPhase>: \"Running\"\nGot instead:\n    <*v1.Pod | 0xc000ce3208>: \n        metadata:\n          creationTimestamp: \"2026-02-16T10:40:42Z\"\n          generateName: pvc-tester-\n          managedFields:\n          - apiVersion: v1\n            fieldsType: FieldsV1\n            fieldsV1:\n              f:metadata:\n                f:generateName: {}\n              f:spec:\n                f:containers:\n                  k:{\"name\":\"write-pod\"}:\n                    .: {}\n                    f:command: {}\n                    f:image: {}\n                    f:imagePullPolicy: {}\n                    f:name: {}\n                    f:resources: {}\n                    f:securityContext:\n                      .: {}\n                      f:allowPrivilegeEscalation: {}\n                      f:capabilities:\n                        .: {}\n                        f:drop: {}\n                      f:runAsGroup: {}\n                      f:runAsNonRoot: {}\n                      f:runAsUser: {}\n                    f:terminationMessagePath: {}\n                    f:terminationMessagePolicy: {}\n                    f:volumeMounts:\n                      .: {}\n                      k:{\"mountPath\":\"/mnt/volume1\"}:\n                        .: {}\n                        f:mountPath: {}\n                        f:name: {}\n                f:dnsPolicy: {}\n                f:enableServiceLinks: {}\n                f:restartPolicy: {}\n                f:schedulerName: {}\n                f:securityContext:\n                  .: {}\n                  f:fsGroup: {}\n                  f:runAsGroup: {}\n                  f:runAsNonRoot: {}\n                  f:runAsUser: {}\n                  f:seccompProfile:\n                    .: {}\n                    f:type: {}\n                f:terminationGracePeriodSeconds: {}\n                f:volumes:\n                  .: {}\n                  k:{\"name\":\"volume1\"}:\n                    .: {}\n                    f:name: {}\n                    f:persistentVolumeClaim:\n                      .: {}\n                      f:claimName: {}\n            manager: e2e.test\n            operation: Update\n            time: \"2026-02-16T10:40:42Z\"\n          - apiVersion: v1\n            fieldsType: FieldsV1\n            fieldsV1:\n              f:status:\n                f:conditions:\n                  k:{\"type\":\"ContainersReady\"}:\n                    .: {}\n                    f:lastProbeTime: {}\n                    f:lastTransitionTime: {}\n                    f:message: {}\n                    f:reason: {}\n                    f:status: {}\n                    f:type: {}\n                  k:{\"type\":\"Initialized\"}:\n                    .: {}\n                    f:lastProbeTime: {}\n                    f:lastTransitionTime: {}\n                    f:status: {}\n                    f:type: {}\n                  k:{\"type\":\"PodReadyToStartContainers\"}:\n                    .: {}\n                    f:lastProbeTime: {}\n                    f:lastTransitionTime: {}\n                    f:status: {}\n                    f:type: {}\n                  k:{\"type\":\"Ready\"}:\n                    .: {}\n                    f:lastProbeTime: {}\n                    f:lastTransitionTime: {}\n                    f:message: {}\n                    f:reason: {}\n                    f:status: {}\n                    f:type: {}\n                f:containerStatuses: {}\n                f:hostIP: {}\n                f:hostIPs: {}\n                f:startTime: {}\n            manager: kubelet\n            operation: Update\n            subresource: status\n            time: \"2026-02-16T10:40:42Z\"\n          name: fsgroup-dyn-share-1-f562d8df\n          namespace: mounterpod-3034\n          resourceVersion: \"6000\"\n          uid: 71d6474f-c7cd-4580-b92d-5d1bf396fb47\n        spec:\n          containers:\n          - command:\n            - /bin/sh\n            - -c\n            - trap exit TERM; while true; do sleep 1; done\n            image: registry.k8s.io/e2e-test-images/busybox:1.36.1-1\n            imagePullPolicy: IfNotPresent\n            name: write-pod\n            resources: {}\n            securityContext:\n              allowPrivilegeEscalation: false\n              capabilities:\n                drop:\n                - ALL\n              runAsGroup: 2000\n              runAsNonRoot: true\n              runAsUser: 1001\n            terminationMessagePath: /dev/termination-log\n            terminationMessagePolicy: File\n            volumeMounts:\n            - mountPath: /mnt/volume1\n              name: volume1\n            - mountPath: .../run/secrets/kubernetes.io/serviceaccount\n              name: kube-api-access-fl8vd\n              readOnly: true\n          dnsPolicy: ClusterFirst\n          enableServiceLinks: true\n          nodeName: helm-test-cluster-control-plane\n          preemptionPolicy: PreemptLowerPriority\n          priority: 0\n          restartPolicy: OnFailure\n          schedulerName: default-scheduler\n          securityContext:\n            fsGroup: 1001\n            runAsGroup: 2000\n            runAsNonRoot: true\n            runAsUser: 1001\n            seccompProfile:\n              type: RuntimeDefault\n          serviceAccount: default\n          serviceAccountName: default\n          terminationGracePeriodSeconds: 30\n          tolerations:\n          - effect: NoExecute\n            key: node.kubernetes.io/not-ready\n            operator: Exists\n            tolerationSeconds: 300\n          - effect: NoExecute\n            key: node.kubernetes.io/unreachable\n            operator: Exists\n            tolerationSeconds: 300\n          volumes:\n          - name: volume1\n            persistentVolumeClaim:\n              claimName: fsgroup-share-pvc-f9d12b78\n          - name: kube-api-access-fl8vd\n            projected:\n              defaultMode: 420\n              sources:\n              - serviceAccountToken:\n                  expirationSeconds: 3607\n                  path: token\n              - configMap:\n                  items:\n                  - key: ca.crt\n                    path: ca.crt\n                  name: kube-root-ca.crt\n              - downwardAPI:\n                  items:\n                  - fieldRef:\n                      apiVersion: v1\n                      fieldPath: metadata.namespace\n                    path: namespace\n        status:\n          conditions:\n          - lastProbeTime: null\n            lastTransitionTime: \"2026-02-16T10:40:42Z\"\n            status: \"False\"\n            type: PodReadyToStartContainers\n          - lastProbeTime: null\n            lastTransitionTime: \"2026-02-16T10:40:42Z\"\n            status: \"True\"\n            type: Initialized\n          - lastProbeTime: null\n            lastTransitionTime: \"2026-02-16T10:40:42Z\"\n            message: 'containers with unready status: [write-pod]'\n            reason: ContainersNotReady\n            status: \"False\"\n            type: Ready\n          - lastProbeTime: null\n            lastTransitionTime: \"2026-02-16T10:40:42Z\"\n            message: 'containers with unready status: [write-pod]'\n            reason: ContainersNotReady\n            status: \"False\"\n            type: ContainersReady\n          - lastProbeTime: null\n            lastTransitionTime: \"2026-02-16T10:40:42Z\"\n            status: \"True\"\n            type: PodScheduled\n          containerStatuses:\n          - image: registry.k8s.io/e2e-test-images/busybox:1.36.1-1\n            imageID: \"\"\n            lastState: {}\n            name: write-pod\n            ready: false\n            restartCount: 0\n            started: false\n            state:\n              waiting:\n                reason: ContainerCreating\n          hostIP: 172.18.0.2\n          hostIPs:\n          - ip: 172.18.0.2\n          phase: Pending\n          qosClass: BestEffort\n          startTime: \"2026-02-16T10:40:42Z\"",
            fullStackTrace: "k8s..../e2e/framework/pod.WaitTimeoutForPodRunningInNamespace({0x7fb35c4dbf40, 0xc0012cf140}, {0x2e74af0?, 0xc0000f9860?}, {0xc0007ce7c0, 0x1c}, {0xc0005a3130, 0xf}, 0x45d964b800)\n\.../pkg/mod/k8s.io/kubernetes@v1.29..../framework/pod/wait.go:459 +0x529\nk8s..../e2e/framework/pod.WaitForPodNameRunningInNamespace(...)\n\.../pkg/mod/k8s.io/kubernetes@v1.29..../framework/pod/wait.go:443\ngithub..../tests/e2e/customsuites.createPod({0x7fb35c4dbf40, 0xc0012cf140}, {0x2e74af0, 0xc0000f9860}, {0xc0005a3130, 0xf}, 0xc000ce6908)\n\.../e2e/customsuites/util.go:218 +0x205\ngithub..../tests/e2e/customsuites.(*s3CSIMounterPodTestSuite).DefineTests.func7({0x7fb35c4dbf40, 0xc0012cf140})\n\.../e2e/customsuites/mounterpod.go:570 +0x12c5",
        },
    }
occurred
In [It] at: .../e2e/customsuites/mounterpod.go:571 @ 02/16/26 10:45:42.537

Scality CSI Driver for S3 E2E Suite::[It] [sig-storage] CSI Volumes [Driver: s3.csi.scality.com] [Testpattern: Pre-provisioned PV (default fs)] mounterpod should mount dynamically provisioned volume when workload pod has fsGroup set [sig-storage]
Stack Traces | 305s run time 
[FAILED] workload pod with fsGroup should reach Running state
Unexpected error:
    <*fmt.wrapError | 0xc000a91dc0>: 
    pod "fsgroup-dynamic-pod-f57cec93" not Running: Timed out after 300.001s.
    Expected Pod to be in <v1.PodPhase>: "Running"
    Got instead:
        <*v1.Pod | 0xc0006da008>: 
            metadata:
              creationTimestamp: "2026-02-16T10:40:35Z"
              generateName: pvc-tester-
              managedFields:
              - apiVersion: v1
                fieldsType: FieldsV1
                fieldsV1:
                  f:metadata:
                    f:generateName: {}
                  f:spec:
                    f:containers:
                      k:{"name":"write-pod"}:
                        .: {}
                        f:command: {}
                        f:image: {}
                        f:imagePullPolicy: {}
                        f:name: {}
                        f:resources: {}
                        f:securityContext:
                          .: {}
                          f:allowPrivilegeEscalation: {}
                          f:capabilities:
                            .: {}
                            f:drop: {}
                          f:runAsGroup: {}
                          f:runAsNonRoot: {}
                          f:runAsUser: {}
                        f:terminationMessagePath: {}
                        f:terminationMessagePolicy: {}
                        f:volumeMounts:
                          .: {}
                          k:{"mountPath":"/mnt/volume1"}:
                            .: {}
                            f:mountPath: {}
                            f:name: {}
                    f:dnsPolicy: {}
                    f:enableServiceLinks: {}
                    f:restartPolicy: {}
                    f:schedulerName: {}
                    f:securityContext:
                      .: {}
                      f:fsGroup: {}
                      f:runAsGroup: {}
                      f:runAsNonRoot: {}
                      f:runAsUser: {}
                      f:seccompProfile:
                        .: {}
                        f:type: {}
                    f:terminationGracePeriodSeconds: {}
                    f:volumes:
                      .: {}
                      k:{"name":"volume1"}:
                        .: {}
                        f:name: {}
                        f:persistentVolumeClaim:
                          .: {}
                          f:claimName: {}
                manager: e2e.test
                operation: Update
                time: "2026-02-16T10:40:35Z"
              - apiVersion: v1
                fieldsType: FieldsV1
                fieldsV1:
                  f:status:
                    f:conditions:
                      k:{"type":"ContainersReady"}:
                        .: {}
                        f:lastProbeTime: {}
                        f:lastTransitionTime: {}
                        f:message: {}
                        f:reason: {}
                        f:status: {}
                        f:type: {}
                      k:{"type":"Initialized"}:
                        .: {}
                        f:lastProbeTime: {}
                        f:lastTransitionTime: {}
                        f:status: {}
                        f:type: {}
                      k:{"type":"PodReadyToStartContainers"}:
                        .: {}
                        f:lastProbeTime: {}
                        f:lastTransitionTime: {}
                        f:status: {}
                        f:type: {}
                      k:{"type":"Ready"}:
                        .: {}
                        f:lastProbeTime: {}
                        f:lastTransitionTime: {}
                        f:message: {}
                        f:reason: {}
                        f:status: {}
                        f:type: {}
                    f:containerStatuses: {}
                    f:hostIP: {}
                    f:hostIPs: {}
                    f:startTime: {}
                manager: kubelet
                operation: Update
                subresource: status
                time: "2026-02-16T10:40:35Z"
              name: fsgroup-dynamic-pod-f57cec93
              namespace: mounterpod-6806
              resourceVersion: "5800"
              uid: 1a1d64ab-5a7b-4c18-9e88-69b719f571bb
            spec:
              containers:
              - command:
                - /bin/sh
                - -c
                - trap exit TERM; while true; do sleep 1; done
                image: registry.k8s.io/e2e-test-images/busybox:1.36.1-1
                imagePullPolicy: IfNotPresent
                name: write-pod
                resources: {}
                securityContext:
                  allowPrivilegeEscalation: false
                  capabilities:
                    drop:
                    - ALL
                  runAsGroup: 2000
                  runAsNonRoot: true
                  runAsUser: 1001
                terminationMessagePath: /dev/termination-log
                terminationMessagePolicy: File
                volumeMounts:
                - mountPath: /mnt/volume1
                  name: volume1
                - mountPath: .../run/secrets/kubernetes.io/serviceaccount
                  name: kube-api-access-z9vdv
                  readOnly: true
              dnsPolicy: ClusterFirst
              enableServiceLinks: true
              nodeName: helm-test-cluster-control-plane
              preemptionPolicy: PreemptLowerPriority
              priority: 0
              restartPolicy: OnFailure
              schedulerName: default-scheduler
              securityContext:
                fsGroup: 1001
                runAsGroup: 2000
                runAsNonRoot: true
                runAsUser: 1001
                seccompProfile:
                  type: RuntimeDefault
              serviceAccount: default
              serviceAccountName: default
              terminationGracePeriodSeconds: 30
              tolerations:
              - effect: NoExecute
                key: node.kubernetes.io/not-ready
                operator: Exists
                tolerationSeconds: 300
              - effect: NoExecute
                key: node.kubernetes.io/unreachable
                operator: Exists
                tolerationSeconds: 300
              volumes:
              - name: volume1
                persistentVolumeClaim:
                  claimName: fsgroup-dynamic-pvc-ca278dbb
              - name: kube-api-access-z9vdv
                projected:
                  defaultMode: 420
                  sources:
                  - serviceAccountToken:
                      expirationSeconds: 3607
                      path: token
                  - configMap:
                      items:
                      - key: ca.crt
                        path: ca.crt
                      name: kube-root-ca.crt
                  - downwardAPI:
                      items:
                      - fieldRef:
                          apiVersion: v1
                          fieldPath: metadata.namespace
                        path: namespace
            status:
              conditions:
              - lastProbeTime: null
                lastTransitionTime: "2026-02-16T10:40:35Z"
                status: "False"
                type: PodReadyToStartContainers
              - lastProbeTime: null
                lastTransitionTime: "2026-02-16T10:40:35Z"
                status: "True"
                type: Initialized
              - lastProbeTime: null
                lastTransitionTime: "2026-02-16T10:40:35Z"
                message: 'containers with unready status: [write-pod]'
                reason: ContainersNotReady
                status: "False"
                type: Ready
              - lastProbeTime: null
                lastTransitionTime: "2026-02-16T10:40:35Z"
                message: 'containers with unready status: [write-pod]'
                reason: ContainersNotReady
                status: "False"
                type: ContainersReady
              - lastProbeTime: null
                lastTransitionTime: "2026-02-16T10:40:35Z"
                status: "True"
                type: PodScheduled
              containerStatuses:
              - image: registry.k8s.io/e2e-test-images/busybox:1.36.1-1
                imageID: ""
                lastState: {}
                name: write-pod
                ready: false
                restartCount: 0
                started: false
                state:
                  waiting:
                    reason: ContainerCreating
              hostIP: 172.18.0.2
              hostIPs:
              - ip: 172.18.0.2
              phase: Pending
              qosClass: BestEffort
              startTime: "2026-02-16T10:40:35Z"
    {
        msg: "pod \"fsgroup-dynamic-pod-f57cec93\" not Running: Timed out after 300.001s.\nExpected Pod to be in <v1.PodPhase>: \"Running\"\nGot instead:\n    <*v1.Pod | 0xc0006da008>: \n        metadata:\n          creationTimestamp: \"2026-02-16T10:40:35Z\"\n          generateName: pvc-tester-\n          managedFields:\n          - apiVersion: v1\n            fieldsType: FieldsV1\n            fieldsV1:\n              f:metadata:\n                f:generateName: {}\n              f:spec:\n                f:containers:\n                  k:{\"name\":\"write-pod\"}:\n                    .: {}\n                    f:command: {}\n                    f:image: {}\n                    f:imagePullPolicy: {}\n                    f:name: {}\n                    f:resources: {}\n                    f:securityContext:\n                      .: {}\n                      f:allowPrivilegeEscalation: {}\n                      f:capabilities:\n                        .: {}\n                        f:drop: {}\n                      f:runAsGroup: {}\n                      f:runAsNonRoot: {}\n                      f:runAsUser: {}\n                    f:terminationMessagePath: {}\n                    f:terminationMessagePolicy: {}\n                    f:volumeMounts:\n                      .: {}\n                      k:{\"mountPath\":\"/mnt/volume1\"}:\n                        .: {}\n                        f:mountPath: {}\n                        f:name: {}\n                f:dnsPolicy: {}\n                f:enableServiceLinks: {}\n                f:restartPolicy: {}\n                f:schedulerName: {}\n                f:securityContext:\n                  .: {}\n                  f:fsGroup: {}\n                  f:runAsGroup: {}\n                  f:runAsNonRoot: {}\n                  f:runAsUser: {}\n                  f:seccompProfile:\n                    .: {}\n                    f:type: {}\n                f:terminationGracePeriodSeconds: {}\n                f:volumes:\n                  .: {}\n                  k:{\"name\":\"volume1\"}:\n                    .: {}\n                    f:name: {}\n                    f:persistentVolumeClaim:\n                      .: {}\n                      f:claimName: {}\n            manager: e2e.test\n            operation: Update\n            time: \"2026-02-16T10:40:35Z\"\n          - apiVersion: v1\n            fieldsType: FieldsV1\n            fieldsV1:\n              f:status:\n                f:conditions:\n                  k:{\"type\":\"ContainersReady\"}:\n                    .: {}\n                    f:lastProbeTime: {}\n                    f:lastTransitionTime: {}\n                    f:message: {}\n                    f:reason: {}\n                    f:status: {}\n                    f:type: {}\n                  k:{\"type\":\"Initialized\"}:\n                    .: {}\n                    f:lastProbeTime: {}\n                    f:lastTransitionTime: {}\n                    f:status: {}\n                    f:type: {}\n                  k:{\"type\":\"PodReadyToStartContainers\"}:\n                    .: {}\n                    f:lastProbeTime: {}\n                    f:lastTransitionTime: {}\n                    f:status: {}\n                    f:type: {}\n                  k:{\"type\":\"Ready\"}:\n                    .: {}\n                    f:lastProbeTime: {}\n                    f:lastTransitionTime: {}\n                    f:message: {}\n                    f:reason: {}\n                    f:status: {}\n                    f:type: {}\n                f:containerStatuses: {}\n                f:hostIP: {}\n                f:hostIPs: {}\n                f:startTime: {}\n            manager: kubelet\n            operation: Update\n            subresource: status\n            time: \"2026-02-16T10:40:35Z\"\n          name: fsgroup-dynamic-pod-f57cec93\n          namespace: mounterpod-6806\n          resourceVersion: \"5800\"\n          uid: 1a1d64ab-5a7b-4c18-9e88-69b719f571bb\n        spec:\n          containers:\n          - command:\n            - /bin/sh\n            - -c\n            - trap exit TERM; while true; do sleep 1; done\n            image: registry.k8s.io/e2e-test-images/busybox:1.36.1-1\n            imagePullPolicy: IfNotPresent\n            name: write-pod\n            resources: {}\n            securityContext:\n              allowPrivilegeEscalation: false\n              capabilities:\n                drop:\n                - ALL\n              runAsGroup: 2000\n              runAsNonRoot: true\n              runAsUser: 1001\n            terminationMessagePath: /dev/termination-log\n            terminationMessagePolicy: File\n            volumeMounts:\n            - mountPath: /mnt/volume1\n              name: volume1\n            - mountPath: .../run/secrets/kubernetes.io/serviceaccount\n              name: kube-api-access-z9vdv\n              readOnly: true\n          dnsPolicy: ClusterFirst\n          enableServiceLinks: true\n          nodeName: helm-test-cluster-control-plane\n          preemptionPolicy: PreemptLowerPriority\n          priority: 0\n          restartPolicy: OnFailure\n          schedulerName: default-scheduler\n          securityContext:\n            fsGroup: 1001\n            runAsGroup: 2000\n            runAsNonRoot: true\n            runAsUser: 1001\n            seccompProfile:\n              type: RuntimeDefault\n          serviceAccount: default\n          serviceAccountName: default\n          terminationGracePeriodSeconds: 30\n          tolerations:\n          - effect: NoExecute\n            key: node.kubernetes.io/not-ready\n            operator: Exists\n            tolerationSeconds: 300\n          - effect: NoExecute\n            key: node.kubernetes.io/unreachable\n            operator: Exists\n            tolerationSeconds: 300\n          volumes:\n          - name: volume1\n            persistentVolumeClaim:\n              claimName: fsgroup-dynamic-pvc-ca278dbb\n          - name: kube-api-access-z9vdv\n            projected:\n              defaultMode: 420\n              sources:\n              - serviceAccountToken:\n                  expirationSeconds: 3607\n                  path: token\n              - configMap:\n                  items:\n                  - key: ca.crt\n                    path: ca.crt\n                  name: kube-root-ca.crt\n              - downwardAPI:\n                  items:\n                  - fieldRef:\n                      apiVersion: v1\n                      fieldPath: metadata.namespace\n                    path: namespace\n        status:\n          conditions:\n          - lastProbeTime: null\n            lastTransitionTime: \"2026-02-16T10:40:35Z\"\n            status: \"False\"\n            type: PodReadyToStartContainers\n          - lastProbeTime: null\n            lastTransitionTime: \"2026-02-16T10:40:35Z\"\n            status: \"True\"\n            type: Initialized\n          - lastProbeTime: null\n            lastTransitionTime: \"2026-02-16T10:40:35Z\"\n            message: 'containers with unready status: [write-pod]'\n            reason: ContainersNotReady\n            status: \"False\"\n            type: Ready\n          - lastProbeTime: null\n            lastTransitionTime: \"2026-02-16T10:40:35Z\"\n            message: 'containers with unready status: [write-pod]'\n            reason: ContainersNotReady\n            status: \"False\"\n            type: ContainersReady\n          - lastProbeTime: null\n            lastTransitionTime: \"2026-02-16T10:40:35Z\"\n            status: \"True\"\n            type: PodScheduled\n          containerStatuses:\n          - image: registry.k8s.io/e2e-test-images/busybox:1.36.1-1\n            imageID: \"\"\n            lastState: {}\n            name: write-pod\n            ready: false\n            restartCount: 0\n            started: false\n            state:\n              waiting:\n                reason: ContainerCreating\n          hostIP: 172.18.0.2\n          hostIPs:\n          - ip: 172.18.0.2\n          phase: Pending\n          qosClass: BestEffort\n          startTime: \"2026-02-16T10:40:35Z\"",
        err: <framework.FailureError>{
            msg: "Timed out after 300.001s.\nExpected Pod to be in <v1.PodPhase>: \"Running\"\nGot instead:\n    <*v1.Pod | 0xc0006da008>: \n        metadata:\n          creationTimestamp: \"2026-02-16T10:40:35Z\"\n          generateName: pvc-tester-\n          managedFields:\n          - apiVersion: v1\n            fieldsType: FieldsV1\n            fieldsV1:\n              f:metadata:\n                f:generateName: {}\n              f:spec:\n                f:containers:\n                  k:{\"name\":\"write-pod\"}:\n                    .: {}\n                    f:command: {}\n                    f:image: {}\n                    f:imagePullPolicy: {}\n                    f:name: {}\n                    f:resources: {}\n                    f:securityContext:\n                      .: {}\n                      f:allowPrivilegeEscalation: {}\n                      f:capabilities:\n                        .: {}\n                        f:drop: {}\n                      f:runAsGroup: {}\n                      f:runAsNonRoot: {}\n                      f:runAsUser: {}\n                    f:terminationMessagePath: {}\n                    f:terminationMessagePolicy: {}\n                    f:volumeMounts:\n                      .: {}\n                      k:{\"mountPath\":\"/mnt/volume1\"}:\n                        .: {}\n                        f:mountPath: {}\n                        f:name: {}\n                f:dnsPolicy: {}\n                f:enableServiceLinks: {}\n                f:restartPolicy: {}\n                f:schedulerName: {}\n                f:securityContext:\n                  .: {}\n                  f:fsGroup: {}\n                  f:runAsGroup: {}\n                  f:runAsNonRoot: {}\n                  f:runAsUser: {}\n                  f:seccompProfile:\n                    .: {}\n                    f:type: {}\n                f:terminationGracePeriodSeconds: {}\n                f:volumes:\n                  .: {}\n                  k:{\"name\":\"volume1\"}:\n                    .: {}\n                    f:name: {}\n                    f:persistentVolumeClaim:\n                      .: {}\n                      f:claimName: {}\n            manager: e2e.test\n            operation: Update\n            time: \"2026-02-16T10:40:35Z\"\n          - apiVersion: v1\n            fieldsType: FieldsV1\n            fieldsV1:\n              f:status:\n                f:conditions:\n                  k:{\"type\":\"ContainersReady\"}:\n                    .: {}\n                    f:lastProbeTime: {}\n                    f:lastTransitionTime: {}\n                    f:message: {}\n                    f:reason: {}\n                    f:status: {}\n                    f:type: {}\n                  k:{\"type\":\"Initialized\"}:\n                    .: {}\n                    f:lastProbeTime: {}\n                    f:lastTransitionTime: {}\n                    f:status: {}\n                    f:type: {}\n                  k:{\"type\":\"PodReadyToStartContainers\"}:\n                    .: {}\n                    f:lastProbeTime: {}\n                    f:lastTransitionTime: {}\n                    f:status: {}\n                    f:type: {}\n                  k:{\"type\":\"Ready\"}:\n                    .: {}\n                    f:lastProbeTime: {}\n                    f:lastTransitionTime: {}\n                    f:message: {}\n                    f:reason: {}\n                    f:status: {}\n                    f:type: {}\n                f:containerStatuses: {}\n                f:hostIP: {}\n                f:hostIPs: {}\n                f:startTime: {}\n            manager: kubelet\n            operation: Update\n            subresource: status\n            time: \"2026-02-16T10:40:35Z\"\n          name: fsgroup-dynamic-pod-f57cec93\n          namespace: mounterpod-6806\n          resourceVersion: \"5800\"\n          uid: 1a1d64ab-5a7b-4c18-9e88-69b719f571bb\n        spec:\n          containers:\n          - command:\n            - /bin/sh\n            - -c\n            - trap exit TERM; while true; do sleep 1; done\n            image: registry.k8s.io/e2e-test-images/busybox:1.36.1-1\n            imagePullPolicy: IfNotPresent\n            name: write-pod\n            resources: {}\n            securityContext:\n              allowPrivilegeEscalation: false\n              capabilities:\n                drop:\n                - ALL\n              runAsGroup: 2000\n              runAsNonRoot: true\n              runAsUser: 1001\n            terminationMessagePath: /dev/termination-log\n            terminationMessagePolicy: File\n            volumeMounts:\n            - mountPath: /mnt/volume1\n              name: volume1\n            - mountPath: .../run/secrets/kubernetes.io/serviceaccount\n              name: kube-api-access-z9vdv\n              readOnly: true\n          dnsPolicy: ClusterFirst\n          enableServiceLinks: true\n          nodeName: helm-test-cluster-control-plane\n          preemptionPolicy: PreemptLowerPriority\n          priority: 0\n          restartPolicy: OnFailure\n          schedulerName: default-scheduler\n          securityContext:\n            fsGroup: 1001\n            runAsGroup: 2000\n            runAsNonRoot: true\n            runAsUser: 1001\n            seccompProfile:\n              type: RuntimeDefault\n          serviceAccount: default\n          serviceAccountName: default\n          terminationGracePeriodSeconds: 30\n          tolerations:\n          - effect: NoExecute\n            key: node.kubernetes.io/not-ready\n            operator: Exists\n            tolerationSeconds: 300\n          - effect: NoExecute\n            key: node.kubernetes.io/unreachable\n            operator: Exists\n            tolerationSeconds: 300\n          volumes:\n          - name: volume1\n            persistentVolumeClaim:\n              claimName: fsgroup-dynamic-pvc-ca278dbb\n          - name: kube-api-access-z9vdv\n            projected:\n              defaultMode: 420\n              sources:\n              - serviceAccountToken:\n                  expirationSeconds: 3607\n                  path: token\n              - configMap:\n                  items:\n                  - key: ca.crt\n                    path: ca.crt\n                  name: kube-root-ca.crt\n              - downwardAPI:\n                  items:\n                  - fieldRef:\n                      apiVersion: v1\n                      fieldPath: metadata.namespace\n                    path: namespace\n        status:\n          conditions:\n          - lastProbeTime: null\n            lastTransitionTime: \"2026-02-16T10:40:35Z\"\n            status: \"False\"\n            type: PodReadyToStartContainers\n          - lastProbeTime: null\n            lastTransitionTime: \"2026-02-16T10:40:35Z\"\n            status: \"True\"\n            type: Initialized\n          - lastProbeTime: null\n            lastTransitionTime: \"2026-02-16T10:40:35Z\"\n            message: 'containers with unready status: [write-pod]'\n            reason: ContainersNotReady\n            status: \"False\"\n            type: Ready\n          - lastProbeTime: null\n            lastTransitionTime: \"2026-02-16T10:40:35Z\"\n            message: 'containers with unready status: [write-pod]'\n            reason: ContainersNotReady\n            status: \"False\"\n            type: ContainersReady\n          - lastProbeTime: null\n            lastTransitionTime: \"2026-02-16T10:40:35Z\"\n            status: \"True\"\n            type: PodScheduled\n          containerStatuses:\n          - image: registry.k8s.io/e2e-test-images/busybox:1.36.1-1\n            imageID: \"\"\n            lastState: {}\n            name: write-pod\n            ready: false\n            restartCount: 0\n            started: false\n            state:\n              waiting:\n                reason: ContainerCreating\n          hostIP: 172.18.0.2\n          hostIPs:\n          - ip: 172.18.0.2\n          phase: Pending\n          qosClass: BestEffort\n          startTime: \"2026-02-16T10:40:35Z\"",
            fullStackTrace: "k8s..../e2e/framework/pod.WaitTimeoutForPodRunningInNamespace({0x7f3d0c4bc040, 0xc001001560}, {0x2e74af0?, 0xc000ba2340?}, {0xc00066df20, 0x1c}, {0xc00079d960, 0xf}, 0x45d964b800)\n\.../pkg/mod/k8s.io/kubernetes@v1.29..../framework/pod/wait.go:459 +0x529\nk8s..../e2e/framework/pod.WaitForPodNameRunningInNamespace(...)\n\.../pkg/mod/k8s.io/kubernetes@v1.29..../framework/pod/wait.go:443\ngithub..../tests/e2e/customsuites.createPod({0x7f3d0c4bc040, 0xc001001560}, {0x2e74af0, 0xc000ba2340}, {0xc00079d960, 0xf}, 0xc0000fd208)\n\.../e2e/customsuites/util.go:218 +0x205\ngithub..../tests/e2e/customsuites.(*s3CSIMounterPodTestSuite).DefineTests.func5({0x7f3d0c4bc040, 0xc001001560})\n\.../e2e/customsuites/mounterpod.go:374 +0x10f8",
        },
    }
occurred
In [It] at: .../e2e/customsuites/mounterpod.go:375 @ 02/16/26 10:45:35.675

Scality CSI Driver for S3 E2E Suite::[It] [sig-storage] CSI Volumes [Driver: s3.csi.scality.com] [Testpattern: Pre-provisioned PV (default fs)] mounterpod should auto-apply gid and mode mount options from workload pod fsGroup [sig-storage]
Stack Traces | 308s run time 
[FAILED] workload pod with fsGroup should reach Running state
Unexpected error:
    <*fmt.wrapError | 0xc0007d3a80>: 
    pod "fsgroup-autoapply-3aae87dc" not Running: Timed out after 300.001s.
    Expected Pod to be in <v1.PodPhase>: "Running"
    Got instead:
        <*v1.Pod | 0xc0008d0008>: 
            metadata:
              creationTimestamp: "2026-02-16T10:40:47Z"
              generateName: pvc-tester-
              managedFields:
              - apiVersion: v1
                fieldsType: FieldsV1
                fieldsV1:
                  f:metadata:
                    f:generateName: {}
                  f:spec:
                    f:containers:
                      k:{"name":"write-pod"}:
                        .: {}
                        f:command: {}
                        f:image: {}
                        f:imagePullPolicy: {}
                        f:name: {}
                        f:resources: {}
                        f:securityContext:
                          .: {}
                          f:allowPrivilegeEscalation: {}
                          f:capabilities:
                            .: {}
                            f:drop: {}
                          f:runAsGroup: {}
                          f:runAsNonRoot: {}
                          f:runAsUser: {}
                        f:terminationMessagePath: {}
                        f:terminationMessagePolicy: {}
                        f:volumeMounts:
                          .: {}
                          k:{"mountPath":"/mnt/volume1"}:
                            .: {}
                            f:mountPath: {}
                            f:name: {}
                    f:dnsPolicy: {}
                    f:enableServiceLinks: {}
                    f:restartPolicy: {}
                    f:schedulerName: {}
                    f:securityContext:
                      .: {}
                      f:fsGroup: {}
                      f:runAsGroup: {}
                      f:runAsNonRoot: {}
                      f:runAsUser: {}
                      f:seccompProfile:
                        .: {}
                        f:type: {}
                    f:terminationGracePeriodSeconds: {}
                    f:volumes:
                      .: {}
                      k:{"name":"volume1"}:
                        .: {}
                        f:name: {}
                        f:persistentVolumeClaim:
                          .: {}
                          f:claimName: {}
                manager: e2e.test
                operation: Update
                time: "2026-02-16T10:40:47Z"
              - apiVersion: v1
                fieldsType: FieldsV1
                fieldsV1:
                  f:status:
                    f:conditions:
                      k:{"type":"ContainersReady"}:
                        .: {}
                        f:lastProbeTime: {}
                        f:lastTransitionTime: {}
                        f:message: {}
                        f:reason: {}
                        f:status: {}
                        f:type: {}
                      k:{"type":"Initialized"}:
                        .: {}
                        f:lastProbeTime: {}
                        f:lastTransitionTime: {}
                        f:status: {}
                        f:type: {}
                      k:{"type":"PodReadyToStartContainers"}:
                        .: {}
                        f:lastProbeTime: {}
                        f:lastTransitionTime: {}
                        f:status: {}
                        f:type: {}
                      k:{"type":"Ready"}:
                        .: {}
                        f:lastProbeTime: {}
                        f:lastTransitionTime: {}
                        f:message: {}
                        f:reason: {}
                        f:status: {}
                        f:type: {}
                    f:containerStatuses: {}
                    f:hostIP: {}
                    f:hostIPs: {}
                    f:startTime: {}
                manager: kubelet
                operation: Update
                subresource: status
                time: "2026-02-16T10:40:47Z"
              name: fsgroup-autoapply-3aae87dc
              namespace: mounterpod-3400
              resourceVersion: "6091"
              uid: 0527c0b3-56a2-4cb9-b879-bca4867a3826
            spec:
              containers:
              - command:
                - /bin/sh
                - -c
                - trap exit TERM; while true; do sleep 1; done
                image: registry.k8s.io/e2e-test-images/busybox:1.36.1-1
                imagePullPolicy: IfNotPresent
                name: write-pod
                resources: {}
                securityContext:
                  allowPrivilegeEscalation: false
                  capabilities:
                    drop:
                    - ALL
                  runAsGroup: 2000
                  runAsNonRoot: true
                  runAsUser: 1001
                terminationMessagePath: /dev/termination-log
                terminationMessagePolicy: File
                volumeMounts:
                - mountPath: /mnt/volume1
                  name: volume1
                - mountPath: .../run/secrets/kubernetes.io/serviceaccount
                  name: kube-api-access-t9r9k
                  readOnly: true
              dnsPolicy: ClusterFirst
              enableServiceLinks: true
              nodeName: helm-test-cluster-control-plane
              preemptionPolicy: PreemptLowerPriority
              priority: 0
              restartPolicy: OnFailure
              schedulerName: default-scheduler
              securityContext:
                fsGroup: 1001
                runAsGroup: 2000
                runAsNonRoot: true
                runAsUser: 1001
                seccompProfile:
                  type: RuntimeDefault
              serviceAccount: default
              serviceAccountName: default
              terminationGracePeriodSeconds: 30
              tolerations:
              - effect: NoExecute
                key: node.kubernetes.io/not-ready
                operator: Exists
                tolerationSeconds: 300
              - effect: NoExecute
                key: node.kubernetes.io/unreachable
                operator: Exists
                tolerationSeconds: 300
              volumes:
              - name: volume1
                persistentVolumeClaim:
                  claimName: s3-e2e-pvc-566a0988-5f0a-41fd-bc04-23b6227aa244
              - name: kube-api-access-t9r9k
                projected:
                  defaultMode: 420
                  sources:
                  - serviceAccountToken:
                      expirationSeconds: 3607
                      path: token
                  - configMap:
                      items:
                      - key: ca.crt
                        path: ca.crt
                      name: kube-root-ca.crt
                  - downwardAPI:
                      items:
                      - fieldRef:
                          apiVersion: v1
                          fieldPath: metadata.namespace
                        path: namespace
            status:
              conditions:
              - lastProbeTime: null
                lastTransitionTime: "2026-02-16T10:40:47Z"
                status: "False"
                type: PodReadyToStartContainers
              - lastProbeTime: null
                lastTransitionTime: "2026-02-16T10:40:47Z"
                status: "True"
                type: Initialized
              - lastProbeTime: null
                lastTransitionTime: "2026-02-16T10:40:47Z"
                message: 'containers with unready status: [write-pod]'
                reason: ContainersNotReady
                status: "False"
                type: Ready
              - lastProbeTime: null
                lastTransitionTime: "2026-02-16T10:40:47Z"
                message: 'containers with unready status: [write-pod]'
                reason: ContainersNotReady
                status: "False"
                type: ContainersReady
              - lastProbeTime: null
                lastTransitionTime: "2026-02-16T10:40:47Z"
                status: "True"
                type: PodScheduled
              containerStatuses:
              - image: registry.k8s.io/e2e-test-images/busybox:1.36.1-1
                imageID: ""
                lastState: {}
                name: write-pod
                ready: false
                restartCount: 0
                started: false
                state:
                  waiting:
                    reason: ContainerCreating
              hostIP: 172.18.0.2
              hostIPs:
              - ip: 172.18.0.2
              phase: Pending
              qosClass: BestEffort
              startTime: "2026-02-16T10:40:47Z"
    {
        msg: "pod \"fsgroup-autoapply-3aae87dc\" not Running: Timed out after 300.001s.\nExpected Pod to be in <v1.PodPhase>: \"Running\"\nGot instead:\n    <*v1.Pod | 0xc0008d0008>: \n        metadata:\n          creationTimestamp: \"2026-02-16T10:40:47Z\"\n          generateName: pvc-tester-\n          managedFields:\n          - apiVersion: v1\n            fieldsType: FieldsV1\n            fieldsV1:\n              f:metadata:\n                f:generateName: {}\n              f:spec:\n                f:containers:\n                  k:{\"name\":\"write-pod\"}:\n                    .: {}\n                    f:command: {}\n                    f:image: {}\n                    f:imagePullPolicy: {}\n                    f:name: {}\n                    f:resources: {}\n                    f:securityContext:\n                      .: {}\n                      f:allowPrivilegeEscalation: {}\n                      f:capabilities:\n                        .: {}\n                        f:drop: {}\n                      f:runAsGroup: {}\n                      f:runAsNonRoot: {}\n                      f:runAsUser: {}\n                    f:terminationMessagePath: {}\n                    f:terminationMessagePolicy: {}\n                    f:volumeMounts:\n                      .: {}\n                      k:{\"mountPath\":\"/mnt/volume1\"}:\n                        .: {}\n                        f:mountPath: {}\n                        f:name: {}\n                f:dnsPolicy: {}\n                f:enableServiceLinks: {}\n                f:restartPolicy: {}\n                f:schedulerName: {}\n                f:securityContext:\n                  .: {}\n                  f:fsGroup: {}\n                  f:runAsGroup: {}\n                  f:runAsNonRoot: {}\n                  f:runAsUser: {}\n                  f:seccompProfile:\n                    .: {}\n                    f:type: {}\n                f:terminationGracePeriodSeconds: {}\n                f:volumes:\n                  .: {}\n                  k:{\"name\":\"volume1\"}:\n                    .: {}\n                    f:name: {}\n                    f:persistentVolumeClaim:\n                      .: {}\n                      f:claimName: {}\n            manager: e2e.test\n            operation: Update\n            time: \"2026-02-16T10:40:47Z\"\n          - apiVersion: v1\n            fieldsType: FieldsV1\n            fieldsV1:\n              f:status:\n                f:conditions:\n                  k:{\"type\":\"ContainersReady\"}:\n                    .: {}\n                    f:lastProbeTime: {}\n                    f:lastTransitionTime: {}\n                    f:message: {}\n                    f:reason: {}\n                    f:status: {}\n                    f:type: {}\n                  k:{\"type\":\"Initialized\"}:\n                    .: {}\n                    f:lastProbeTime: {}\n                    f:lastTransitionTime: {}\n                    f:status: {}\n                    f:type: {}\n                  k:{\"type\":\"PodReadyToStartContainers\"}:\n                    .: {}\n                    f:lastProbeTime: {}\n                    f:lastTransitionTime: {}\n                    f:status: {}\n                    f:type: {}\n                  k:{\"type\":\"Ready\"}:\n                    .: {}\n                    f:lastProbeTime: {}\n                    f:lastTransitionTime: {}\n                    f:message: {}\n                    f:reason: {}\n                    f:status: {}\n                    f:type: {}\n                f:containerStatuses: {}\n                f:hostIP: {}\n                f:hostIPs: {}\n                f:startTime: {}\n            manager: kubelet\n            operation: Update\n            subresource: status\n            time: \"2026-02-16T10:40:47Z\"\n          name: fsgroup-autoapply-3aae87dc\n          namespace: mounterpod-3400\n          resourceVersion: \"6091\"\n          uid: 0527c0b3-56a2-4cb9-b879-bca4867a3826\n        spec:\n          containers:\n          - command:\n            - /bin/sh\n            - -c\n            - trap exit TERM; while true; do sleep 1; done\n            image: registry.k8s.io/e2e-test-images/busybox:1.36.1-1\n            imagePullPolicy: IfNotPresent\n            name: write-pod\n            resources: {}\n            securityContext:\n              allowPrivilegeEscalation: false\n              capabilities:\n                drop:\n                - ALL\n              runAsGroup: 2000\n              runAsNonRoot: true\n              runAsUser: 1001\n            terminationMessagePath: /dev/termination-log\n            terminationMessagePolicy: File\n            volumeMounts:\n            - mountPath: /mnt/volume1\n              name: volume1\n            - mountPath: .../run/secrets/kubernetes.io/serviceaccount\n              name: kube-api-access-t9r9k\n              readOnly: true\n          dnsPolicy: ClusterFirst\n          enableServiceLinks: true\n          nodeName: helm-test-cluster-control-plane\n          preemptionPolicy: PreemptLowerPriority\n          priority: 0\n          restartPolicy: OnFailure\n          schedulerName: default-scheduler\n          securityContext:\n            fsGroup: 1001\n            runAsGroup: 2000\n            runAsNonRoot: true\n            runAsUser: 1001\n            seccompProfile:\n              type: RuntimeDefault\n          serviceAccount: default\n          serviceAccountName: default\n          terminationGracePeriodSeconds: 30\n          tolerations:\n          - effect: NoExecute\n            key: node.kubernetes.io/not-ready\n            operator: Exists\n            tolerationSeconds: 300\n          - effect: NoExecute\n            key: node.kubernetes.io/unreachable\n            operator: Exists\n            tolerationSeconds: 300\n          volumes:\n          - name: volume1\n            persistentVolumeClaim:\n              claimName: s3-e2e-pvc-566a0988-5f0a-41fd-bc04-23b6227aa244\n          - name: kube-api-access-t9r9k\n            projected:\n              defaultMode: 420\n              sources:\n              - serviceAccountToken:\n                  expirationSeconds: 3607\n                  path: token\n              - configMap:\n                  items:\n                  - key: ca.crt\n                    path: ca.crt\n                  name: kube-root-ca.crt\n              - downwardAPI:\n                  items:\n                  - fieldRef:\n                      apiVersion: v1\n                      fieldPath: metadata.namespace\n                    path: namespace\n        status:\n          conditions:\n          - lastProbeTime: null\n            lastTransitionTime: \"2026-02-16T10:40:47Z\"\n            status: \"False\"\n            type: PodReadyToStartContainers\n          - lastProbeTime: null\n            lastTransitionTime: \"2026-02-16T10:40:47Z\"\n            status: \"True\"\n            type: Initialized\n          - lastProbeTime: null\n            lastTransitionTime: \"2026-02-16T10:40:47Z\"\n            message: 'containers with unready status: [write-pod]'\n            reason: ContainersNotReady\n            status: \"False\"\n            type: Ready\n          - lastProbeTime: null\n            lastTransitionTime: \"2026-02-16T10:40:47Z\"\n            message: 'containers with unready status: [write-pod]'\n            reason: ContainersNotReady\n            status: \"False\"\n            type: ContainersReady\n          - lastProbeTime: null\n            lastTransitionTime: \"2026-02-16T10:40:47Z\"\n            status: \"True\"\n            type: PodScheduled\n          containerStatuses:\n          - image: registry.k8s.io/e2e-test-images/busybox:1.36.1-1\n            imageID: \"\"\n            lastState: {}\n            name: write-pod\n            ready: false\n            restartCount: 0\n            started: false\n            state:\n              waiting:\n                reason: ContainerCreating\n          hostIP: 172.18.0.2\n          hostIPs:\n          - ip: 172.18.0.2\n          phase: Pending\n          qosClass: BestEffort\n          startTime: \"2026-02-16T10:40:47Z\"",
        err: <framework.FailureError>{
            msg: "Timed out after 300.001s.\nExpected Pod to be in <v1.PodPhase>: \"Running\"\nGot instead:\n    <*v1.Pod | 0xc0008d0008>: \n        metadata:\n          creationTimestamp: \"2026-02-16T10:40:47Z\"\n          generateName: pvc-tester-\n          managedFields:\n          - apiVersion: v1\n            fieldsType: FieldsV1\n            fieldsV1:\n              f:metadata:\n                f:generateName: {}\n              f:spec:\n                f:containers:\n                  k:{\"name\":\"write-pod\"}:\n                    .: {}\n                    f:command: {}\n                    f:image: {}\n                    f:imagePullPolicy: {}\n                    f:name: {}\n                    f:resources: {}\n                    f:securityContext:\n                      .: {}\n                      f:allowPrivilegeEscalation: {}\n                      f:capabilities:\n                        .: {}\n                        f:drop: {}\n                      f:runAsGroup: {}\n                      f:runAsNonRoot: {}\n                      f:runAsUser: {}\n                    f:terminationMessagePath: {}\n                    f:terminationMessagePolicy: {}\n                    f:volumeMounts:\n                      .: {}\n                      k:{\"mountPath\":\"/mnt/volume1\"}:\n                        .: {}\n                        f:mountPath: {}\n                        f:name: {}\n                f:dnsPolicy: {}\n                f:enableServiceLinks: {}\n                f:restartPolicy: {}\n                f:schedulerName: {}\n                f:securityContext:\n                  .: {}\n                  f:fsGroup: {}\n                  f:runAsGroup: {}\n                  f:runAsNonRoot: {}\n                  f:runAsUser: {}\n                  f:seccompProfile:\n                    .: {}\n                    f:type: {}\n                f:terminationGracePeriodSeconds: {}\n                f:volumes:\n                  .: {}\n                  k:{\"name\":\"volume1\"}:\n                    .: {}\n                    f:name: {}\n                    f:persistentVolumeClaim:\n                      .: {}\n                      f:claimName: {}\n            manager: e2e.test\n            operation: Update\n            time: \"2026-02-16T10:40:47Z\"\n          - apiVersion: v1\n            fieldsType: FieldsV1\n            fieldsV1:\n              f:status:\n                f:conditions:\n                  k:{\"type\":\"ContainersReady\"}:\n                    .: {}\n                    f:lastProbeTime: {}\n                    f:lastTransitionTime: {}\n                    f:message: {}\n                    f:reason: {}\n                    f:status: {}\n                    f:type: {}\n                  k:{\"type\":\"Initialized\"}:\n                    .: {}\n                    f:lastProbeTime: {}\n                    f:lastTransitionTime: {}\n                    f:status: {}\n                    f:type: {}\n                  k:{\"type\":\"PodReadyToStartContainers\"}:\n                    .: {}\n                    f:lastProbeTime: {}\n                    f:lastTransitionTime: {}\n                    f:status: {}\n                    f:type: {}\n                  k:{\"type\":\"Ready\"}:\n                    .: {}\n                    f:lastProbeTime: {}\n                    f:lastTransitionTime: {}\n                    f:message: {}\n                    f:reason: {}\n                    f:status: {}\n                    f:type: {}\n                f:containerStatuses: {}\n                f:hostIP: {}\n                f:hostIPs: {}\n                f:startTime: {}\n            manager: kubelet\n            operation: Update\n            subresource: status\n            time: \"2026-02-16T10:40:47Z\"\n          name: fsgroup-autoapply-3aae87dc\n          namespace: mounterpod-3400\n          resourceVersion: \"6091\"\n          uid: 0527c0b3-56a2-4cb9-b879-bca4867a3826\n        spec:\n          containers:\n          - command:\n            - /bin/sh\n            - -c\n            - trap exit TERM; while true; do sleep 1; done\n            image: registry.k8s.io/e2e-test-images/busybox:1.36.1-1\n            imagePullPolicy: IfNotPresent\n            name: write-pod\n            resources: {}\n            securityContext:\n              allowPrivilegeEscalation: false\n              capabilities:\n                drop:\n                - ALL\n              runAsGroup: 2000\n              runAsNonRoot: true\n              runAsUser: 1001\n            terminationMessagePath: /dev/termination-log\n            terminationMessagePolicy: File\n            volumeMounts:\n            - mountPath: /mnt/volume1\n              name: volume1\n            - mountPath: .../run/secrets/kubernetes.io/serviceaccount\n              name: kube-api-access-t9r9k\n              readOnly: true\n          dnsPolicy: ClusterFirst\n          enableServiceLinks: true\n          nodeName: helm-test-cluster-control-plane\n          preemptionPolicy: PreemptLowerPriority\n          priority: 0\n          restartPolicy: OnFailure\n          schedulerName: default-scheduler\n          securityContext:\n            fsGroup: 1001\n            runAsGroup: 2000\n            runAsNonRoot: true\n            runAsUser: 1001\n            seccompProfile:\n              type: RuntimeDefault\n          serviceAccount: default\n          serviceAccountName: default\n          terminationGracePeriodSeconds: 30\n          tolerations:\n          - effect: NoExecute\n            key: node.kubernetes.io/not-ready\n            operator: Exists\n            tolerationSeconds: 300\n          - effect: NoExecute\n            key: node.kubernetes.io/unreachable\n            operator: Exists\n            tolerationSeconds: 300\n          volumes:\n          - name: volume1\n            persistentVolumeClaim:\n              claimName: s3-e2e-pvc-566a0988-5f0a-41fd-bc04-23b6227aa244\n          - name: kube-api-access-t9r9k\n            projected:\n              defaultMode: 420\n              sources:\n              - serviceAccountToken:\n                  expirationSeconds: 3607\n                  path: token\n              - configMap:\n                  items:\n                  - key: ca.crt\n                    path: ca.crt\n                  name: kube-root-ca.crt\n              - downwardAPI:\n                  items:\n                  - fieldRef:\n                      apiVersion: v1\n                      fieldPath: metadata.namespace\n                    path: namespace\n        status:\n          conditions:\n          - lastProbeTime: null\n            lastTransitionTime: \"2026-02-16T10:40:47Z\"\n            status: \"False\"\n            type: PodReadyToStartContainers\n          - lastProbeTime: null\n            lastTransitionTime: \"2026-02-16T10:40:47Z\"\n            status: \"True\"\n            type: Initialized\n          - lastProbeTime: null\n            lastTransitionTime: \"2026-02-16T10:40:47Z\"\n            message: 'containers with unready status: [write-pod]'\n            reason: ContainersNotReady\n            status: \"False\"\n            type: Ready\n          - lastProbeTime: null\n            lastTransitionTime: \"2026-02-16T10:40:47Z\"\n            message: 'containers with unready status: [write-pod]'\n            reason: ContainersNotReady\n            status: \"False\"\n            type: ContainersReady\n          - lastProbeTime: null\n            lastTransitionTime: \"2026-02-16T10:40:47Z\"\n            status: \"True\"\n            type: PodScheduled\n          containerStatuses:\n          - image: registry.k8s.io/e2e-test-images/busybox:1.36.1-1\n            imageID: \"\"\n            lastState: {}\n            name: write-pod\n            ready: false\n            restartCount: 0\n            started: false\n            state:\n              waiting:\n                reason: ContainerCreating\n          hostIP: 172.18.0.2\n          hostIPs:\n          - ip: 172.18.0.2\n          phase: Pending\n          qosClass: BestEffort\n          startTime: \"2026-02-16T10:40:47Z\"",
            fullStackTrace: "k8s..../e2e/framework/pod.WaitTimeoutForPodRunningInNamespace({0x7f14d454cb78, 0xc000da24b0}, {0x2e74af0?, 0xc0004fe820?}, {0xc000d0c6a0, 0x1a}, {0xc000353250, 0xf}, 0x45d964b800)\n\.../pkg/mod/k8s.io/kubernetes@v1.29..../framework/pod/wait.go:459 +0x529\nk8s..../e2e/framework/pod.WaitForPodNameRunningInNamespace(...)\n\.../pkg/mod/k8s.io/kubernetes@v1.29..../framework/pod/wait.go:443\ngithub..../tests/e2e/customsuites.createPod({0x7f14d454cb78, 0xc000da24b0}, {0x2e74af0, 0xc0004fe820}, {0xc000353250, 0xf}, 0xc000af6008)\n\.../e2e/customsuites/util.go:218 +0x205\ngithub..../tests/e2e/customsuites.(*s3CSIMounterPodTestSuite).DefineTests.func8({0x7f14d454cb78, 0xc000da24b0})\n\.../e2e/customsuites/mounterpod.go:653 +0x44b",
        },
    }
occurred
In [It] at: .../e2e/customsuites/mounterpod.go:654 @ 02/16/26 10:45:47.977

Scality CSI Driver for S3 E2E Suite::[It] [sig-storage] CSI Volumes [Driver: s3.csi.scality.com] [Testpattern: Pre-provisioned PV (default fs)] mounterpod should override gid mount option with workload pod fsGroup [sig-storage]
Stack Traces | 308s run time 
[FAILED] workload pod with fsGroup should reach Running state
Unexpected error:
    <*fmt.wrapError | 0xc000d39820>: 
    pod "fsgroup-override-1271c326" not Running: Timed out after 300.000s.
    Expected Pod to be in <v1.PodPhase>: "Running"
    Got instead:
        <*v1.Pod | 0xc0001ead88>: 
            metadata:
              creationTimestamp: "2026-02-16T10:40:48Z"
              generateName: pvc-tester-
              managedFields:
              - apiVersion: v1
                fieldsType: FieldsV1
                fieldsV1:
                  f:metadata:
                    f:generateName: {}
                  f:spec:
                    f:containers:
                      k:{"name":"write-pod"}:
                        .: {}
                        f:command: {}
                        f:image: {}
                        f:imagePullPolicy: {}
                        f:name: {}
                        f:resources: {}
                        f:securityContext:
                          .: {}
                          f:allowPrivilegeEscalation: {}
                          f:capabilities:
                            .: {}
                            f:drop: {}
                          f:runAsGroup: {}
                          f:runAsNonRoot: {}
                          f:runAsUser: {}
                        f:terminationMessagePath: {}
                        f:terminationMessagePolicy: {}
                        f:volumeMounts:
                          .: {}
                          k:{"mountPath":"/mnt/volume1"}:
                            .: {}
                            f:mountPath: {}
                            f:name: {}
                    f:dnsPolicy: {}
                    f:enableServiceLinks: {}
                    f:restartPolicy: {}
                    f:schedulerName: {}
                    f:securityContext:
                      .: {}
                      f:fsGroup: {}
                      f:runAsGroup: {}
                      f:runAsNonRoot: {}
                      f:runAsUser: {}
                      f:seccompProfile:
                        .: {}
                        f:type: {}
                    f:terminationGracePeriodSeconds: {}
                    f:volumes:
                      .: {}
                      k:{"name":"volume1"}:
                        .: {}
                        f:name: {}
                        f:persistentVolumeClaim:
                          .: {}
                          f:claimName: {}
                manager: e2e.test
                operation: Update
                time: "2026-02-16T10:40:48Z"
              - apiVersion: v1
                fieldsType: FieldsV1
                fieldsV1:
                  f:status:
                    f:conditions:
                      k:{"type":"ContainersReady"}:
                        .: {}
                        f:lastProbeTime: {}
                        f:lastTransitionTime: {}
                        f:message: {}
                        f:reason: {}
                        f:status: {}
                        f:type: {}
                      k:{"type":"Initialized"}:
                        .: {}
                        f:lastProbeTime: {}
                        f:lastTransitionTime: {}
                        f:status: {}
                        f:type: {}
                      k:{"type":"PodReadyToStartContainers"}:
                        .: {}
                        f:lastProbeTime: {}
                        f:lastTransitionTime: {}
                        f:status: {}
                        f:type: {}
                      k:{"type":"Ready"}:
                        .: {}
                        f:lastProbeTime: {}
                        f:lastTransitionTime: {}
                        f:message: {}
                        f:reason: {}
                        f:status: {}
                        f:type: {}
                    f:containerStatuses: {}
                    f:hostIP: {}
                    f:hostIPs: {}
                    f:startTime: {}
                manager: kubelet
                operation: Update
                subresource: status
                time: "2026-02-16T10:40:48Z"
              name: fsgroup-override-1271c326
              namespace: mounterpod-497
              resourceVersion: "6104"
              uid: 0481d5cf-4591-4511-a8ab-b94dbdd8f577
            spec:
              containers:
              - command:
                - /bin/sh
                - -c
                - trap exit TERM; while true; do sleep 1; done
                image: registry.k8s.io/e2e-test-images/busybox:1.36.1-1
                imagePullPolicy: IfNotPresent
                name: write-pod
                resources: {}
                securityContext:
                  allowPrivilegeEscalation: false
                  capabilities:
                    drop:
                    - ALL
                  runAsGroup: 2000
                  runAsNonRoot: true
                  runAsUser: 1001
                terminationMessagePath: /dev/termination-log
                terminationMessagePolicy: File
                volumeMounts:
                - mountPath: /mnt/volume1
                  name: volume1
                - mountPath: .../run/secrets/kubernetes.io/serviceaccount
                  name: kube-api-access-2z4d6
                  readOnly: true
              dnsPolicy: ClusterFirst
              enableServiceLinks: true
              nodeName: helm-test-cluster-control-plane
              preemptionPolicy: PreemptLowerPriority
              priority: 0
              restartPolicy: OnFailure
              schedulerName: default-scheduler
              securityContext:
                fsGroup: 1001
                runAsGroup: 2000
                runAsNonRoot: true
                runAsUser: 1001
                seccompProfile:
                  type: RuntimeDefault
              serviceAccount: default
              serviceAccountName: default
              terminationGracePeriodSeconds: 30
              tolerations:
              - effect: NoExecute
                key: node.kubernetes.io/not-ready
                operator: Exists
                tolerationSeconds: 300
              - effect: NoExecute
                key: node.kubernetes.io/unreachable
                operator: Exists
                tolerationSeconds: 300
              volumes:
              - name: volume1
                persistentVolumeClaim:
                  claimName: s3-e2e-pvc-0b2a0e36-d076-44ef-922a-9e7a4505ab0e
              - name: kube-api-access-2z4d6
                projected:
                  defaultMode: 420
                  sources:
                  - serviceAccountToken:
                      expirationSeconds: 3607
                      path: token
                  - configMap:
                      items:
                      - key: ca.crt
                        path: ca.crt
                      name: kube-root-ca.crt
                  - downwardAPI:
                      items:
                      - fieldRef:
                          apiVersion: v1
                          fieldPath: metadata.namespace
                        path: namespace
            status:
              conditions:
              - lastProbeTime: null
                lastTransitionTime: "2026-02-16T10:40:48Z"
                status: "False"
                type: PodReadyToStartContainers
              - lastProbeTime: null
                lastTransitionTime: "2026-02-16T10:40:48Z"
                status: "True"
                type: Initialized
              - lastProbeTime: null
                lastTransitionTime: "2026-02-16T10:40:48Z"
                message: 'containers with unready status: [write-pod]'
                reason: ContainersNotReady
                status: "False"
                type: Ready
              - lastProbeTime: null
                lastTransitionTime: "2026-02-16T10:40:48Z"
                message: 'containers with unready status: [write-pod]'
                reason: ContainersNotReady
                status: "False"
                type: ContainersReady
              - lastProbeTime: null
                lastTransitionTime: "2026-02-16T10:40:48Z"
                status: "True"
                type: PodScheduled
              containerStatuses:
              - image: registry.k8s.io/e2e-test-images/busybox:1.36.1-1
                imageID: ""
                lastState: {}
                name: write-pod
                ready: false
                restartCount: 0
                started: false
                state:
                  waiting:
                    reason: ContainerCreating
              hostIP: 172.18.0.2
              hostIPs:
              - ip: 172.18.0.2
              phase: Pending
              qosClass: BestEffort
              startTime: "2026-02-16T10:40:48Z"
    {
        msg: "pod \"fsgroup-override-1271c326\" not Running: Timed out after 300.000s.\nExpected Pod to be in <v1.PodPhase>: \"Running\"\nGot instead:\n    <*v1.Pod | 0xc0001ead88>: \n        metadata:\n          creationTimestamp: \"2026-02-16T10:40:48Z\"\n          generateName: pvc-tester-\n          managedFields:\n          - apiVersion: v1\n            fieldsType: FieldsV1\n            fieldsV1:\n              f:metadata:\n                f:generateName: {}\n              f:spec:\n                f:containers:\n                  k:{\"name\":\"write-pod\"}:\n                    .: {}\n                    f:command: {}\n                    f:image: {}\n                    f:imagePullPolicy: {}\n                    f:name: {}\n                    f:resources: {}\n                    f:securityContext:\n                      .: {}\n                      f:allowPrivilegeEscalation: {}\n                      f:capabilities:\n                        .: {}\n                        f:drop: {}\n                      f:runAsGroup: {}\n                      f:runAsNonRoot: {}\n                      f:runAsUser: {}\n                    f:terminationMessagePath: {}\n                    f:terminationMessagePolicy: {}\n                    f:volumeMounts:\n                      .: {}\n                      k:{\"mountPath\":\"/mnt/volume1\"}:\n                        .: {}\n                        f:mountPath: {}\n                        f:name: {}\n                f:dnsPolicy: {}\n                f:enableServiceLinks: {}\n                f:restartPolicy: {}\n                f:schedulerName: {}\n                f:securityContext:\n                  .: {}\n                  f:fsGroup: {}\n                  f:runAsGroup: {}\n                  f:runAsNonRoot: {}\n                  f:runAsUser: {}\n                  f:seccompProfile:\n                    .: {}\n                    f:type: {}\n                f:terminationGracePeriodSeconds: {}\n                f:volumes:\n                  .: {}\n                  k:{\"name\":\"volume1\"}:\n                    .: {}\n                    f:name: {}\n                    f:persistentVolumeClaim:\n                      .: {}\n                      f:claimName: {}\n            manager: e2e.test\n            operation: Update\n            time: \"2026-02-16T10:40:48Z\"\n          - apiVersion: v1\n            fieldsType: FieldsV1\n            fieldsV1:\n              f:status:\n                f:conditions:\n                  k:{\"type\":\"ContainersReady\"}:\n                    .: {}\n                    f:lastProbeTime: {}\n                    f:lastTransitionTime: {}\n                    f:message: {}\n                    f:reason: {}\n                    f:status: {}\n                    f:type: {}\n                  k:{\"type\":\"Initialized\"}:\n                    .: {}\n                    f:lastProbeTime: {}\n                    f:lastTransitionTime: {}\n                    f:status: {}\n                    f:type: {}\n                  k:{\"type\":\"PodReadyToStartContainers\"}:\n                    .: {}\n                    f:lastProbeTime: {}\n                    f:lastTransitionTime: {}\n                    f:status: {}\n                    f:type: {}\n                  k:{\"type\":\"Ready\"}:\n                    .: {}\n                    f:lastProbeTime: {}\n                    f:lastTransitionTime: {}\n                    f:message: {}\n                    f:reason: {}\n                    f:status: {}\n                    f:type: {}\n                f:containerStatuses: {}\n                f:hostIP: {}\n                f:hostIPs: {}\n                f:startTime: {}\n            manager: kubelet\n            operation: Update\n            subresource: status\n            time: \"2026-02-16T10:40:48Z\"\n          name: fsgroup-override-1271c326\n          namespace: mounterpod-497\n          resourceVersion: \"6104\"\n          uid: 0481d5cf-4591-4511-a8ab-b94dbdd8f577\n        spec:\n          containers:\n          - command:\n            - /bin/sh\n            - -c\n            - trap exit TERM; while true; do sleep 1; done\n            image: registry.k8s.io/e2e-test-images/busybox:1.36.1-1\n            imagePullPolicy: IfNotPresent\n            name: write-pod\n            resources: {}\n            securityContext:\n              allowPrivilegeEscalation: false\n              capabilities:\n                drop:\n                - ALL\n              runAsGroup: 2000\n              runAsNonRoot: true\n              runAsUser: 1001\n            terminationMessagePath: /dev/termination-log\n            terminationMessagePolicy: File\n            volumeMounts:\n            - mountPath: /mnt/volume1\n              name: volume1\n            - mountPath: .../run/secrets/kubernetes.io/serviceaccount\n              name: kube-api-access-2z4d6\n              readOnly: true\n          dnsPolicy: ClusterFirst\n          enableServiceLinks: true\n          nodeName: helm-test-cluster-control-plane\n          preemptionPolicy: PreemptLowerPriority\n          priority: 0\n          restartPolicy: OnFailure\n          schedulerName: default-scheduler\n          securityContext:\n            fsGroup: 1001\n            runAsGroup: 2000\n            runAsNonRoot: true\n            runAsUser: 1001\n            seccompProfile:\n              type: RuntimeDefault\n          serviceAccount: default\n          serviceAccountName: default\n          terminationGracePeriodSeconds: 30\n          tolerations:\n          - effect: NoExecute\n            key: node.kubernetes.io/not-ready\n            operator: Exists\n            tolerationSeconds: 300\n          - effect: NoExecute\n            key: node.kubernetes.io/unreachable\n            operator: Exists\n            tolerationSeconds: 300\n          volumes:\n          - name: volume1\n            persistentVolumeClaim:\n              claimName: s3-e2e-pvc-0b2a0e36-d076-44ef-922a-9e7a4505ab0e\n          - name: kube-api-access-2z4d6\n            projected:\n              defaultMode: 420\n              sources:\n              - serviceAccountToken:\n                  expirationSeconds: 3607\n                  path: token\n              - configMap:\n                  items:\n                  - key: ca.crt\n                    path: ca.crt\n                  name: kube-root-ca.crt\n              - downwardAPI:\n                  items:\n                  - fieldRef:\n                      apiVersion: v1\n                      fieldPath: metadata.namespace\n                    path: namespace\n        status:\n          conditions:\n          - lastProbeTime: null\n            lastTransitionTime: \"2026-02-16T10:40:48Z\"\n            status: \"False\"\n            type: PodReadyToStartContainers\n          - lastProbeTime: null\n            lastTransitionTime: \"2026-02-16T10:40:48Z\"\n            status: \"True\"\n            type: Initialized\n          - lastProbeTime: null\n            lastTransitionTime: \"2026-02-16T10:40:48Z\"\n            message: 'containers with unready status: [write-pod]'\n            reason: ContainersNotReady\n            status: \"False\"\n            type: Ready\n          - lastProbeTime: null\n            lastTransitionTime: \"2026-02-16T10:40:48Z\"\n            message: 'containers with unready status: [write-pod]'\n            reason: ContainersNotReady\n            status: \"False\"\n            type: ContainersReady\n          - lastProbeTime: null\n            lastTransitionTime: \"2026-02-16T10:40:48Z\"\n            status: \"True\"\n            type: PodScheduled\n          containerStatuses:\n          - image: registry.k8s.io/e2e-test-images/busybox:1.36.1-1\n            imageID: \"\"\n            lastState: {}\n            name: write-pod\n            ready: false\n            restartCount: 0\n            started: false\n            state:\n              waiting:\n                reason: ContainerCreating\n          hostIP: 172.18.0.2\n          hostIPs:\n          - ip: 172.18.0.2\n          phase: Pending\n          qosClass: BestEffort\n          startTime: \"2026-02-16T10:40:48Z\"",
        err: <framework.FailureError>{
            msg: "Timed out after 300.000s.\nExpected Pod to be in <v1.PodPhase>: \"Running\"\nGot instead:\n    <*v1.Pod | 0xc0001ead88>: \n        metadata:\n          creationTimestamp: \"2026-02-16T10:40:48Z\"\n          generateName: pvc-tester-\n          managedFields:\n          - apiVersion: v1\n            fieldsType: FieldsV1\n            fieldsV1:\n              f:metadata:\n                f:generateName: {}\n              f:spec:\n                f:containers:\n                  k:{\"name\":\"write-pod\"}:\n                    .: {}\n                    f:command: {}\n                    f:image: {}\n                    f:imagePullPolicy: {}\n                    f:name: {}\n                    f:resources: {}\n                    f:securityContext:\n                      .: {}\n                      f:allowPrivilegeEscalation: {}\n                      f:capabilities:\n                        .: {}\n                        f:drop: {}\n                      f:runAsGroup: {}\n                      f:runAsNonRoot: {}\n                      f:runAsUser: {}\n                    f:terminationMessagePath: {}\n                    f:terminationMessagePolicy: {}\n                    f:volumeMounts:\n                      .: {}\n                      k:{\"mountPath\":\"/mnt/volume1\"}:\n                        .: {}\n                        f:mountPath: {}\n                        f:name: {}\n                f:dnsPolicy: {}\n                f:enableServiceLinks: {}\n                f:restartPolicy: {}\n                f:schedulerName: {}\n                f:securityContext:\n                  .: {}\n                  f:fsGroup: {}\n                  f:runAsGroup: {}\n                  f:runAsNonRoot: {}\n                  f:runAsUser: {}\n                  f:seccompProfile:\n                    .: {}\n                    f:type: {}\n                f:terminationGracePeriodSeconds: {}\n                f:volumes:\n                  .: {}\n                  k:{\"name\":\"volume1\"}:\n                    .: {}\n                    f:name: {}\n                    f:persistentVolumeClaim:\n                      .: {}\n                      f:claimName: {}\n            manager: e2e.test\n            operation: Update\n            time: \"2026-02-16T10:40:48Z\"\n          - apiVersion: v1\n            fieldsType: FieldsV1\n            fieldsV1:\n              f:status:\n                f:conditions:\n                  k:{\"type\":\"ContainersReady\"}:\n                    .: {}\n                    f:lastProbeTime: {}\n                    f:lastTransitionTime: {}\n                    f:message: {}\n                    f:reason: {}\n                    f:status: {}\n                    f:type: {}\n                  k:{\"type\":\"Initialized\"}:\n                    .: {}\n                    f:lastProbeTime: {}\n                    f:lastTransitionTime: {}\n                    f:status: {}\n                    f:type: {}\n                  k:{\"type\":\"PodReadyToStartContainers\"}:\n                    .: {}\n                    f:lastProbeTime: {}\n                    f:lastTransitionTime: {}\n                    f:status: {}\n                    f:type: {}\n                  k:{\"type\":\"Ready\"}:\n                    .: {}\n                    f:lastProbeTime: {}\n                    f:lastTransitionTime: {}\n                    f:message: {}\n                    f:reason: {}\n                    f:status: {}\n                    f:type: {}\n                f:containerStatuses: {}\n                f:hostIP: {}\n                f:hostIPs: {}\n                f:startTime: {}\n            manager: kubelet\n            operation: Update\n            subresource: status\n            time: \"2026-02-16T10:40:48Z\"\n          name: fsgroup-override-1271c326\n          namespace: mounterpod-497\n          resourceVersion: \"6104\"\n          uid: 0481d5cf-4591-4511-a8ab-b94dbdd8f577\n        spec:\n          containers:\n          - command:\n            - /bin/sh\n            - -c\n            - trap exit TERM; while true; do sleep 1; done\n            image: registry.k8s.io/e2e-test-images/busybox:1.36.1-1\n            imagePullPolicy: IfNotPresent\n            name: write-pod\n            resources: {}\n            securityContext:\n              allowPrivilegeEscalation: false\n              capabilities:\n                drop:\n                - ALL\n              runAsGroup: 2000\n              runAsNonRoot: true\n              runAsUser: 1001\n            terminationMessagePath: /dev/termination-log\n            terminationMessagePolicy: File\n            volumeMounts:\n            - mountPath: /mnt/volume1\n              name: volume1\n            - mountPath: .../run/secrets/kubernetes.io/serviceaccount\n              name: kube-api-access-2z4d6\n              readOnly: true\n          dnsPolicy: ClusterFirst\n          enableServiceLinks: true\n          nodeName: helm-test-cluster-control-plane\n          preemptionPolicy: PreemptLowerPriority\n          priority: 0\n          restartPolicy: OnFailure\n          schedulerName: default-scheduler\n          securityContext:\n            fsGroup: 1001\n            runAsGroup: 2000\n            runAsNonRoot: true\n            runAsUser: 1001\n            seccompProfile:\n              type: RuntimeDefault\n          serviceAccount: default\n          serviceAccountName: default\n          terminationGracePeriodSeconds: 30\n          tolerations:\n          - effect: NoExecute\n            key: node.kubernetes.io/not-ready\n            operator: Exists\n            tolerationSeconds: 300\n          - effect: NoExecute\n            key: node.kubernetes.io/unreachable\n            operator: Exists\n            tolerationSeconds: 300\n          volumes:\n          - name: volume1\n            persistentVolumeClaim:\n              claimName: s3-e2e-pvc-0b2a0e36-d076-44ef-922a-9e7a4505ab0e\n          - name: kube-api-access-2z4d6\n            projected:\n              defaultMode: 420\n              sources:\n              - serviceAccountToken:\n                  expirationSeconds: 3607\n                  path: token\n              - configMap:\n                  items:\n                  - key: ca.crt\n                    path: ca.crt\n                  name: kube-root-ca.crt\n              - downwardAPI:\n                  items:\n                  - fieldRef:\n                      apiVersion: v1\n                      fieldPath: metadata.namespace\n                    path: namespace\n        status:\n          conditions:\n          - lastProbeTime: null\n            lastTransitionTime: \"2026-02-16T10:40:48Z\"\n            status: \"False\"\n            type: PodReadyToStartContainers\n          - lastProbeTime: null\n            lastTransitionTime: \"2026-02-16T10:40:48Z\"\n            status: \"True\"\n            type: Initialized\n          - lastProbeTime: null\n            lastTransitionTime: \"2026-02-16T10:40:48Z\"\n            message: 'containers with unready status: [write-pod]'\n            reason: ContainersNotReady\n            status: \"False\"\n            type: Ready\n          - lastProbeTime: null\n            lastTransitionTime: \"2026-02-16T10:40:48Z\"\n            message: 'containers with unready status: [write-pod]'\n            reason: ContainersNotReady\n            status: \"False\"\n            type: ContainersReady\n          - lastProbeTime: null\n            lastTransitionTime: \"2026-02-16T10:40:48Z\"\n            status: \"True\"\n            type: PodScheduled\n          containerStatuses:\n          - image: registry.k8s.io/e2e-test-images/busybox:1.36.1-1\n            imageID: \"\"\n            lastState: {}\n            name: write-pod\n            ready: false\n            restartCount: 0\n            started: false\n            state:\n              waiting:\n                reason: ContainerCreating\n          hostIP: 172.18.0.2\n          hostIPs:\n          - ip: 172.18.0.2\n          phase: Pending\n          qosClass: BestEffort\n          startTime: \"2026-02-16T10:40:48Z\"",
            fullStackTrace: "k8s..../e2e/framework/pod.WaitTimeoutForPodRunningInNamespace({0x7f73b473b4f0, 0xc000ad1e90}, {0x2e74af0?, 0xc000228000?}, {0xc000371c20, 0x19}, {0xc0014b82c0, 0xe}, 0x45d964b800)\n\.../pkg/mod/k8s.io/kubernetes@v1.29..../framework/pod/wait.go:459 +0x529\nk8s..../e2e/framework/pod.WaitForPodNameRunningInNamespace(...)\n\.../pkg/mod/k8s.io/kubernetes@v1.29..../framework/pod/wait.go:443\ngithub..../tests/e2e/customsuites.createPod({0x7f73b473b4f0, 0xc000ad1e90}, {0x2e74af0, 0xc000228000}, {0xc0014b82c0, 0xe}, 0xc000e00008)\n\.../e2e/customsuites/util.go:218 +0x205\ngithub..../tests/e2e/customsuites.(*s3CSIMounterPodTestSuite).DefineTests.func9({0x7f73b473b4f0, 0xc000ad1e90})\n\.../e2e/customsuites/mounterpod.go:710 +0x3eb",
        },
    }
occurred
In [It] at: .../e2e/customsuites/mounterpod.go:711 @ 02/16/26 10:45:48.083

Scality CSI Driver for S3 E2E Suite::[It] [sig-storage] CSI Volumes [Driver: s3.csi.scality.com] [Testpattern: Pre-provisioned PV (default fs)] mounterpod should create separate mounter pods for workload pods with different fsGroup values [sig-storage]
Stack Traces | 314s run time 
[FAILED] first workload pod should reach Running state
Unexpected error:
    <*fmt.wrapError | 0xc000b78840>: 
    pod "fsgroup-share-1-b698e426" not Running: Timed out after 300.001s.
    Expected Pod to be in <v1.PodPhase>: "Running"
    Got instead:
        <*v1.Pod | 0xc000245b08>: 
            metadata:
              creationTimestamp: "2026-02-16T10:40:48Z"
              generateName: pvc-tester-
              managedFields:
              - apiVersion: v1
                fieldsType: FieldsV1
                fieldsV1:
                  f:metadata:
                    f:generateName: {}
                  f:spec:
                    f:containers:
                      k:{"name":"write-pod"}:
                        .: {}
                        f:command: {}
                        f:image: {}
                        f:imagePullPolicy: {}
                        f:name: {}
                        f:resources: {}
                        f:securityContext:
                          .: {}
                          f:allowPrivilegeEscalation: {}
                          f:capabilities:
                            .: {}
                            f:drop: {}
                          f:runAsGroup: {}
                          f:runAsNonRoot: {}
                          f:runAsUser: {}
                        f:terminationMessagePath: {}
                        f:terminationMessagePolicy: {}
                        f:volumeMounts:
                          .: {}
                          k:{"mountPath":"/mnt/volume1"}:
                            .: {}
                            f:mountPath: {}
                            f:name: {}
                    f:dnsPolicy: {}
                    f:enableServiceLinks: {}
                    f:restartPolicy: {}
                    f:schedulerName: {}
                    f:securityContext:
                      .: {}
                      f:fsGroup: {}
                      f:runAsGroup: {}
                      f:runAsNonRoot: {}
                      f:runAsUser: {}
                      f:seccompProfile:
                        .: {}
                        f:type: {}
                    f:terminationGracePeriodSeconds: {}
                    f:volumes:
                      .: {}
                      k:{"name":"volume1"}:
                        .: {}
                        f:name: {}
                        f:persistentVolumeClaim:
                          .: {}
                          f:claimName: {}
                manager: e2e.test
                operation: Update
                time: "2026-02-16T10:40:48Z"
              - apiVersion: v1
                fieldsType: FieldsV1
                fieldsV1:
                  f:status:
                    f:conditions:
                      k:{"type":"ContainersReady"}:
                        .: {}
                        f:lastProbeTime: {}
                        f:lastTransitionTime: {}
                        f:message: {}
                        f:reason: {}
                        f:status: {}
                        f:type: {}
                      k:{"type":"Initialized"}:
                        .: {}
                        f:lastProbeTime: {}
                        f:lastTransitionTime: {}
                        f:status: {}
                        f:type: {}
                      k:{"type":"PodReadyToStartContainers"}:
                        .: {}
                        f:lastProbeTime: {}
                        f:lastTransitionTime: {}
                        f:status: {}
                        f:type: {}
                      k:{"type":"Ready"}:
                        .: {}
                        f:lastProbeTime: {}
                        f:lastTransitionTime: {}
                        f:message: {}
                        f:reason: {}
                        f:status: {}
                        f:type: {}
                    f:containerStatuses: {}
                    f:hostIP: {}
                    f:hostIPs: {}
                    f:startTime: {}
                manager: kubelet
                operation: Update
                subresource: status
                time: "2026-02-16T10:40:48Z"
              name: fsgroup-share-1-b698e426
              namespace: mounterpod-7653
              resourceVersion: "6112"
              uid: e2abf29a-5333-4a07-bf52-e6ba9aeb8662
            spec:
              containers:
              - command:
                - /bin/sh
                - -c
                - trap exit TERM; while true; do sleep 1; done
                image: registry.k8s.io/e2e-test-images/busybox:1.36.1-1
                imagePullPolicy: IfNotPresent
                name: write-pod
                resources: {}
                securityContext:
                  allowPrivilegeEscalation: false
                  capabilities:
                    drop:
                    - ALL
                  runAsGroup: 2000
                  runAsNonRoot: true
                  runAsUser: 1001
                terminationMessagePath: /dev/termination-log
                terminationMessagePolicy: File
                volumeMounts:
                - mountPath: /mnt/volume1
                  name: volume1
                - mountPath: .../run/secrets/kubernetes.io/serviceaccount
                  name: kube-api-access-zs69z
                  readOnly: true
              dnsPolicy: ClusterFirst
              enableServiceLinks: true
              nodeName: helm-test-cluster-control-plane
              preemptionPolicy: PreemptLowerPriority
              priority: 0
              restartPolicy: OnFailure
              schedulerName: default-scheduler
              securityContext:
                fsGroup: 1001
                runAsGroup: 2000
                runAsNonRoot: true
                runAsUser: 1001
                seccompProfile:
                  type: RuntimeDefault
              serviceAccount: default
              serviceAccountName: default
              terminationGracePeriodSeconds: 30
              tolerations:
              - effect: NoExecute
                key: node.kubernetes.io/not-ready
                operator: Exists
                tolerationSeconds: 300
              - effect: NoExecute
                key: node.kubernetes.io/unreachable
                operator: Exists
                tolerationSeconds: 300
              volumes:
              - name: volume1
                persistentVolumeClaim:
                  claimName: s3-e2e-pvc-42be2a93-2586-43df-8dbc-3ecd367883c8
              - name: kube-api-access-zs69z
                projected:
                  defaultMode: 420
                  sources:
                  - serviceAccountToken:
                      expirationSeconds: 3607
                      path: token
                  - configMap:
                      items:
                      - key: ca.crt
                        path: ca.crt
                      name: kube-root-ca.crt
                  - downwardAPI:
                      items:
                      - fieldRef:
                          apiVersion: v1
                          fieldPath: metadata.namespace
                        path: namespace
            status:
              conditions:
              - lastProbeTime: null
                lastTransitionTime: "2026-02-16T10:40:48Z"
                status: "False"
                type: PodReadyToStartContainers
              - lastProbeTime: null
                lastTransitionTime: "2026-02-16T10:40:48Z"
                status: "True"
                type: Initialized
              - lastProbeTime: null
                lastTransitionTime: "2026-02-16T10:40:48Z"
                message: 'containers with unready status: [write-pod]'
                reason: ContainersNotReady
                status: "False"
                type: Ready
              - lastProbeTime: null
                lastTransitionTime: "2026-02-16T10:40:48Z"
                message: 'containers with unready status: [write-pod]'
                reason: ContainersNotReady
                status: "False"
                type: ContainersReady
              - lastProbeTime: null
                lastTransitionTime: "2026-02-16T10:40:48Z"
                status: "True"
                type: PodScheduled
              containerStatuses:
              - image: registry.k8s.io/e2e-test-images/busybox:1.36.1-1
                imageID: ""
                lastState: {}
                name: write-pod
                ready: false
                restartCount: 0
                started: false
                state:
                  waiting:
                    reason: ContainerCreating
              hostIP: 172.18.0.2
              hostIPs:
              - ip: 172.18.0.2
              phase: Pending
              qosClass: BestEffort
              startTime: "2026-02-16T10:40:48Z"
    {
        msg: "pod \"fsgroup-share-1-b698e426\" not Running: Timed out after 300.001s.\nExpected Pod to be in <v1.PodPhase>: \"Running\"\nGot instead:\n    <*v1.Pod | 0xc000245b08>: \n        metadata:\n          creationTimestamp: \"2026-02-16T10:40:48Z\"\n          generateName: pvc-tester-\n          managedFields:\n          - apiVersion: v1\n            fieldsType: FieldsV1\n            fieldsV1:\n              f:metadata:\n                f:generateName: {}\n              f:spec:\n                f:containers:\n                  k:{\"name\":\"write-pod\"}:\n                    .: {}\n                    f:command: {}\n                    f:image: {}\n                    f:imagePullPolicy: {}\n                    f:name: {}\n                    f:resources: {}\n                    f:securityContext:\n                      .: {}\n                      f:allowPrivilegeEscalation: {}\n                      f:capabilities:\n                        .: {}\n                        f:drop: {}\n                      f:runAsGroup: {}\n                      f:runAsNonRoot: {}\n                      f:runAsUser: {}\n                    f:terminationMessagePath: {}\n                    f:terminationMessagePolicy: {}\n                    f:volumeMounts:\n                      .: {}\n                      k:{\"mountPath\":\"/mnt/volume1\"}:\n                        .: {}\n                        f:mountPath: {}\n                        f:name: {}\n                f:dnsPolicy: {}\n                f:enableServiceLinks: {}\n                f:restartPolicy: {}\n                f:schedulerName: {}\n                f:securityContext:\n                  .: {}\n                  f:fsGroup: {}\n                  f:runAsGroup: {}\n                  f:runAsNonRoot: {}\n                  f:runAsUser: {}\n                  f:seccompProfile:\n                    .: {}\n                    f:type: {}\n                f:terminationGracePeriodSeconds: {}\n                f:volumes:\n                  .: {}\n                  k:{\"name\":\"volume1\"}:\n                    .: {}\n                    f:name: {}\n                    f:persistentVolumeClaim:\n                      .: {}\n                      f:claimName: {}\n            manager: e2e.test\n            operation: Update\n            time: \"2026-02-16T10:40:48Z\"\n          - apiVersion: v1\n            fieldsType: FieldsV1\n            fieldsV1:\n              f:status:\n                f:conditions:\n                  k:{\"type\":\"ContainersReady\"}:\n                    .: {}\n                    f:lastProbeTime: {}\n                    f:lastTransitionTime: {}\n                    f:message: {}\n                    f:reason: {}\n                    f:status: {}\n                    f:type: {}\n                  k:{\"type\":\"Initialized\"}:\n                    .: {}\n                    f:lastProbeTime: {}\n                    f:lastTransitionTime: {}\n                    f:status: {}\n                    f:type: {}\n                  k:{\"type\":\"PodReadyToStartContainers\"}:\n                    .: {}\n                    f:lastProbeTime: {}\n                    f:lastTransitionTime: {}\n                    f:status: {}\n                    f:type: {}\n                  k:{\"type\":\"Ready\"}:\n                    .: {}\n                    f:lastProbeTime: {}\n                    f:lastTransitionTime: {}\n                    f:message: {}\n                    f:reason: {}\n                    f:status: {}\n                    f:type: {}\n                f:containerStatuses: {}\n                f:hostIP: {}\n                f:hostIPs: {}\n                f:startTime: {}\n            manager: kubelet\n            operation: Update\n            subresource: status\n            time: \"2026-02-16T10:40:48Z\"\n          name: fsgroup-share-1-b698e426\n          namespace: mounterpod-7653\n          resourceVersion: \"6112\"\n          uid: e2abf29a-5333-4a07-bf52-e6ba9aeb8662\n        spec:\n          containers:\n          - command:\n            - /bin/sh\n            - -c\n            - trap exit TERM; while true; do sleep 1; done\n            image: registry.k8s.io/e2e-test-images/busybox:1.36.1-1\n            imagePullPolicy: IfNotPresent\n            name: write-pod\n            resources: {}\n            securityContext:\n              allowPrivilegeEscalation: false\n              capabilities:\n                drop:\n                - ALL\n              runAsGroup: 2000\n              runAsNonRoot: true\n              runAsUser: 1001\n            terminationMessagePath: /dev/termination-log\n            terminationMessagePolicy: File\n            volumeMounts:\n            - mountPath: /mnt/volume1\n              name: volume1\n            - mountPath: .../run/secrets/kubernetes.io/serviceaccount\n              name: kube-api-access-zs69z\n              readOnly: true\n          dnsPolicy: ClusterFirst\n          enableServiceLinks: true\n          nodeName: helm-test-cluster-control-plane\n          preemptionPolicy: PreemptLowerPriority\n          priority: 0\n          restartPolicy: OnFailure\n          schedulerName: default-scheduler\n          securityContext:\n            fsGroup: 1001\n            runAsGroup: 2000\n            runAsNonRoot: true\n            runAsUser: 1001\n            seccompProfile:\n              type: RuntimeDefault\n          serviceAccount: default\n          serviceAccountName: default\n          terminationGracePeriodSeconds: 30\n          tolerations:\n          - effect: NoExecute\n            key: node.kubernetes.io/not-ready\n            operator: Exists\n            tolerationSeconds: 300\n          - effect: NoExecute\n            key: node.kubernetes.io/unreachable\n            operator: Exists\n            tolerationSeconds: 300\n          volumes:\n          - name: volume1\n            persistentVolumeClaim:\n              claimName: s3-e2e-pvc-42be2a93-2586-43df-8dbc-3ecd367883c8\n          - name: kube-api-access-zs69z\n            projected:\n              defaultMode: 420\n              sources:\n              - serviceAccountToken:\n                  expirationSeconds: 3607\n                  path: token\n              - configMap:\n                  items:\n                  - key: ca.crt\n                    path: ca.crt\n                  name: kube-root-ca.crt\n              - downwardAPI:\n                  items:\n                  - fieldRef:\n                      apiVersion: v1\n                      fieldPath: metadata.namespace\n                    path: namespace\n        status:\n          conditions:\n          - lastProbeTime: null\n            lastTransitionTime: \"2026-02-16T10:40:48Z\"\n            status: \"False\"\n            type: PodReadyToStartContainers\n          - lastProbeTime: null\n            lastTransitionTime: \"2026-02-16T10:40:48Z\"\n            status: \"True\"\n            type: Initialized\n          - lastProbeTime: null\n            lastTransitionTime: \"2026-02-16T10:40:48Z\"\n            message: 'containers with unready status: [write-pod]'\n            reason: ContainersNotReady\n            status: \"False\"\n            type: Ready\n          - lastProbeTime: null\n            lastTransitionTime: \"2026-02-16T10:40:48Z\"\n            message: 'containers with unready status: [write-pod]'\n            reason: ContainersNotReady\n            status: \"False\"\n            type: ContainersReady\n          - lastProbeTime: null\n            lastTransitionTime: \"2026-02-16T10:40:48Z\"\n            status: \"True\"\n            type: PodScheduled\n          containerStatuses:\n          - image: registry.k8s.io/e2e-test-images/busybox:1.36.1-1\n            imageID: \"\"\n            lastState: {}\n            name: write-pod\n            ready: false\n            restartCount: 0\n            started: false\n            state:\n              waiting:\n                reason: ContainerCreating\n          hostIP: 172.18.0.2\n          hostIPs:\n          - ip: 172.18.0.2\n          phase: Pending\n          qosClass: BestEffort\n          startTime: \"2026-02-16T10:40:48Z\"",
        err: <framework.FailureError>{
            msg: "Timed out after 300.001s.\nExpected Pod to be in <v1.PodPhase>: \"Running\"\nGot instead:\n    <*v1.Pod | 0xc000245b08>: \n        metadata:\n          creationTimestamp: \"2026-02-16T10:40:48Z\"\n          generateName: pvc-tester-\n          managedFields:\n          - apiVersion: v1\n            fieldsType: FieldsV1\n            fieldsV1:\n              f:metadata:\n                f:generateName: {}\n              f:spec:\n                f:containers:\n                  k:{\"name\":\"write-pod\"}:\n                    .: {}\n                    f:command: {}\n                    f:image: {}\n                    f:imagePullPolicy: {}\n                    f:name: {}\n                    f:resources: {}\n                    f:securityContext:\n                      .: {}\n                      f:allowPrivilegeEscalation: {}\n                      f:capabilities:\n                        .: {}\n                        f:drop: {}\n                      f:runAsGroup: {}\n                      f:runAsNonRoot: {}\n                      f:runAsUser: {}\n                    f:terminationMessagePath: {}\n                    f:terminationMessagePolicy: {}\n                    f:volumeMounts:\n                      .: {}\n                      k:{\"mountPath\":\"/mnt/volume1\"}:\n                        .: {}\n                        f:mountPath: {}\n                        f:name: {}\n                f:dnsPolicy: {}\n                f:enableServiceLinks: {}\n                f:restartPolicy: {}\n                f:schedulerName: {}\n                f:securityContext:\n                  .: {}\n                  f:fsGroup: {}\n                  f:runAsGroup: {}\n                  f:runAsNonRoot: {}\n                  f:runAsUser: {}\n                  f:seccompProfile:\n                    .: {}\n                    f:type: {}\n                f:terminationGracePeriodSeconds: {}\n                f:volumes:\n                  .: {}\n                  k:{\"name\":\"volume1\"}:\n                    .: {}\n                    f:name: {}\n                    f:persistentVolumeClaim:\n                      .: {}\n                      f:claimName: {}\n            manager: e2e.test\n            operation: Update\n            time: \"2026-02-16T10:40:48Z\"\n          - apiVersion: v1\n            fieldsType: FieldsV1\n            fieldsV1:\n              f:status:\n                f:conditions:\n                  k:{\"type\":\"ContainersReady\"}:\n                    .: {}\n                    f:lastProbeTime: {}\n                    f:lastTransitionTime: {}\n                    f:message: {}\n                    f:reason: {}\n                    f:status: {}\n                    f:type: {}\n                  k:{\"type\":\"Initialized\"}:\n                    .: {}\n                    f:lastProbeTime: {}\n                    f:lastTransitionTime: {}\n                    f:status: {}\n                    f:type: {}\n                  k:{\"type\":\"PodReadyToStartContainers\"}:\n                    .: {}\n                    f:lastProbeTime: {}\n                    f:lastTransitionTime: {}\n                    f:status: {}\n                    f:type: {}\n                  k:{\"type\":\"Ready\"}:\n                    .: {}\n                    f:lastProbeTime: {}\n                    f:lastTransitionTime: {}\n                    f:message: {}\n                    f:reason: {}\n                    f:status: {}\n                    f:type: {}\n                f:containerStatuses: {}\n                f:hostIP: {}\n                f:hostIPs: {}\n                f:startTime: {}\n            manager: kubelet\n            operation: Update\n            subresource: status\n            time: \"2026-02-16T10:40:48Z\"\n          name: fsgroup-share-1-b698e426\n          namespace: mounterpod-7653\n          resourceVersion: \"6112\"\n          uid: e2abf29a-5333-4a07-bf52-e6ba9aeb8662\n        spec:\n          containers:\n          - command:\n            - /bin/sh\n            - -c\n            - trap exit TERM; while true; do sleep 1; done\n            image: registry.k8s.io/e2e-test-images/busybox:1.36.1-1\n            imagePullPolicy: IfNotPresent\n            name: write-pod\n            resources: {}\n            securityContext:\n              allowPrivilegeEscalation: false\n              capabilities:\n                drop:\n                - ALL\n              runAsGroup: 2000\n              runAsNonRoot: true\n              runAsUser: 1001\n            terminationMessagePath: /dev/termination-log\n            terminationMessagePolicy: File\n            volumeMounts:\n            - mountPath: /mnt/volume1\n              name: volume1\n            - mountPath: .../run/secrets/kubernetes.io/serviceaccount\n              name: kube-api-access-zs69z\n              readOnly: true\n          dnsPolicy: ClusterFirst\n          enableServiceLinks: true\n          nodeName: helm-test-cluster-control-plane\n          preemptionPolicy: PreemptLowerPriority\n          priority: 0\n          restartPolicy: OnFailure\n          schedulerName: default-scheduler\n          securityContext:\n            fsGroup: 1001\n            runAsGroup: 2000\n            runAsNonRoot: true\n            runAsUser: 1001\n            seccompProfile:\n              type: RuntimeDefault\n          serviceAccount: default\n          serviceAccountName: default\n          terminationGracePeriodSeconds: 30\n          tolerations:\n          - effect: NoExecute\n            key: node.kubernetes.io/not-ready\n            operator: Exists\n            tolerationSeconds: 300\n          - effect: NoExecute\n            key: node.kubernetes.io/unreachable\n            operator: Exists\n            tolerationSeconds: 300\n          volumes:\n          - name: volume1\n            persistentVolumeClaim:\n              claimName: s3-e2e-pvc-42be2a93-2586-43df-8dbc-3ecd367883c8\n          - name: kube-api-access-zs69z\n            projected:\n              defaultMode: 420\n              sources:\n              - serviceAccountToken:\n                  expirationSeconds: 3607\n                  path: token\n              - configMap:\n                  items:\n                  - key: ca.crt\n                    path: ca.crt\n                  name: kube-root-ca.crt\n              - downwardAPI:\n                  items:\n                  - fieldRef:\n                      apiVersion: v1\n                      fieldPath: metadata.namespace\n                    path: namespace\n        status:\n          conditions:\n          - lastProbeTime: null\n            lastTransitionTime: \"2026-02-16T10:40:48Z\"\n            status: \"False\"\n            type: PodReadyToStartContainers\n          - lastProbeTime: null\n            lastTransitionTime: \"2026-02-16T10:40:48Z\"\n            status: \"True\"\n            type: Initialized\n          - lastProbeTime: null\n            lastTransitionTime: \"2026-02-16T10:40:48Z\"\n            message: 'containers with unready status: [write-pod]'\n            reason: ContainersNotReady\n            status: \"False\"\n            type: Ready\n          - lastProbeTime: null\n            lastTransitionTime: \"2026-02-16T10:40:48Z\"\n            message: 'containers with unready status: [write-pod]'\n            reason: ContainersNotReady\n            status: \"False\"\n            type: ContainersReady\n          - lastProbeTime: null\n            lastTransitionTime: \"2026-02-16T10:40:48Z\"\n            status: \"True\"\n            type: PodScheduled\n          containerStatuses:\n          - image: registry.k8s.io/e2e-test-images/busybox:1.36.1-1\n            imageID: \"\"\n            lastState: {}\n            name: write-pod\n            ready: false\n            restartCount: 0\n            started: false\n            state:\n              waiting:\n                reason: ContainerCreating\n          hostIP: 172.18.0.2\n          hostIPs:\n          - ip: 172.18.0.2\n          phase: Pending\n          qosClass: BestEffort\n          startTime: \"2026-02-16T10:40:48Z\"",
            fullStackTrace: "k8s..../e2e/framework/pod.WaitTimeoutForPodRunningInNamespace({0x7f76d0e8d1d8, 0xc0008c0d20}, {0x2e74af0?, 0xc00065fa00?}, {0xc000429428, 0x18}, {0xc0002ff3b0, 0xf}, 0x45d964b800)\n\.../pkg/mod/k8s.io/kubernetes@v1.29..../framework/pod/wait.go:459 +0x529\nk8s..../e2e/framework/pod.WaitForPodNameRunningInNamespace(...)\n\.../pkg/mod/k8s.io/kubernetes@v1.29..../framework/pod/wait.go:443\ngithub..../tests/e2e/customsuites.createPod({0x7f76d0e8d1d8, 0xc0008c0d20}, {0x2e74af0, 0xc00065fa00}, {0xc0002ff3b0, 0xf}, 0xc000777688)\n\.../e2e/customsuites/util.go:218 +0x205\ngithub..../tests/e2e/customsuites.(*s3CSIMounterPodTestSuite).DefineTests.func6({0x7f76d0e8d1d8, 0xc0008c0d20})\n\.../e2e/customsuites/mounterpod.go:428 +0x3cb",
        },
    }
occurred
In [It] at: .../e2e/customsuites/mounterpod.go:429 @ 02/16/26 10:45:48.217

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

@anurag4DSB anurag4DSB marked this pull request as draft February 16, 2026 08:05
@anurag4DSB
S3CSI-213: Add Pod Security and FSGroup documentation
82d52ee 
Add two E2E tests to the mounter pod suite:
- Dynamic provisioning with fsGroup on workload pod
- Separate mounter pods for different fsGroup values on shared volume

Add a reference page explaining how Kubernetes fsGroup interacts
with the CSI driver's pod mounter architecture, including static
and dynamic provisioning examples and the relationship between
fsGroup and mount options (uid/gid).
@anurag4DSB anurag4DSB force-pushed the bugfix/S3CSI-213-fsgroup-mounter-pods branch from a60b7ab to 82d52ee Compare February 16, 2026 08:08
@anurag4DSB
S3CSI-213: Resolve fsGroup from pod spec for RWX volumes
1d8402c 
The reconciler reads fsGroup from pod.Spec.SecurityContext.FSGroup
to build S3PA field filters, but the node driver reads from
VolumeMountGroup in the CSI request. For RWX volumes with the
default fsGroupPolicy (ReadWriteOnceWithFSType), kubelet does not
populate VolumeMountGroup, causing an S3PA lookup mismatch and
mount timeout.

Fix: when VolumeMountGroup is empty, the pod mounter resolves
fsGroup directly from the workload pod's SecurityContext via the
k8s API — the same authoritative source the reconciler uses.
This also applies the matching mount options (--gid, --allow-other,
--dir-mode, --file-mode) for proper file access.

Issue: S3CSI-213
@anurag4DSB
S3CSI-213: Make fsGroup override --gid mount option
5224fad 
Pod-level fsGroup always takes precedence over --gid in PV/SC
mount options. This ensures the workload pod's security intent
is honored when both are set.

Issue: S3CSI-213
@anurag4DSB
S3CSI-213: Add E2E tests for fsGroup auto-apply and override
86c5032 
Add three new E2E tests covering fsGroup mount option behaviors:
- Dynamic provisioning volume sharing with different fsGroup values
- FSGroup auto-applies --gid, --dir-mode, --file-mode when absent
- FSGroup overrides --gid from PV mount options

Update docs with automatic mount option resolution details and
simplified configuration examples.
@anurag4DSB
S3CSI-213: Disable Gomega output truncation in E2E tests
b02bd26 
Set format.MaxLength to 0 so test failures print full error details
instead of truncated representations.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@anurag4DSB