Kubernetes iSCSI ReadWriteMany - RWX not working as intended?

[haroonb]

Hello,

the company I work for bought a Synology SAN to be used as a CSI for our Kubernetes clusters. We need support for ReadWriteMany volumes. I couldn't find anything on the Synology documentation website, if It is officially supported. During my evaluation of the Synology SAN, I however encountered some mixed results. It's seems to be not working as intended.

I created 3 pods with the same RWX pvc and created an individual file for each pod, on the shared location.

kubectl get po -owide
NAME            READY   STATUS    RESTARTS   AGE   IP            NODE        NOMINATED NODE   READINESS GATES
busybox-pod-1   1/1     Running   0          23m   10.244.1.72   devops-w1   <none>           <none>
busybox-pod-2   1/1     Running   0          23m   10.244.2.64   devops-w2   <none>           <none>
busybox-pod-3   1/1     Running   0          23m   10.244.2.65   devops-w2   <none>           <none>

for i in {1..3} ; do kubectl exec -it busybox-pod-"${i}" -- ls /mnt/shared/ ; done
busybox-pod-1.file  lost+found
busybox-pod-2.file  busybox-pod-3.file  lost+found
busybox-pod-2.file  busybox-pod-3.file  lost+found

Expected behaviour:
Every file should be visible in each pod.

Bug:
Only files on the same host are visible.

Replication of the bug:

helm install --atomic --values synology_iscsi_values.yaml --namespace synology --create-namespace synology-scsi synology-csi-chart/synology-csi

---
clientInfoSecret:
  clients:
    - host: xxxxx
      https: false
      password: xxxxx
      port: 5000
      username: k8s-iscsi
    - host: xxxxx
      https: true
      password: xxxxx
      port: 5001
      username: k8s-iscsi
  create: true
  name: "client-info-secret"
installCSIDriver: true
storageClasses:
  synology-iscsi-storage:
    disabled: false
    reclaimPolicy: Delete
    volumeBindingMode: Immediate
    parameters:
      dsm: xxxxx
      fsType: ext4
      location: /volume1
volumeSnapshotClasses:
  synology-snapshotclass:
    deletionPolicy: Delete

Then create 3 busybox pods using the same ReadWriteMany PersistentVolumeClaim.

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: shared-rwx-pvc
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: synology-scsi-synology-csi-synology-iscsi-storage
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: Pod
metadata:
  name: busybox-pod-1
  labels:
    app: busybox
spec:
  containers:
  - name: busybox
    image: busybox
    command:
      - sh
      - -c
      - |
        echo "Hello from busybox-pod-1" > /mnt/shared/busybox-pod-1.file
        sleep 3600
    volumeMounts:
    - mountPath: "/mnt/shared"
      name: shared-storage
  volumes:
  - name: shared-storage
    persistentVolumeClaim:
      claimName: shared-rwx-pvc
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector:
            matchExpressions:
            - key: app
              operator: In
              values:
              - busybox
          topologyKey: kubernetes.io/hostname
---
apiVersion: v1
kind: Pod
metadata:
  name: busybox-pod-2
  labels:
    app: busybox
spec:
  containers:
  - name: busybox
    image: busybox
    command:
      - sh
      - -c
      - |
        echo "Hello from busybox-pod-2" > /mnt/shared/busybox-pod-2.file
        sleep 3600
    volumeMounts:
    - mountPath: "/mnt/shared"
      name: shared-storage
  volumes:
  - name: shared-storage
    persistentVolumeClaim:
      claimName: shared-rwx-pvc
  affinity:
    podAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector:
            matchExpressions:
            - key: app
              operator: In
              values:
              - busybox
          topologyKey: kubernetes.io/hostname
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector:
            matchExpressions:
            - key: app
              operator: In
              values:
              - busybox
          topologyKey: kubernetes.io/hostname
---
apiVersion: v1
kind: Pod
metadata:
  name: busybox-pod-3
  labels:
    app: busybox
spec:
  containers:
  - name: busybox
    image: busybox
    command:
      - sh
      - -c
      - |
        echo "Hello from busybox-pod-3" > /mnt/shared/busybox-pod-3.file
        sleep 3600
    volumeMounts:
    - mountPath: "/mnt/shared"
      name: shared-storage
  volumes:
  - name: shared-storage
    persistentVolumeClaim:
      claimName: shared-rwx-pvc
  affinity:
    podAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector:
            matchExpressions:
            - key: app
              operator: In
              values:
              - busybox
          topologyKey: kubernetes.io/hostname

Thanks for any help or suggestions you can offer.

Kind regards.