What happened:
Connecting to a windows file share using SMB3 from Linux AKS and works great initially. Have 3 nodes with multiple pods in each that all mount to the same windows file share. On a few occasions after it has been up for a few days and a pod has been deleted through the kubernetes dashboard, all pods running on that node are no longer able to connect to the windows file share. Existing pods that have not be restarted get this error when trying to navigate to the mounted share:
mount error(112): Host is down
New pods trying to start on the node get this error:
MountVolume.SetUp failed for volume "app-smb-share-pv" : mount command failed, status: Failure, reason: Failed to mount device at /var/lib/kubelet/pods/4a20d527-116e-4aca-a2f1-59e0c9319a89/volumes/microsoft.com~smb/app-smb-share-pv, user:XXXX, //WINSERVER/file-share, error log:Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)
Nothing different in the smb-driver.log file
In the warn log, get these:
Mar 30 12:46:18 aks-nodepool1-XXX-vmss00000H kernel: [328413.074290] CIFS VFS: cifs_mount failed w/return code = -112
Mar 30 12:46:18 aks-nodepool1-XXX-vmss00000H kernel: [328413.081632] CIFS VFS: ioctl error in smb2_get_dfs_refer rc=-5
Can use smbclient to mount to the same windows file share without a problem so know it is not a fundamental networking issue. If I restart the node from the Azure portal, then pods in node can mount fine.
YAML files below.
Seems like something isn't gracefully happening sometimes when pod is deleted. Am I doing something wrong?
What you expected to happen:
Deleting a pod does not cause the mount for the entire node to become unusable.
How to reproduce it:
Start of AKS cluster that has a mount to a SMB 3 windows file share, unceremoniously delete a pod (cancel a helm deployment, delete pod using kubernetes dashboard) and other pods and any new pods on the node can no longer communicate with the Windows File Share.
Anything else we need to know?:
Here is the PersistentVolume YAML being used:
apiVersion: v1
kind: PersistentVolume
metadata:
name: app-smb-share-pv
spec:
capacity:
storage: 100Gi
accessModes:
- ReadWriteMany
storageClassName: appshare
persistentVolumeReclaimPolicy: Retain
flexVolume:
driver: "microsoft.com/smb"
secretRef:
name: smbcreds
namespace: default
options:
source: "//<server>/file-share"
mountoptions: "vers=3.0,dir_mode=0777,file_mode=0777,sec=ntlmv2,domain=MYDOMAIN"
Volume mount in each pod:
- name: appshare-volume
mountPath: /mnt/app
Volume in each pod:
- name: appshare-volume
persistentVolumeClaim:
claimName: app-share-pvc
Environment:
What happened:
Connecting to a windows file share using SMB3 from Linux AKS and works great initially. Have 3 nodes with multiple pods in each that all mount to the same windows file share. On a few occasions after it has been up for a few days and a pod has been deleted through the kubernetes dashboard, all pods running on that node are no longer able to connect to the windows file share. Existing pods that have not be restarted get this error when trying to navigate to the mounted share:
mount error(112): Host is down
New pods trying to start on the node get this error:
Nothing different in the smb-driver.log file
In the warn log, get these:
Can use smbclient to mount to the same windows file share without a problem so know it is not a fundamental networking issue. If I restart the node from the Azure portal, then pods in node can mount fine.
YAML files below.
Seems like something isn't gracefully happening sometimes when pod is deleted. Am I doing something wrong?
What you expected to happen:
Deleting a pod does not cause the mount for the entire node to become unusable.
How to reproduce it:
Start of AKS cluster that has a mount to a SMB 3 windows file share, unceremoniously delete a pod (cancel a helm deployment, delete pod using kubernetes dashboard) and other pods and any new pods on the node can no longer communicate with the Windows File Share.
Anything else we need to know?:
Here is the PersistentVolume YAML being used:
Volume mount in each pod:
Volume in each pod:
Environment:
kubectl version): 1.15.7uname -a): Linux app-back-office-deployment-7fd86c5bc7-v5klp 4.15.0-1069-azure smb flex volume plugin volumes timeout when fsGroup is included #74-Ubuntu SMP Fri Feb 7 17:22:24 UTC 2020 x86_64 GNU/Linux