RemoveSmbGlobalMapping makes Windows node very slow #1925
Description
What happened:
Do to the RemoveSmbGlobalMapping cleaning up a lot of mounts at the same time (see below) . Either the POD gets overloaded or the storage account hits some sort off rate limiter because we're seeing high latency errors. We tried validating this on the storage account but all seems good here.
Windows nodes can no longer create mounts and the node becomes unusable and needs to be cordoned until the process is done unmounting. No new mounts can be created during this process. We expect this to be the cause.
How to reproduce it:
Hard to say. We expect its a load issue where we create/delete volumes in a very short time but we cannot reliable reproduce this. We then hit high latency. At times we see 10sec latency 30sec 160 210 sec's multible times and nodes become unusable.
Anything else we need to know?:
This seems to be introduced by #1847
We're also running a case at Microsoft support but that is slow going. Ticket start 07 juni.
Expected start time off the issue 7 juni 2024 20:00 utc +. seems to correlate with the new CSI version.
Storage class:
`spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 30Gi
storageClassName: azurefile-csi
Storageclass settings are:
provisioner: file.csi.azure.com
parameters:
skuName: Standard_LRS
reclaimPolicy: Delete
mountOptions:
- mfsymlinks
- actimeo=30
allowVolumeExpansion: true
volumeBindingMode: Immediate`
Environment:
- CSI Driver version: v1.30.2 (mcr.microsoft.com/oss/kubernetes-csi/azurefile-csi:v1.30.2-windows-hp)
- Kubernetes version: Azure Kubernetes Services 1.29.4
Logs from (kubectl cluster-info dump > ouput.txt)
`I0618 10:20:01.948928 10816 smb.go:62] begin to run RemoveSmbGlobalMapping with \.file.core.windows.net\pvc-d54bf4ed-647d-480a-8de5-3f647ca1126f
I0618 10:20:02.184149 10816 smb.go:97] checking remote server path UNC<redacted>.file.core.windows.net\pvc-e1118b6f-8b8e-4fbe-b680-0671072fbf2e on local path C:\var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\f9d0b87889d2774ec5e434e4f21da3a34b60667e8473ff91641328c209ba2c8f\globalmount
I0618 10:20:02.184149 10816 smb.go:62] begin to run RemoveSmbGlobalMapping with \.file.core.windows.net\pvc-ff818ba6-ec9f-4486-80e4-7cc61350f6d5
I0618 10:20:02.422931 10816 smb.go:97] checking remote server path UNC<redacted>.file.core.windows.net\pvc-e1118b6f-8b8e-4fbe-b680-0671072fbf2e on local path C:\var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\f9d0b87889d2774ec5e434e4f21da3a34b60667e8473ff91641328c209ba2c8f\globalmount
I0618 10:20:02.422931 10816 smb.go:62] begin to run RemoveSmbGlobalMapping with \.file.core.windows.net\pvc-b034ea6e-80a0-4b98-a036-a7557c1d2595
I0618 10:20:02.697686 10816 smb.go:97] checking remote server path UNC<redacted>.file.core.windows.net\pvc-e1118b6f-8b8e-4fbe-b680-0671072fbf2e on local path C:\var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\f9d0b87889d2774ec5e434e4f21da3a34b60667e8473ff91641328c209ba2c8f\globalmount
I0618 10:20:02.697686 10816 smb.go:62] begin to run RemoveSmbGlobalMapping with \.file.core.windows.net\pvc-f9f952d8-fd7b-423b-8384-3230cbbef2d2
I0618 10:20:03.701545 10816 safe_mounter_host_process_windows.go:153] Unmount: remote path: \.file.core.windows.net\pvc-73aa6b77-becf-49d0-bfcc-203c68b89695 local path: c:\var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\68be1105184c280e13b3b149afbfbe50dfe12a8428a23aabbccdb60143461c61\globalmount
I0618 10:20:03.702218 10816 nodeserver.go:438] NodeUnstageVolume: unmount volume ##pvc-73aa6b77-becf-49d0-bfcc-203c68b89695### on \var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\68be1105184c280e13b3b149afbfbe50dfe12a8428a23aabbccdb60143461c61\globalmount successfully
I0618 10:20:03.702218 10816 azure_metrics.go:118] "Observed Request Latency" latency_seconds=199.0857827 request="azurefile_csi_driver_node_unstage_volume" resource_group="" subscription_id="" source="file.csi.azure.com" volumeid="##pvc-73aa6b77-becf-49d0-bfcc-203c68b89695###" result_code="succeeded"
I0618 10:20:03.702218 10816 utils.go:108] GRPC response: {}
I0618 10:20:03.989377 10816 smb.go:97] checking remote server path UNC<redacted>.file.core.windows.net\pvc-e1118b6f-8b8e-4fbe-b680-0671072fbf2e on local path C:\var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\f9d0b87889d2774ec5e434e4f21da3a34b60667e8473ff91641328c209ba2c8f\globalmount
I0618 10:20:03.989377 10816 smb.go:62] begin to run RemoveSmbGlobalMapping with \.file.core.windows.net\pvc-373137fb-16fa-4445-9526-fa69d783ec00
I0618 10:20:04.286036 10816 smb.go:97] checking remote server path UNC<redacted>.file.core.windows.net\pvc-e1118b6f-8b8e-4fbe-b680-0671072fbf2e on local path C:\var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\f9d0b87889d2774ec5e434e4f21da3a34b60667e8473ff91641328c209ba2c8f\globalmount
I0618 10:20:04.286036 10816 smb.go:62] begin to run RemoveSmbGlobalMapping with \.file.core.windows.net\pvc-e465606b-ee96-42f4-9108-266751f62cc0
I0618 10:20:04.533291 10816 smb.go:97] checking remote server path UNC<redacted>.file.core.windows.net\pvc-e1118b6f-8b8e-4fbe-b680-0671072fbf2e on local path C:\var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\f9d0b87889d2774ec5e434e4f21da3a34b60667e8473ff91641328c209ba2c8f\globalmount
I0618 10:20:04.533291 10816 smb.go:62] begin to run RemoveSmbGlobalMapping with \.file.core.windows.net\pvc-9cc1f701-08df-45fa-9a63-6316ce0f4521
I0618 10:20:04.770168 10816 smb.go:97] checking remote server path UNC<redacted>.file.core.windows.net\pvc-e1118b6f-8b8e-4fbe-b680-0671072fbf2e on local path C:\var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\f9d0b87889d2774ec5e434e4f21da3a34b60667e8473ff91641328c209ba2c8f\globalmount
I0618 10:20:04.770198 10816 smb.go:62] begin to run RemoveSmbGlobalMapping with \.file.core.windows.net\pvc-57d12ee1-f847-44ec-830b-e6375971a791
I0618 10:20:05.068931 10816 smb.go:97] checking remote server path UNC<redacted>.file.core.windows.net\pvc-e1118b6f-8b8e-4fbe-b680-0671072fbf2e on local path C:\var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\f9d0b87889d2774ec5e434e4f21da3a34b60667e8473ff91641328c209ba2c8f\globalmount
I0618 10:20:05.068931 10816 smb.go:62] begin to run RemoveSmbGlobalMapping with \.file.core.windows.net\pvc-0eb2b4ab-9f0d-48e8-81e0-1639f3b8f26a
I0618 10:20:05.474311 10816 smb.go:97] checking remote server path UNC<redacted>.file.core.windows.net\pvc-e1118b6f-8b8e-4fbe-b680-0671072fbf2e on local path C:\var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\f9d0b87889d2774ec5e434e4f21da3a34b60667e8473ff91641328c209ba2c8f\globalmount
I0618 10:20:05.474311 10816 smb.go:62] begin to run RemoveSmbGlobalMapping with \.file.core.windows.net\pvc-f3e3de62-8522-473b-99b3-ee77c7492d9c
I0618 10:20:06.699287 10816 safe_mounter_host_process_windows.go:153] Unmount: remote path: \.file.core.windows.net\pvc-72761240-0540-421b-8dab-fe0715219f8e local path: c:\var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\c12f70176ba6695ba94e730da66a9fa3598f85b824f9855ae55ee54ed85492cf\globalmount
I0618 10:20:06.699287 10816 nodeserver.go:438] NodeUnstageVolume: unmount volume ##pvc-72761240-0540-421b-8dab-fe0715219f8e### on \var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\c12f70176ba6695ba94e730da66a9fa3598f85b824f9855ae55ee54ed85492cf\globalmount successfully
I0618 10:20:06.699287 10816 azure_metrics.go:118] "Observed Request Latency" latency_seconds=202.2881918 request="azurefile_csi_driver_node_unstage_volume" resource_group="" subscription_id="" source="file.csi.azure.com" volumeid="##pvc-72761240-0540-421b-8dab-fe0715219f8e###" result_code="succeeded"
I0618 10:20:06.699287 10816 utils.go:108] GRPC response: {}
I0618 10:20:07.722251 10816 safe_mounter_host_process_windows.go:153] Unmount: remote path: \.file.core.windows.net\pvc-79ecdf05-1c90-47d4-826a-77f19a8ef701 local path: c:\var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\36cadd6f3c6efd50be224333d28fce0715ebf57ccb46583c9f572e61f4b74455\globalmount
I0618 10:20:07.722251 10816 nodeserver.go:438] NodeUnstageVolume: unmount volume ##pvc-79ecdf05-1c90-47d4-826a-77f19a8ef701###-suite on \var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\36cadd6f3c6efd50be224333d28fce0715ebf57ccb46583c9f572e61f4b74455\globalmount successfully
I0618 10:20:07.722251 10816 azure_metrics.go:118] "Observed Request Latency" latency_seconds=203.2887571 request="azurefile_csi_driver_node_unstage_volume" resource_group="" subscription_id="" source="file.csi.azure.com" volumeid="##pvc-79ecdf05-1c90-47d4-826a-77f19a8ef701###-suite" result_code="succeeded"
I0618 10:20:07.722251 10816 utils.go:108] GRPC response: {}
I0618 10:20:08.828625 10816 safe_mounter_host_process_windows.go:153] Unmount: remote path: \.file.core.windows.net\pvc-4ac50876-5989-405a-9143-5d587fcd2988 local path: c:\var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\034c21971c3ececa002bdf34aff6700c4d8158d26011a278e01b72ccda712ece\globalmount
I0618 10:20:08.829524 10816 nodeserver.go:438] NodeUnstageVolume: unmount volume ##pvc-4ac50876-5989-405a-9143-5d587fcd2988### on \var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\034c21971c3ececa002bdf34aff6700c4d8158d26011a278e01b72ccda712ece\globalmount successfully
I0618 10:20:08.829598 10816 azure_metrics.go:118] "Observed Request Latency" latency_seconds=204.3185916 request="azurefile_csi_driver_node_unstage_volume" resource_group="" subscription_id="" source="file.csi.azure.com" volumeid="##pvc-4ac50876-5989-405a-9143-5d587fcd2988###" result_code="succeeded"
I0618 10:20:08.829632 10816 utils.go:108] GRPC response: {}
I0618 10:20:09.902257 10816 safe_mounter_host_process_windows.go:153] Unmount: remote path: \.file.core.windows.net\pvc-0242f32f-f56b-4cb9-b7da-7226cc26b57c local path: c:\var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\738c2ee924a3cd1c0f7d6b7402f0340b800a969b535303b27ae756a769c1d8c5\globalmount
I0618 10:20:09.902678 10816 nodeserver.go:438] NodeUnstageVolume: unmount volume ##pvc-0242f32f-f56b-4cb9-b7da-7226cc26b57c### on \var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\738c2ee924a3cd1c0f7d6b7402f0340b800a969b535303b27ae756a769c1d8c5\globalmount successfully
I0618 10:20:09.902702 10816 azure_metrics.go:118] "Observed Request Latency" latency_seconds=205.3916363 request="azurefile_csi_driver_node_unstage_volume" resource_group="" subscription_id="" source="file.csi.azure.com" volumeid="##pvc-0242f32f-f56b-4cb9-b7da-7226cc26b57c###" result_code="succeeded"
I0618 10:20:09.902756 10816 utils.go:108] GRPC response: {}
I0618 10:20:10.878791 10816 safe_mounter_host_process_windows.go:153] Unmount: remote path: \.file.core.windows.net\pvc-3009b498-4ff5-45cc-afe3-9e5b218198f0 local path: c:\var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\93f41f081097d72376a47274df2be10a55b0ed09627df13e69671b18909b282d\globalmount
I0618 10:20:10.878791 10816 nodeserver.go:438] NodeUnstageVolume: unmount volume ##pvc-3009b498-4ff5-45cc-afe3-9e5b218198f0### on \var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\93f41f081097d72376a47274df2be10a55b0ed09627df13e69671b18909b282d\globalmount successfully
I0618 10:20:10.878791 10816 azure_metrics.go:118] "Observed Request Latency" latency_seconds=206.3627631 request="azurefile_csi_driver_node_unstage_volume" resource_group="" subscription_id="" source="file.csi.azure.com" volumeid="##pvc-3009b498-4ff5-45cc-afe3-9e5b218198f0###" result_code="succeeded"
I0618 10:20:10.878791 10816 utils.go:108] GRPC response: {}
I0618 10:20:11.786121 10816 safe_mounter_host_process_windows.go:153] Unmount: remote path: \.file.core.windows.net\pvc-78ae83bf-0534-44ce-a2bf-8e91cf904444 local path: c:\var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\bdcbb4137593f2fb676c9215aa83a13d054bf0763edd31695686483cef010f34\globalmount
I0618 10:20:11.789218 10816 nodeserver.go:438] NodeUnstageVolume: unmount volume ##pvc-78ae83bf-0534-44ce-a2bf-8e91cf904444### on \var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\bdcbb4137593f2fb676c9215aa83a13d054bf0763edd31695686483cef010f34\globalmount successfully
I0618 10:20:11.789218 10816 azure_metrics.go:118] "Observed Request Latency" latency_seconds=207.2731914 request="azurefile_csi_driver_node_unstage_volume" resource_group="" subscription_id="" source="file.csi.azure.com" volumeid="##pvc-78ae83bf-0534-44ce-a2bf-8e91cf904444###" result_code="succeeded"
I0618 10:20:11.789218 10816 utils.go:108] GRPC response: {}
I0618 10:20:12.697301 10816 safe_mounter_host_process_windows.go:153] Unmount: remote path: \.file.core.windows.net\pvc-2949f134-77a9-49fe-bbe8-a56869a66da5 local path: c:\var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\8bb3100c43ed731f7bf1eb91db32110762dc8812447044cd2a1898c72013c4c8\globalmount
I0618 10:20:12.697724 10816 nodeserver.go:438] NodeUnstageVolume: unmount volume ##pvc-2949f134-77a9-49fe-bbe8-a56869a66da5### on \var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\8bb3100c43ed731f7bf1eb91db32110762dc8812447044cd2a1898c72013c4c8\globalmount successfully
I0618 10:20:12.697802 10816 azure_metrics.go:118] "Observed Request Latency" latency_seconds=208.1809329 request="azurefile_csi_driver_node_unstage_volume" resource_group="" subscription_id="" source="file.csi.azure.com" volumeid="##pvc-2949f134-77a9-49fe-bbe8-a56869a66da5###" result_code="succeeded"
I0618 10:20:12.697835 10816 utils.go:108] GRPC response: {}
I0618 10:20:12.946110 10816 smb.go:97] checking remote server path UNC<redacted>.file.core.windows.net\pvc-f9f952d8-fd7b-423b-8384-3230cbbef2d2 on local path C:\var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\c9d4eedbd9b825d6afca7c8ae5b55d2deeea2e1ab125d9502446d5b62c0de9fa\globalmount
I0618 10:20:13.202160 10816 smb.go:97] checking remote server path UNC<redacted>.file.core.windows.net\pvc-f9f952d8-fd7b-423b-8384-3230cbbef2d2 on local path C:\var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\c9d4eedbd9b825d6afca7c8ae5b55d2deeea2e1ab125d9502446d5b62c0de9fa\globalmount
I0618 10:20:13.476106 10816 smb.go:97] checking remote server path UNC<redacted>.file.core.windows.net\pvc-f9f952d8-fd7b-423b-8384-3230cbbef2d2 on local path C:\var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\c9d4eedbd9b825d6afca7c8ae5b55d2deeea2e1ab125d9502446d5b62c0de9fa\globalmount
I0618 10:20:14.636814 10816 safe_mounter_host_process_windows.go:153] Unmount: remote path: \.file.core.windows.net\pvc-713ee767-8180-4301-ba8d-28493cc53e93 local path: c:\var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\f94621da65e3f3e84fea189a34121ae5cf00901c29de8fae77900bdd37cffd64\globalmount
I0618 10:20:14.636814 10816 nodeserver.go:438] NodeUnstageVolume: unmount volume ##pvc-713ee767-8180-4301-ba8d-28493cc53e93### on \var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\f94621da65e3f3e84fea189a34121ae5cf00901c29de8fae77900bdd37cffd64\globalmount successfully
I0618 10:20:14.636814 10816 azure_metrics.go:118] "Observed Request Latency" latency_seconds=210.119928 request="azurefile_csi_driver_node_unstage_volume" resource_group="" subscription_id="" source="file.csi.azure.com" volumeid="##pvc-713ee767-8180-4301-ba8d-28493cc53e93###" result_code="succeeded"
I0618 10:20:14.636814 10816 utils.go:108] GRPC response: {}
I0618 10:20:15.534033 10816 safe_mounter_host_process_windows.go:153] Unmount: remote path: \.file.core.windows.net\pvc-2db7726b-6f2b-4356-96ce-23205ca74256 local path: c:\var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\79b5697b9072bfb2fbdeb3ade8c4a1518580957fc5a68e2d4ff0ebd333b11a23\globalmount
I0618 10:20:15.534636 10816 nodeserver.go:438] NodeUnstageVolume: unmount volume ##pvc-2db7726b-6f2b-4356-96ce-23205ca74256### on \var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\79b5697b9072bfb2fbdeb3ade8c4a1518580957fc5a68e2d4ff0ebd333b11a23\globalmount successfully
I0618 10:20:15.534636 10816 azure_metrics.go:118] "Observed Request Latency" latency_seconds=211.0177506 request="azurefile_csi_driver_node_unstage_volume" resource_group="" subscription_id="" source="file.csi.azure.com" volumeid="##pvc-2db7726b-6f2b-4356-96ce-23205ca74256###" result_code="succeeded"
I0618 10:20:15.534636 10816 utils.go:108] GRPC response: {}
I0618 10:20:16.526202 10816 safe_mounter_host_process_windows.go:153] Unmount: remote path: \.file.core.windows.net\pvc-5a6a954d-5190-4d60-9109-88aa0451fe4b local path: c:\var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\21bfe19e2ba4d47cbb05c6d41a5870b528440959370643cbc690441dbb5d1076\globalmount
I0618 10:20:16.526202 10816 nodeserver.go:438] NodeUnstageVolume: unmount volume ##pvc-5a6a954d-5190-4d60-9109-88aa0451fe4b### on \var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\21bfe19e2ba4d47cbb05c6d41a5870b528440959370643cbc690441dbb5d1076\globalmount successfully
I0618 10:20:16.526202 10816 azure_metrics.go:118] "Observed Request Latency" latency_seconds=212.0083226 request="azurefile_csi_driver_node_unstage_volume" resource_group="" subscription_id="" source="file.csi.azure.com" volumeid="##pvc-5a6a954d-5190-4d60-9109-88aa0451fe4b###" result_code="succeeded"
I0618 10:20:16.526202 10816 utils.go:108] GRPC response: {}
I0618 10:20:17.392968 10816 safe_mounter_host_process_windows.go:153] Unmount: remote path: \.file.core.windows.net\pvc-44c9f43c-e568-4fc5-a367-54d83780d72d local path: c:\var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\6f61b1c4aff21ed830d9688c61158b783824cd05856cb6b4ea51892d49167d2b\globalmount
I0618 10:20:17.393701 10816 nodeserver.go:438] NodeUnstageVolume: unmount volume ##pvc-44c9f43c-e568-4fc5-a367-54d83780d72d### on \var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\6f61b1c4aff21ed830d9688c61158b783824cd05856cb6b4ea51892d49167d2b\globalmount successfully
I0618 10:20:17.393803 10816 azure_metrics.go:118] "Observed Request Latency" latency_seconds=212.8753152 request="azurefile_csi_driver_node_unstage_volume" resource_group="" subscription_id="" source="file.csi.azure.com" volumeid="##pvc-44c9f43c-e568-4fc5-a367-54d83780d72d###" result_code="succeeded"
I0618 10:20:17.393879 10816 utils.go:108] GRPC response: {}
I0618 10:20:18.224804 10816 safe_mounter_host_process_windows.go:153] Unmount: remote path: \.file.core.windows.net\pvc-24179808-e2c1-4437-974e-01ebb4f02b13 local path: c:\var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\9edcba724428c1bee6be9a30ece697a4e78a7d47d2c29f421d9be77a41207db3\globalmount
I0618 10:20:18.225363 10816 nodeserver.go:438] NodeUnstageVolume: unmount volume ##pvc-24179808-e2c1-4437-974e-01ebb4f02b13### on \var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\9edcba724428c1bee6be9a30ece697a4e78a7d47d2c29f421d9be77a41207db3\globalmount successfully
I0618 10:20:18.225363 10816 azure_metrics.go:118] "Observed Request Latency" latency_seconds=213.7069489 request="azurefile_csi_driver_node_unstage_volume" resource_group="" subscription_id="" source="file.csi.azure.com" volumeid="##pvc-24179808-e2c1-4437-974e-01ebb4f02b13###" result_code="succeeded"
I0618 10:20:18.225363 10816 utils.go:108] GRPC response: {}
I0618 10:20:19.136511 10816 safe_mounter_host_process_windows.go:153] Unmount: remote path: \.file.core.windows.net\pvc-9fba43bc-d909-4ed8-b35b-8334f31eb153 local path: c:\var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\05166299f887278812bc8dce6779991718b760c1e62cd128750ae9880e1c9e95\globalmount
I0618 10:20:19.137299 10816 nodeserver.go:438] NodeUnstageVolume: unmount volume ##pvc-9fba43bc-d909-4ed8-b35b-8334f31eb153### on \var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\05166299f887278812bc8dce6779991718b760c1e62cd128750ae9880e1c9e95\globalmount successfully
I0618 10:20:19.137299 10816 azure_metrics.go:118] "Observed Request Latency" latency_seconds=214.6188864 request="azurefile_csi_driver_node_unstage_volume" resource_group="" subscription_id="" source="file.csi.azure.com" volumeid="##pvc-9fba43bc-d909-4ed8-b35b-8334f31eb153###" result_code="succeeded"
I0618 10:20:19.137299 10816 utils.go:108] GRPC response: {}
I0618 10:20:20.060895 10816 safe_mounter_host_process_windows.go:153] Unmount: remote path: \.file.core.windows.net\pvc-2d21d178-6878-4ed8-812b-bc0475b5ff6b local path: c:\var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\c99e9e73af00de6731ba0d597b58df22ac4c6051d80b17f91bccfbf5e9b1c5ba\globalmount
I0618 10:20:20.061880 10816 nodeserver.go:438] NodeUnstageVolume: unmount volume ##pvc-2d21d178-6878-4ed8-812b-bc0475b5ff6b###-suite on \var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\c99e9e73af00de6731ba0d597b58df22ac4c6051d80b17f91bccfbf5e9b1c5ba\globalmount successfully
I0618 10:20:20.061971 10816 azure_metrics.go:118] "Observed Request Latency" latency_seconds=215.5428814 request="azurefile_csi_driver_node_unstage_volume" resource_group="" subscription_id="" source="file.csi.azure.com" volumeid="##pvc-2d21d178-6878-4ed8-812b-bc0475b5ff6b###-suite" result_code="succeeded"
I0618 10:20:20.062000 10816 utils.go:108] GRPC response: {}
I0618 10:20:21.028418 10816 safe_mounter_host_process_windows.go:153] Unmount: remote path: \.file.core.windows.net\pvc-e27b7c29-471a-438c-8af3-2bac9c24d699 local path: c:\var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\ca3534d0553fab117563cb8a57e1428c986890a081e1ad90b6671e14f1b5df60\globalmount
I0618 10:20:21.029038 10816 nodeserver.go:438] NodeUnstageVolume: unmount volume ##pvc-e27b7c29-471a-438c-8af3-2bac9c24d699### on \var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\ca3534d0553fab117563cb8a57e1428c986890a081e1ad90b6671e14f1b5df60\globalmount successfully
I0618 10:20:21.029038 10816 azure_metrics.go:118] "Observed Request Latency" latency_seconds=216.4953155 request="azurefile_csi_driver_node_unstage_volume" resource_group="" subscription_id="" source="file.csi.azure.com" volumeid="##pvc-e27b7c29-471a-438c-8af3-2bac9c24d699###" result_code="succeeded"
I0618 10:20:21.029038 10816 utils.go:108] GRPC response: {}
I0618 10:20:21.934535 10816 safe_mounter_host_process_windows.go:153] Unmount: remote path: \.file.core.windows.net\pvc-e1118b6f-8b8e-4fbe-b680-0671072fbf2e local path: c:\var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\f9d0b87889d2774ec5e434e4f21da3a34b60667e8473ff91641328c209ba2c8f\globalmount
I0618 10:20:21.935276 10816 nodeserver.go:438] NodeUnstageVolume: unmount volume ##pvc-e1118b6f-8b8e-4fbe-b680-0671072fbf2e### on \var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\f9d0b87889d2774ec5e434e4f21da3a34b60667e8473ff91641328c209ba2c8f\globalmount successfully
I0618 10:20:21.935364 10816 azure_metrics.go:118] "Observed Request Latency" latency_seconds=217.4015968 request="azurefile_csi_driver_node_unstage_volume" resource_group="" subscription_id="" source="file.csi.azure.com" volumeid="##pvc-e1118b6f-8b8e-4fbe-b680-0671072fbf2e###" result_code="succeeded"
I0618 10:20:21.935384 10816 utils.go:108] GRPC response: {}
I0618 10:20:22.847634 10816 safe_mounter_host_process_windows.go:153] Unmount: remote path: \.file.core.windows.net\pvc-d54bf4ed-647d-480a-8de5-3f647ca1126f local path: c:\var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\283cb9d1568781162f98d129fc9701b7c94d5c357616bbe8d06ae6dead8f97c7\globalmount
I0618 10:20:22.857050 10816 nodeserver.go:438] NodeUnstageVolume: unmount volume ##pvc-d54bf4ed-647d-480a-8de5-3f647ca1126f###-suite on \var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\283cb9d1568781162f98d129fc9701b7c94d5c357616bbe8d06ae6dead8f97c7\globalmount successfully
I0618 10:20:22.857050 10816 azure_metrics.go:118] "Observed Request Latency" latency_seconds=218.3233306 request="azurefile_csi_driver_node_unstage_volume" resource_group="" subscription_id="" source="file.csi.azure.com" volumeid="##pvc-d54bf4ed-647d-480a-8de5-3f647ca1126f###-suite" result_code="succeeded"
I0618 10:20:22.857050 10816 utils.go:108] GRPC response: {}
I0618 10:20:22.998545 10816 utils.go:101] GRPC call: /csi.v1.Node/NodeUnstageVolume
I0618 10:20:22.999582 10816 utils.go:101] GRPC call: /csi.v1.Node/NodeUnstageVolume
I0618 10:20:22.998545 10816 utils.go:102] GRPC request: {"staging_target_path":"\var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\86e0827bb9cbd0427bfabe2ef368b87f24995bcd1f97f071a152786fa1dfb37f\globalmount","volume_id":"##pvc-1517e05b-fdb0-4947-a7c6-da943cf99efa###"}
E0618 10:20:22.999582 10816 utils.go:106] GRPC error: rpc error: code = Aborted desc = An operation with the given Volume ID ##pvc-1517e05b-fdb0-4947-a7c6-da943cf99efa### already exists
I0618 10:20:22.999582 10816 utils.go:102] GRPC request: {"staging_target_path":"\var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\0e8d433c6973fb38384981c32271f2e5299a8b5f828099586f0efad994ba67c2\globalmount","volume_id":"##pvc-b4f7a590-4570-4fc1-a978-05558603dca1###"}
E0618 10:20:22.999882 10816 utils.go:106] GRPC error: rpc error: code = Aborted desc = An operation with the given Volume ID ##pvc-b4f7a590-4570-4fc1-a978-05558603dca1### already exists
I0618 10:20:23.003864 10816 utils.go:101] GRPC call: /csi.v1.Node/NodeUnstageVolume
I0618 10:20:23.003864 10816 utils.go:102] GRPC request: {"staging_target_path":"\var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\99ef496ccb72e2f549ef78b842855c22765e1f899df6248b8d815cb36f0eae8d\globalmount","volume_id":"##pvc-a7ac343a-aff1-4a2a-8eec-959855b2b15b###"}
E0618 10:20:23.003864 10816 utils.go:106] GRPC error: rpc error: code = Aborted desc = An operation with the given Volume ID ##pvc-a7ac343a-aff1-4a2a-8eec-959855b2b15b### already exists
I0618 10:20:23.836056 10816 safe_mounter_host_process_windows.go:153] Unmount: remote path: \.file.core.windows.net\pvc-ff818ba6-ec9f-4486-80e4-7cc61350f6d5 local path: c:\var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\08afa7f5599b501c3ec032c18fbb1f626a7e12ee8672fa52b05dce745f28e873\globalmount
I0618 10:20:23.836814 10816 nodeserver.go:438] NodeUnstageVolume: unmount volume ##pvc-ff818ba6-ec9f-4486-80e4-7cc61350f6d5### on \var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\08afa7f5599b501c3ec032c18fbb1f626a7e12ee8672fa52b05dce745f28e873\globalmount successfully
I0618 10:20:23.837362 10816 azure_metrics.go:118] "Observed Request Latency" latency_seconds=219.2973164 request="azurefile_csi_driver_node_unstage_volume" resource_group="" subscription_id="" source="file.csi.azure.com" volumeid="##pvc-ff818ba6-ec9f-4486-80e4-7cc61350f6d5###" result_code="succeeded"
I0618 10:20:23.837826 10816 utils.go:108] GRPC response: {}
I0618 10:20:24.773724 10816 safe_mounter_host_process_windows.go:153] Unmount: remote path: \.file.core.windows.net\pvc-b034ea6e-80a0-4b98-a036-a7557c1d2595 local path: c:\var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\d0572836866328f35f4a64fbf26a85c83ef3c903cfd92d82f10c5c0d721ad463\globalmount
I0618 10:20:24.773724 10816 nodeserver.go:438] NodeUnstageVolume: unmount volume ##pvc-b034ea6e-80a0-4b98-a036-a7557c1d2595###-suite on \var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\d0572836866328f35f4a64fbf26a85c83ef3c903cfd92d82f10c5c0d721ad463\globalmount successfully
I0618 10:20:24.773724 10816 azure_metrics.go:118] "Observed Request Latency" latency_seconds=220.2335692 request="azurefile_csi_driver_node_unstage_volume" resource_group="" subscription_id="" source="file.csi.azure.com" volumeid="##pvc-b034ea6e-80a0-4b98-a036-a7557c1d2595###-suite" result_code="succeeded"
I0618 10:20:24.773724 10816 utils.go:108] GRPC response: {}
I0618 10:20:25.605800 10816 safe_mounter_host_process_windows.go:153] Unmount: remote path: \.file.core.windows.net\pvc-f9f952d8-fd7b-423b-8384-3230cbbef2d2 local path: c:\var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\c9d4eedbd9b825d6afca7c8ae5b55d2deeea2e1ab125d9502446d5b62c0de9fa\globalmount
I0618 10:20:25.606589 10816 nodeserver.go:438] NodeUnstageVolume: unmount volume ##pvc-f9f952d8-fd7b-423b-8384-3230cbbef2d2###-suite on \var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\c9d4eedbd9b825d6afca7c8ae5b55d2deeea2e1ab125d9502446d5b62c0de9fa\globalmount successfully
I0618 10:20:25.606659 10816 azure_metrics.go:118] "Observed Request Latency" latency_seconds=221.0657469 request="azurefile_csi_driver_node_unstage_volume" resource_group="" subscription_id="" source="file.csi.azure.com" volumeid="##pvc-f9f952d8-fd7b-423b-8384-3230cbbef2d2###-suite" result_code="succeeded"
I0618 10:20:25.606685 10816 utils.go:108] GRPC response: {}
I0618 10:20:26.530883 10816 safe_mounter_host_process_windows.go:153] Unmount: remote path: \.file.core.windows.net\pvc-373137fb-16fa-4445-9526-fa69d783ec00 local path: c:\var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\d1789d276f47d27aa0fbb3d446801109a044ccc5ba7efb1fe475edb4765412ee\globalmount
I0618 10:20:26.531513 10816 nodeserver.go:438] NodeUnstageVolume: unmount volume ##pvc-373137fb-16fa-4445-9526-fa69d783ec00### on \var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\d1789d276f47d27aa0fbb3d446801109a044ccc5ba7efb1fe475edb4765412ee\globalmount successfully
I0618 10:20:26.531635 10816 azure_metrics.go:118] "Observed Request Latency" latency_seconds=221.8119688 request="azurefile_csi_driver_node_unstage_volume" resource_group="" subscription_id="" source="file.csi.azure.com" volumeid="##pvc-373137fb-16fa-4445-9526-fa69d783ec00###" result_code="succeeded"
I0618 10:20:26.531661 10816 utils.go:108] GRPC response: {}
I0618 10:20:27.386371 10816 safe_mounter_host_process_windows.go:153] Unmount: remote path: \.file.core.windows.net\pvc-e465606b-ee96-42f4-9108-266751f62cc0 local path: c:\var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\7aaf7ebb435bc4e96ca15a1f3d94f31fead5f923b509ecab572a6228a3e96406\globalmount
I0618 10:20:27.386371 10816 nodeserver.go:438] NodeUnstageVolume: unmount volume ##pvc-e465606b-ee96-42f4-9108-266751f62cc0###-suite on \var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\7aaf7ebb435bc4e96ca15a1f3d94f31fead5f923b509ecab572a6228a3e96406\globalmount successfully
I0618 10:20:27.386371 10816 azure_metrics.go:118] "Observed Request Latency" latency_seconds=222.6667772 request="azurefile_csi_driver_node_unstage_volume" resource_group="" subscription_id="" source="file.csi.azure.com" volumeid="##pvc-e465606b-ee96-42f4-9108-266751f62cc0###-suite" result_code="succeeded"
I0618 10:20:27.386371 10816 utils.go:108] GRPC response: {}
I0618 10:20:28.241399 10816 safe_mounter_host_process_windows.go:153] Unmount: remote path: \.file.core.windows.net\pvc-9cc1f701-08df-45fa-9a63-6316ce0f4521 local path: c:\var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\b5a8f71ac9345abb3846ff099001ef91961525cbeda2ef05a084fd85175e1973\globalmount
I0618 10:20:28.241399 10816 nodeserver.go:438] NodeUnstageVolume: unmount volume ##pvc-9cc1f701-08df-45fa-9a63-6316ce0f4521###-suite on \var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\b5a8f71ac9345abb3846ff099001ef91961525cbeda2ef05a084fd85175e1973\globalmount successfully
I0618 10:20:28.241399 10816 azure_metrics.go:118] "Observed Request Latency" latency_seconds=223.5207706 request="azurefile_csi_driver_node_unstage_volume" resource_group="" subscription_id="" source="file.csi.azure.com" volumeid="##pvc-9cc1f701-08df-45fa-9a63-6316ce0f4521###-suite" result_code="succeeded"
I0618 10:20:28.241399 10816 utils.go:108] GRPC response: {}
I0618 10:20:29.222249 10816 safe_mounter_host_process_windows.go:153] Unmount: remote path: \.file.core.windows.net\pvc-57d12ee1-f847-44ec-830b-e6375971a791 local path: c:\var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\426633f8b1f494922897121ccf62104b21279f8a4d3bca2e189b30c83dd1a1e1\globalmount
I0618 10:20:29.223610 10816 nodeserver.go:438] NodeUnstageVolume: unmount volume ##pvc-57d12ee1-f847-44ec-830b-e6375971a791###-suite on \var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\426633f8b1f494922897121ccf62104b21279f8a4d3bca2e189b30c83dd1a1e1\globalmount successfully
I0618 10:20:29.223610 10816 azure_metrics.go:118] "Observed Request Latency" latency_seconds=224.3052176 request="azurefile_csi_driver_node_unstage_volume" resource_group="" subscription_id="" source="file.csi.azure.com" volumeid="##pvc-57d12ee1-f847-44ec-830b-e6375971a791###-suite" result_code="succeeded"
I0618 10:20:29.223610 10816 utils.go:108] GRPC response: {}
I0618 10:20:30.200191 10816 safe_mounter_host_process_windows.go:153] Unmount: remote path: \.file.core.windows.net\pvc-0eb2b4ab-9f0d-48e8-81e0-1639f3b8f26a local path: c:\var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\e43c03d50a8520c906a87a68e2869d94c67f0f6beac75b150d5d633223e51b53\globalmount
I0618 10:20:30.201492 10816 nodeserver.go:438] NodeUnstageVolume: unmount volume ##pvc-0eb2b4ab-9f0d-48e8-81e0-1639f3b8f26a###-suite on \var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\e43c03d50a8520c906a87a68e2869d94c67f0f6beac75b150d5d633223e51b53\globalmount successfully
I0618 10:20:30.201553 10816 azure_metrics.go:118] "Observed Request Latency" latency_seconds=225.2751726 request="azurefile_csi_driver_node_unstage_volume" resource_group="" subscription_id="" source="file.csi.azure.com" volumeid="##pvc-0eb2b4ab-9f0d-48e8-81e0-1639f3b8f26a###-suite" result_code="succeeded"
I0618 10:20:30.201590 10816 utils.go:108] GRPC response: {}
I0618 10:20:31.174562 10816 safe_mounter_host_process_windows.go:153] Unmount: remote path: \.file.core.windows.net\pvc-f3e3de62-8522-473b-99b3-ee77c7492d9c local path: c:\var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\2828eb0a20f4e36addf1412311ed2db77c2294bdd1119315fcfbd7cd969ec847\globalmount
I0618 10:20:31.174562 10816 nodeserver.go:438] NodeUnstageVolume: unmount volume ##pvc-f3e3de62-8522-473b-99b3-ee77c7492d9c###-suite on \var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\2828eb0a20f4e36addf1412311ed2db77c2294bdd1119315fcfbd7cd969ec847\globalmount successfully
I0618 10:20:31.175100 10816 azure_metrics.go:118] "Observed Request Latency" latency_seconds=226.2482117 request="azurefile_csi_driver_node_unstage_volume" resource_group="" subscription_id="" source="file.csi.azure.com" volumeid="##pvc-f3e3de62-8522-473b-99b3-ee77c7492d9c###-suite" result_code="succeeded"
I0618 10:20:31.175135 10816 utils.go:108] GRPC response: {}
I0618 10:20:31.556868 10816 smb.go:97] checking remote server path Get-Item : Cannot find path 'C:\var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\ca3534d0553fab117563cb8a57
e1428c986890a081e1ad90b6671e14f1b5df60\globalmount' because it does not exist.
At line:1 char:2
- (Get-Item -Path $Env:mount).Target
-
+ CategoryInfo : ObjectNotFound: (C:\var\lib\kube...f60\globalmount:String) [Get-Item], ItemNotFoundExcep tion + FullyQualifiedErrorId : PathNotFound,Microsoft.PowerShell.Commands.GetItemCommand on local path C:\var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\ca3534d0553fab117563cb8a57e1428c986890a081e1ad90b6671e14f1b5df60\globalmount
I0618 10:20:31.557389 10816 smb.go:62] begin to run RemoveSmbGlobalMapping with \.file.core.windows.net\pvc-b4f7a590-4570-4fc1-a978-05558603dca1
I0618 10:20:31.926936 10816 smb.go:97] checking remote server path Get-Item : Cannot find path 'C:\var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\ca3534d0553fab117563cb8a57
e1428c986890a081e1ad90b6671e14f1b5df60\globalmount' because it does not exist.
At line:1 char:2
- (Get-Item -Path $Env:mount).Target
-
+ CategoryInfo : ObjectNotFound: (C:\var\lib\kube...f60\globalmount:String) [Get-Item], ItemNotFoundExcep tion + FullyQualifiedErrorId : PathNotFound,Microsoft.PowerShell.Commands.GetItemCommand on local path C:\var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\ca3534d0553fab117563cb8a57e1428c986890a081e1ad90b6671e14f1b5df60\globalmount
I0618 10:20:31.928963 10816 smb.go:62] begin to run RemoveSmbGlobalMapping with \.file.core.windows.net\pvc-1517e05b-fdb0-4947-a7c6-da943cf99efa
I0618 10:20:32.250308 10816 smb.go:97] checking remote server path Get-Item : Cannot find path 'C:\var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\ca3534d0553fab117563cb8a57
e1428c986890a081e1ad90b6671e14f1b5df60\globalmount' because it does not exist.
At line:1 char:2
- (Get-Item -Path $Env:mount).Target
-
+ CategoryInfo : ObjectNotFound: (C:\var\lib\kube...f60\globalmount:String) [Get-Item], ItemNotFoundExcep tion + FullyQualifiedErrorId : PathNotFound,Microsoft.PowerShell.Commands.GetItemCommand on local path C:\var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\ca3534d0553fab117563cb8a57e1428c986890a081e1ad90b6671e14f1b5df60\globalmount
`
Warning FailedMount 48s (x5 over 8m56s) kubelet MountVolume.MountDevice failed for volume 'pvc-0e8add25-432b-49bc-bd80-75f6de173faf' : rpc error: code = DeadlineExceeded desc = context deadline exceeded
The Azurefile CSI logging is as follows and similar to the above.
Warning FailedMount 48s (x5 over 8m56s) kubelet MountVolume.MountDevice failed for volume 'pvc-0e8add25-432b-49bc-bd80-75f6de173faf' : rpc error: code = DeadlineExceeded desc = context deadline exceeded
The Azurefile CSI logging is as follows
`I0606 14:11:42.325859 7304 azure_metrics.go:118] 'Observed Request Latency' latency_seconds=120.0105828 request='azurefile_csi_driver_node_stage_volume' resource_group='' subscription_id='' source='file.csi.azure.com' volumeid='pvc-0e8add25-432b-49bc-bd80-75f6de173faf###s' result_code='failed_csi_driver_node_stage_volume'
E0606 14:11:42.325981 7304 utils.go:106] GRPC error: rpc error: code = Internal desc = volume(MC_rg-aks-prd_aks-prd_westeurope#fc7a964cdab3c4c3abd74c7#pvc-0e8add25-432b-49bc-bd80-75f6de173faf###force-mortgages) mount \\fc7a964cdab3c4c3abd74c7.file.core.windows.net\pvc-0e8add25-432b-49bc-bd80-75f6de173faf on \var\lib\kubelet\plugins\kubernetes.io\csi\file.csi.azure.com\a9d936b9e022f1c614ed0052bc2a9a606172dbeca4bca4d830ec0671b89f5662\globalmount failed with time out
`