Blobfuse Flexvol high memory consumption

[Johan2W]

What happened:
I am using AKS to host Azure Machine Learning deployments. Azure Machine Learning installs Blobfuse Flexvol on all nodes and uses it in the pods it deploys.
After a while I noticed on all nodes where pods are using blobfuse, available memory was low. Looking at the processes I noticed the blobfuse process taking up a large amount of memory, increasing over time:

   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                                                                                                                                                                             
 70814 root      20   0 1584.4m 1.020g   3.9m S   1.0 15.0 111:44.20 blobfuse /var/lib/kubelet/pods/7eb4e9f6-b24c-409d-acdc-d6c7c3245db1/volumes/azure~blobfuse/coguk1-0-acc-acc72c4e19281934d378bc07280e08bb043-pvc --container-name=azureml --tmp-path=/tmp/blobfuse/ -o allow_other -o ro --file-cache-timeout-in-seconds=1200 
 70820 root      20   0 1392.4m 908.8m   1.4m S   1.0 13.1 111:21.89 blobfuse /var/lib/kubelet/pods/1b1f33d2-da0b-4c64-bc38-9a766689b36a/volumes/azure~blobfuse/cogukag-acc-accd16a35e863fc41c79976f144ce3100e7-pvc --container-name=azureml --tmp-path=/tmp/blobfuse/ -o allow_other -o ro --file-cache-timeout-in-seconds=1200
 70893 root      20   0 1264.4m 859.4m   2.0m S   0.7 12.4 111:34.31 blobfuse /var/lib/kubelet/pods/d6fac6a9-9718-43d9-b088-e2162ba7a2ef/volumes/azure~blobfuse/cogusdbm-acc-accade73fcaa67b487fa0ed702af4305f82-pvc --container-name=azureml --tmp-path=/tmp/blobfuse/ -o allow_other -o ro --file-cache-timeout-in-seconds=1200 
 70991 root      20   0 1264.4m 740.5m   3.8m S   1.0 10.7 111:33.34 blobfuse /var/lib/kubelet/pods/b713a5bf-f9cb-46c0-8f20-1630c50ff7eb/volumes/azure~blobfuse/cogex-acc-acc24e1419c60844136b765328478223650-pvc --container-name=azureml --tmp-path=/tmp/blobfuse/ -o allow_other -o ro --file-cache-timeout-in-seconds=1200

It doesn't seem the memory usage ever drops.

What you expected to happen:
memory usage going up and down based on usage.

Anything else we need to know?:
not sure

Environment:

• Kubernetes version (use kubectl version):
  Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.8", GitCommit:"ec6eb119b81be488b030e849b9e64fda4caaf33c", GitTreeState:"clean", BuildDate:"2020-03-12T21:00:06Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"windows/amd64"}
  Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.10", GitCommit:"89d8075525967c7a619641fabcb267358d28bf08", GitTreeState:"clean", BuildDate:"2020-06-23T02:52:37Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

• OS (e.g. from /etc/os-release):
  NAME="Ubuntu"
  VERSION="16.04.6 LTS (Xenial Xerus)"
  ID=ubuntu
  ID_LIKE=debian
  PRETTY_NAME="Ubuntu 16.04.6 LTS"
  VERSION_ID="16.04"

• Kernel (e.g. uname -a):
  Linux aks-agentpool-38312370-vmss000002 4.15.0-1091-azure df -h shows no mount for blobfuse container after kubernetes cluster upgrade to 1.19.7 in AKS #101~16.04.1-Ubuntu SMP Fri Jun 26 01:14:12 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

• Install tools:

• Others:
  blobfuse FlexVolume driver 1.0.13

[sauryadas]

@andyzhangx Does this look normal to you ?

[andyzhangx]

Flexvolume driver is in maintenance mode now, we suggest to use blobfuse CSI driver, it may mitigate this issue since one PVC on one node would only have one blobfuse mount, while for flexvolume driver, the num depends on pod num on the node.

[sauryadas]

thanks @andyzhangx how do we test memory consumption over time? is there a tool or capture it through top pods at periodic intervals

@JKrame FYI

[andyzhangx]

thanks @andyzhangx how do we test memory consumption over time? is there a tool or capture it through top pods at periodic intervals

@JKrame FYI

you may run "ps" command on agent node to get blobfuse process memory consumption, and save to local env. I am not sure whether there is a tool to do this, you may figure it out.

[andyzhangx]

I would say this is not a short term fix, try NFSv3 protocl on azure blob storage(alpha stage now), it should fix this high memory issue:
https://github.com/kubernetes-sigs/blobfuse-csi-driver/tree/master/deploy/example/nfs