Failing to mount in microk8s path 108 characters limit v2

[oriolcx]

/kind bug

NOTE: Not a filesystem bug in Mountpoint itself; this is about the CSI node plugin’s kubelet path handling.

What happened?
On Ubuntu's MicroK8s (snap install), kubelet’s root directory is long (/var/snap/microk8s/common/var/lib/kubelet). The node-driver-registrar places the registration socket that exceeds the Linux AF_UNIX pathname limit (107 bytes), so the registrar fails to bind the socket and the driver never registers with kubelet.

What you expected to happen?
The CSI node plugin should register successfully with kubelet on MicroK8s without manual reconfiguration of kubelet’s --root-dir.

How to reproduce it (as minimally and precisely as possible)?
Ubuntu 22.04.4 LTS

snap install microk8s --classic --channel=1.32
sudo usermod -a -G microk8s $USER
su - $USER
microk8s config > ~/.kube/config
helm repo add aws-mountpoint-s3-csi-driver https://awslabs.github.io/mountpoint-s3-csi-driver
helm repo update
helm upgrade --install aws-mountpoint-s3-csi-driver \
    --namespace kube-system \
    --version 2.0.0 \
    --set node.kubeletPath=/var/snap/microk8s/common/var/lib/kubelet aws-mountpoint-s3-csi-driver/aws-mountpoint-s3-csi-driver

Then, just mount a path in kubernetes. It worked without issues with v1.5, but it seems to fail with v2.0. After exploring a bit, it seems a problem with the length of the path.

Anything else we need to know?:
This appears to be a path-length interaction between MicroK8s’ snap root dir and the CSI registration socket naming scheme.

Environment

• Kubernetes version:
  Client Version: v1.33.4
  Kustomize Version: v5.6.0
  Server Version: v1.32.3
• Driver version:
  -2.0.0

[muddyfish]

Hi - thanks for spending the time to make bug report.

Whilst we don't officially support microk8s, I see this can impact other k8s implementations.
I'll keep this open for tracking purposes

[unexge]

Sharing a note for a possible fix to this problem.

There is a hard limit on the Unix socket path on Go/Linux, and any operation fails with invalid argument if the path is longer than 108 characters. This limitation is only on the path passed to the syscall as far as I know, and there is a method in the CSI Driver to try to make the path relative to shorten it:

mountpoint-s3-csi-driver/pkg/mountpoint/mountoptions/mount_options.go

Lines 191 to 216 in 020c70a

	// tryToMakeSockPathRelative tries to make `path` relative to the current working directory
	// if its longer than 108 characters. This is because Go returns `invalid argument` errors
	// if you try to do `net.Listen()` or `net.Dial()`, see https://github.com/golang/go/issues/6895 for more details.
	//
	// If the `path` is still longer than 108 characters after making it relative, this function emits a warning log.
	// If the `path` is not relative, this function just returns unmodified `path` and emits a warning log if needed.
	func tryToMakeSockPathRelative(path string) string {
	if len(path) < 108 {
	// Nothing to do, its already shorter than 108 characters
	return path
	}
	shortPath, err := relative(path)
	if err != nil {
	// Failed to make it relative to the current working directory,
	// just emit a warning if needed and return unmodified path.
	klog.Warningf("Length of Unix domain socket %q is larger than 108 characters, failed to make it relative to the current working directory to shorten it: %v\n", path, err)
	warnAboutLongUnixSocketPath(path)
	return path
	}
	// Successfully turned `path` into a relative path,
	// just return relative path and emit a warning if its still longer than 108 characters.
	warnAboutLongUnixSocketPath(shortPath)
	return shortPath
	}

Making the path relative doesn't work in this case probably (i.e., there is no common prefix on the CSI Driver's binary path and the Unix socket's path). I believe we can change kubelet's mount path in the CSI Driver's container to be fixed to /var/lib/kubelet regardless of the path outside. Here we use the same path as on the host while mounting kubelet:

mountpoint-s3-csi-driver/charts/aws-mountpoint-s3-csi-driver/templates/node.yaml

Line 112 in 020c70a

mountPath: {{ .Values.node.kubeletPath }}

. We should probably hard-code kubelet path within the container and remove this environment variable:

mountpoint-s3-csi-driver/charts/aws-mountpoint-s3-csi-driver/templates/node.yaml

Lines 76 to 77 in 020c70a

	- name: KUBELET_PATH
	value: {{ .Values.node.kubeletPath }}

.

That way we'd always get /var/lib/kubelet/pods/<pod-uuid>/volumes/kubernetes.io~empty-dir/comm/mount.sock which should be 106 characters always as UUIDs are 36 characters long, and we should be fine by one character 😄.

[tadiwa-aizen]

We've addressed this issue here. #656