Skip to content

Commit 87d079e

Browse files
committed
Document kubelet subPath mkdir race condition in troubleshooting guide
1 parent 80122a3 commit 87d079e

1 file changed

Lines changed: 15 additions & 0 deletions

File tree

docs/TROUBLESHOOTING.md

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -93,6 +93,21 @@ There are some possible workarounds:
9393

9494
There is also [a feature request on Mountpoint](https://github.com/awslabs/mountpoint-s3/issues/1055) to improve this behaviour, and if the provided workarounds wouldn't work for you, we'd recommend adding +1 (via 👍 emoji on the original post) to help us to track interest on this feature.
9595

96+
## `subPath` directory creation fails with `file exists` error
97+
98+
When multiple Pods using the same S3 volume with the same `subPath` are scheduled to the same node simultaneously, you may see:
99+
100+
```
101+
CreateContainerConfigError: failed to create subPath directory for volumeMount "data" of container "app":
102+
cannot create directory /var/lib/kubelet/pods/.../mount/subdir: file exists
103+
```
104+
105+
This is a [known kubelet bug](https://github.com/kubernetes/kubernetes/issues/94198) where concurrent `subPath` directory creation races. It is more likely with the CSI driver v2 because multiple Pods [share a single Mountpoint instance](./MOUNTPOINT_POD_SHARING.md). A [fix](https://github.com/kubernetes/kubernetes/pull/134540) is pending upstream.
106+
107+
**Workarounds:**
108+
- Use the `--prefix` mount option instead of Kubernetes `subPath` (see above).
109+
- See the [upstream issue](https://github.com/kubernetes/kubernetes/issues/94198) for other workarounds.
110+
96111
## I'm using an S3 Outposts bucket and am getting 'The bucket does not exist' errors
97112

98113
When using S3 Outposts, it is required to include the full ARN for your Outpost bucket in the `bucketName` field.

0 commit comments

Comments
 (0)