Skip to content

Commit b1b09b5

Browse files
yerzhan7jet-tong
authored andcommitted
Document kubelet subPath mkdir race condition in troubleshooting guide
1 parent 5998b2d commit b1b09b5

1 file changed

Lines changed: 15 additions & 0 deletions

File tree

docs/TROUBLESHOOTING.md

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -93,6 +93,21 @@ There are some possible workarounds:
9393

9494
There is also [a feature request on Mountpoint](https://github.com/awslabs/mountpoint-s3/issues/1055) to improve this behaviour, and if the provided workarounds wouldn't work for you, we'd recommend adding +1 (via 👍 emoji on the original post) to help us to track interest on this feature.
9595

96+
## `subPath` directory creation fails with `file exists` error
97+
98+
When multiple Pods using the same S3 volume with the same `subPath` are scheduled to the same node simultaneously, you may see:
99+
100+
```
101+
CreateContainerConfigError: failed to create subPath directory for volumeMount "data" of container "app":
102+
cannot create directory /var/lib/kubelet/pods/.../mount/subdir: file exists
103+
```
104+
105+
This is a [known kubelet bug](https://github.com/kubernetes/kubernetes/issues/94198) where concurrent `subPath` directory creation races. It is more likely with the CSI driver v2 because multiple Pods [share a single Mountpoint instance](./MOUNTPOINT_POD_SHARING.md). A [fix](https://github.com/kubernetes/kubernetes/pull/134540) is pending upstream.
106+
107+
**Workarounds:**
108+
- Use the `--prefix` mount option instead of Kubernetes `subPath` (see above).
109+
- See the [upstream issue](https://github.com/kubernetes/kubernetes/issues/94198) for other workarounds.
110+
96111
## I'm using an S3 Outposts bucket and am getting 'The bucket does not exist' errors
97112

98113
When using S3 Outposts, it is required to include the full ARN for your Outpost bucket in the `bucketName` field.

0 commit comments

Comments
 (0)