Volumes / PVCs are not getting moved between nodes

[hornjo]

TL;DR

In the recent weeks we experienced multiple outages and servers which got unresponsive. Which lead to pods with pvc using hetzner-csi not able to reschedule.

Expected behavior

The expected behavior would be that the hetzner-csi would be able to release the volume on the unresponsive server and mount it on the node where the pod gets scheduled on. Only with this behavior a real self-healing on kubernetes level is possible.

Observed behavior

The kubernetes cluster already was rescheduling the pods to a new node, as the unresponsive server was shown as notReady, but the hetzner-csi was not able to release the volume to mount it on node which the pod with the pvc is getting scheduled on.

Minimal working example

No response

Log output

"time=2025-11-03T15:10:58.598Z level=ERROR source=/home/runner/work/csi-driver/csi-driver/internal/app/app.go:300 msg=\"handler failed\" component=grpc-server err=\"rpc error: code = Internal desc = failed to publish volume: format of disk \\\"/dev/disk/by-id/scsi-0HC_Volume_10XXXXX84\\\" failed: type:(\\\"ext4\\\") target:(\\\"/var/lib/kubelet/pods/7XXXXXXXd-3394-4a94-a81e-875fXXXXXXX68/volumes/kubernetes.io~csi/pvc-7XXXXXX76-248c-4016-8b88-00XXXXXXXb0/mount\\\") options:(\\\"defaults\\\") errcode:(exit status 1) output:(mke2fs 1.47.2 (1-Jan-2025)\\nThe file /dev/disk/by-id/scsi-0HC_Volume_10XXXXX84 does not exist and no size was specified.\\n) \""
"time=2025-11-03T15:10:57.998Z level=ERROR source=/home/runner/work/csi-driver/csi-driver/internal/app/app.go:300 msg=\"handler failed\" component=grpc-server err=\"rpc error: code = Internal desc = failed to publish volume: format of disk \\\"/dev/disk/by-id/scsi-0HC_Volume_10XXXXX84\\\" failed: type:(\\\"ext4\\\") target:(\\\"/var/lib/kubelet/pods/7XXXXXXXd-3394-4a94-a81e-875fXXXXXXX68/volumes/kubernetes.io~csi/pvc-7XXXXXX76-248c-4016-8b88-00XXXXXXXb0/mount\\\") options:(\\\"defaults\\\") errcode:(exit status 1) output:(mke2fs 1.47.2 (1-Jan-2025)\\nThe file /dev/disk/by-id/scsi-0HC_Volume_10XXXXX84 does not exist and no size was specified.\\n) \""
"time=2025-11-03T15:10:56.733Z level=ERROR source=/home/runner/work/csi-driver/csi-driver/internal/app/app.go:300 msg=\"handler failed\" component=grpc-server err=\"rpc error: code = Internal desc = failed to publish volume: Get \\\"https://api.hetzner.cloud/v1/volumes/10XXXXX84\\\": context canceled\""
"time=2025-11-03T15:10:50.850Z level=ERROR source=/home/runner/work/csi-driver/csi-driver/internal/app/app.go:300 msg=\"handler failed\" component=grpc-server err=\"rpc error: code = Internal desc = failed to publish volume: Get \\\"https://api.hetzner.cloud/v1/volumes/10XXXXX37\\\": context canceled\""
"time=2025-11-03T15:10:40.585Z level=ERROR source=/home/runner/work/csi-driver/csi-driver/internal/app/app.go:300 msg=\"handler failed\" component=grpc-server err=\"rpc error: code = Internal desc = failed to unpublish volume: context canceled\""
"time=2025-11-03T15:07:46.166Z level=ERROR source=/home/runner/work/csi-driver/csi-driver/internal/app/app.go:300 msg=\"handler failed\" component=grpc-server err=\"rpc error: code = Internal desc = failed to publish volume: Get \\\"https://api.hetzner.cloud/v1/volumes/10XXXXX83\\\": context canceled\""
"time=2025-11-03T15:07:40.916Z level=ERROR source=/home/runner/work/csi-driver/csi-driver/internal/app/app.go:300 msg=\"handler failed\" component=grpc-server err=\"rpc error: code = Internal desc = failed to publish volume: Get \\\"https://api.hetzner.cloud/v1/volumes/10XXXXX84\\\": context canceled\""
"time=2025-11-03T15:07:40.633Z level=ERROR source=/home/runner/work/csi-driver/csi-driver/internal/app/app.go:300 msg=\"handler failed\" component=grpc-server err=\"rpc error: code = Internal desc = failed to publish volume: Get \\\"https://api.hetzner.cloud/v1/volumes/10XXXXX24\\\": context canceled\""
"time=2025-11-03T15:07:27.554Z level=ERROR source=/home/runner/work/csi-driver/csi-driver/internal/app/app.go:300 msg=\"handler failed\" component=grpc-server err=\"rpc error: code = Unavailable desc = failed to unpublish volume: server is locked\""
"time=2025-11-03T15:07:27.154Z level=ERROR source=/home/runner/work/csi-driver/csi-driver/internal/app/app.go:300 msg=\"handler failed\" component=grpc-server err=\"rpc error: code = FailedPrecondition desc = failed to publish volume: volume is attached\""
"time=2025-11-03T15:07:26.600Z level=ERROR source=/home/runner/work/csi-driver/csi-driver/internal/app/app.go:300 msg=\"handler failed\" component=grpc-server err=\"rpc error: code = FailedPrecondition desc = failed to publish volume: volume is attached\""
"time=2025-11-03T15:07:26.152Z level=ERROR source=/home/runner/work/csi-driver/csi-driver/internal/app/app.go:300 msg=\"handler failed\" component=grpc-server err=\"rpc error: code = FailedPrecondition desc = failed to publish volume: volume is attached\""
"time=2025-11-03T15:07:25.761Z level=ERROR source=/home/runner/work/csi-driver/csi-driver/internal/app/app.go:300 msg=\"handler failed\" component=grpc-server err=\"rpc error: code = FailedPrecondition desc = failed to publish volume: volume is attached\""
"time=2025-11-03T15:07:25.688Z level=ERROR source=/home/runner/work/csi-driver/csi-driver/internal/app/app.go:300 msg=\"handler failed\" component=grpc-server err=\"rpc error: code = FailedPrecondition desc = failed to publish volume: volume is attached\""
"time=2025-11-03T15:07:25.600Z level=ERROR source=/home/runner/work/csi-driver/csi-driver/internal/app/app.go:300 msg=\"handler failed\" component=grpc-server err=\"rpc error: code = FailedPrecondition desc = failed to publish volume: volume is attached\""
"time=2025-11-03T15:07:25.559Z level=ERROR source=/home/runner/work/csi-driver/csi-driver/internal/app/app.go:300 msg=\"handler failed\" component=grpc-server err=\"rpc error: code = Unavailable desc = failed to unpublish volume: server is locked\""
"time=2025-11-03T15:07:25.550Z level=ERROR source=/home/runner/work/csi-driver/csi-driver/internal/app/app.go:300 msg=\"handler failed\" component=grpc-server err=\"rpc error: code = FailedPrecondition desc = failed to publish volume: volume is attached\""
"time=2025-11-03T15:07:25.548Z level=ERROR source=/home/runner/work/csi-driver/csi-driver/internal/app/app.go:300 msg=\"handler failed\" component=grpc-server err=\"rpc error: code = Unavailable desc = failed to unpublish volume: server is locked\""
"time=2025-11-03T15:07:25.523Z level=ERROR source=/home/runner/work/csi-driver/csi-driver/internal/app/app.go:300 msg=\"handler failed\" component=grpc-server err=\"rpc error: code = Unavailable desc = failed to unpublish volume: server is locked\""
"time=2025-11-03T15:07:24.525Z level=ERROR source=/home/runner/work/csi-driver/csi-driver/internal/app/app.go:300 msg=\"handler failed\" component=grpc-server err=\"rpc error: code = Internal desc = failed to unpublish volume: context canceled\""
"time=2025-11-03T15:07:24.521Z level=ERROR source=/home/runner/work/csi-driver/csi-driver/internal/app/app.go:300 msg=\"handler failed\" component=grpc-server err=\"rpc error: code = Internal desc = failed to unpublish volume: context canceled\""
"time=2025-11-03T15:07:24.517Z level=ERROR source=/home/runner/work/csi-driver/csi-driver/internal/app/app.go:300 msg=\"handler failed\" component=grpc-server err=\"rpc error: code = Internal desc = failed to unpublish volume: context canceled\""
"time=2025-11-03T15:07:24.491Z level=ERROR source=/home/runner/work/csi-driver/csi-driver/internal/app/app.go:300 msg=\"handler failed\" component=grpc-server err=\"rpc error: code = Internal desc = failed to unpublish volume: context canceled\""
"time=2025-11-03T15:06:05.538Z level=ERROR source=/home/runner/work/csi-driver/csi-driver/internal/app/app.go:300 msg=\"handler failed\" component=grpc-server err=\"rpc error: code = Unavailable desc = failed to unpublish volume: server is locked\""
"time=2025-11-03T15:01:24.444Z level=ERROR source=/home/runner/work/csi-driver/csi-driver/internal/app/app.go:300 msg=\"handler failed\" component=grpc-server err=\"rpc error: code = Unavailable desc = failed to unpublish volume: server is locked\""
"time=2025-11-03T15:01:05.475Z level=ERROR source=/home/runner/work/csi-driver/csi-driver/internal/app/app.go:300 msg=\"handler failed\" component=grpc-server err=\"rpc error: code = Internal desc = failed to unpublish volume: context canceled\""
"time=2025-11-03T14:55:50.464Z level=ERROR source=/home/runner/work/csi-driver/csi-driver/internal/app/app.go:300 msg=\"handler failed\" component=grpc-server err=\"rpc error: code = Unavailable desc = failed to unpublish volume: server is locked\""
"time=2025-11-03T14:51:24.455Z level=ERROR source=/home/runner/work/csi-driver/csi-driver/internal/app/app.go:300 msg=\"handler failed\" component=grpc-server err=\"rpc error: code = Unavailable desc = failed to unpublish volume: server is locked\""

Additional information

hcloud-csi-2.16.0

[lukasmetzner]

Hey,

does the issue you describe occur after the outage was resolved?

release the volume on the unresponsive server and mount it on the node where the pod gets scheduled on

In case the public API is down/flaky, it might not be foreseeable how the API will handle an unmount of the Volume. If the Server is completely unreachable, you can see that the Volume can't be detached by the error message server is locked. Only if we can successfully detach the Volume from the Server we can mount it to a different Server.

In general the decision IF and WHEN a Volume should be attached/detached is done by the external-attacher, which then sends a gRPC request to our csi-controller to attach/detach the Volume from the Server. So any self-healing mechanism, by retrying these calls is very limited for us.

I will investigate a bit and see what options we might have.