Reducing the Fence File Recovery time

[bobo]

We are having issues with the Fence file blocking for too long after an abrupt pod failure. 5 minutes will fill up any buffers we can have multiple times over.
I think it should be possible to lower the time quite alot in some different ways, but im open for suggestions.

Im leaning towards the first option here, since it feels easiest. Although i think the Kubernetes Leases is the more correct option.

Options
Heartbeat timestamp in fence file

• Pod writes {pod_name}:{timestamp}, updates periodically
• New pod detects stale timestamp (~30s) and takes over
• Simple, no external dependencies

Kubernetes Lease objects

• Use native k8s coordination primitive
• Automatic expiry built-in
• Requires API access from data plane

Configurable timeout

• Just add a config option to wait for a shorter time (with added risks)

Add pod identity to the file

• Check if the pod owning the file is still alive, and try to take over if it is not.

Id be happy to implement any of them if it is something that is wanted.

[vigith]

The fence file you are talking about, is it the WAL fence or K8s controller fence? This is WAL fence where we wait for 5mins.

[bobo]

yes sorry i should have been more specific!

[vigith]

What about we make it configurable using an ENV var?

Also, now that we have ReadWriteOncePod (#2865) perhaps we could remove the fence file altogether.

For security reasons, we try our best not to make K8s API calls.

[bobo]

Ah yes i forgot about the existance of ReadWriteOncePod! With that i agree, i dont see any reason for a fence file.
Maybe jus disable it when ReadWriteOncePod is used? if i understand correct it is not the default choice.

[vigith]

Created an issue from this discussion: #3192