NVMeTCP reconnects blocking node shutdown

[michaelbeaumont]

I'm running into openebs/mayastor#1764 at the moment:

I think the problem is the network is being brought down before the nvme devices.

My node to node communication and thus NVMeTCP connections run over Tailscale. So I believe it's coming down to Tailscale being taken down before volumes are unmounted, which leads to NVMeTCP reconnects blocking node shutdown.

There was previously unrelated discussion at #7287 (comment) about the idea of a "system extension", which to me could mean Tailscale would go down when "the network" goes down, i.e. after volumes are unmounted. Would that apply here potentially?

Is there a clever way to solve this with the current state of Talos/extensions?

[smira]

Yes, probably, but it's unclear as Tailscale might need some persistence as well (so volumes).

[michaelbeaumont]

Hmm should the watchdog be catching this scenario and restarting the node? If not, would adding that capability be a potential solution?