Unable to delete Dynamic Host Volumes from removed hosts

[BastienClement]

Nomad version

Nomad v1.10.0

Issue

Hi, I've posted on discuss first, but go no answer. Since it might as well be a bug (error 500), I'm trying my luck here.

I’ve created Dynamic Host Volumes (v1.10) on multiple hosts and some of these hosts then got removed from the cluster.

The UI still shows me these volumes, but without any associated “Node”. Obviously.

(this is an old screenshot, the UI now shows "16 days ago")

I tried removing those using the CLI, but to no avail.

$ nomad volume delete -type host -namespace system 17a3cc0d-c218-2122-9f0d-519c49f2b583
Error deleting volume: Unexpected response code: 500 (rpc error: Unknown node a9d343e2-b6c7-8636-f62c-c057828a7373)

Expected Result

I would expect those volumes to get garbage collected at the same time as the associated node. Or at least for the CLI to offer a way to force delete a leftover volume.

[jrasell]

Hi @BastienClement and thanks for raising this issue. I have been able to reproduce this locally using the following steps:

• register a single host volume
• stop the Nomad service on the host where the volume has been registered and wait for Nomad to consider the node down
• run the nomad system gc command
• try and remove the orphaned volume

I suspect we may want some form of force flag on the delete command/api, so we can bypass the client RPC error. I'll move this onto our backlog to look into further.

[tgross]

Having a force flag sounds like a great idea because the default GC window for nodes is really long. We definitely want to make sure that has warnings about potential data loss though -- just because a node is down doesn't mean it can't come back. GC'ing a volume when the node gets GC'd seems reasonable as well, but users who allow a node to get GC'd and then come back would need to re-register the DHV's on that node.