Cannot stop an allocation on client in case of network partition

[akamensky]

Nomad version

Nomad v1.6.1
BuildDate 2023-07-21T13:49:42Z
Revision 515895c7690cdc72278018dc5dc58aca41204ccc

Operating system and Environment details

Fedora 37

Issue

Clients running on remote nodes may experience loss of connectivity to the servers. It is great that this does not interrupt running allocations on those clients. However there is no CLI or API call to make this client shutdown the allocation.

In case of simple allocations the process can be killed (or in case of docker as runtime it can be stopped using docker cli), but in many cases the shutdown may require additional steps for a "clean" state, i.e. calling some remote service to do something, or stopping all processes in allocation in specific order.

It would be great to be able to stop running allocations in a way defined in the job spec from cli when agent/client lost connectivity to the servers.

Reproduction steps

1. Start separate processes for server and client (btw terminology is confusing, some places it is called client, other places it is called agent, just pick one)
2. Deploy a job to run on a client (i.e. hello-world example template)
3. Kill server process or in some other way ensure that client can no longer communicate with server
4. Try to stop the allocation on client from CLI or API

Expected Result

There is documented and working way to shutdown any allocation running on disconnected client via CLI or API

Actual Result

There is no such method

Job file (if appropriate)

N/A

Nomad Server logs (if appropriate)

N/A

Nomad Client logs (if appropriate)

N/A

[jrasell]

Hi @akamensky and thanks for raising this issue.

The idea of stopping allocations via a client only API is an interesting one and not currently possible as all scheduling decisions must be made via the Nomad servers. In order to implement this type of feature, the Nomad client would need an API to accept such requests. Once received and acted upon, the client would need to cache the update until it can send this to the Nomad server cluster where it can be reconciled with any other job updates and written to disk.

It would be a complex feature to add, however, I can see the benefits in clusters with edge deployments which can go "offline" for significant amounts of time with local access. I'll mark this as an enhancement that requires more investigation.

[tgross]

Leaving a cross-reference to #14605 here, which is similar inasmuch as it would require all the same plumbing in the client agent to make work.