Impossible to restart a pending task.

[scyd-cb]

Nomad version

V1.0.2

Operating system and Environment details

CentOS 8

Issue

When a task is in pending state (because restart has been delayed for example due to too many attempts), It seems impossible to restart the pending task or even the full allocation. Only solution seems to stop the full allocation and re-trigger a new evaluation.

Reproduction steps

create a group with multiple tasks and a restart policy with a high interval
ex:
restart {
attempts = 1
delay = "15s"
interval = "24h"
mode = "delay"
}

kill one of the task twice. The first time Nomad will restart the task and the second time it will update the task to pending (reason="Exceeded allowed attempts, applying a delay" delay=23h56m36.226005802s).

At that point if we want to force task to restart via nomad cli or UI, it will fail with error:

nomad alloc restart 41b915e5 monitor
Failed to restart allocation:
Unexpected response code: 500 (Task not running)

Expected Result

Nomad should allow user to restart task without waiting for delay, in case issue has been fixed.

Actual Result

nomad alloc restart 41b915e5 monitor
Failed to restart allocation:
Unexpected response code: 500 (Task not running)

Job file (if appropriate)

Nomad Server logs (if appropriate)

Nomad Client logs (if appropriate)

[angrycub]

@SCYD The nomad job eval command allows you to reevaluate a job on-demand. If the restart/reschedule has a delay applied to it, you will need to use the -force-reschedule flag to have Nomad immediately evaluate it in spite of the delay.

[scyd-cb]

@angrycub the allocation is still running, but one of the task is in pending state waiting to be restarted after the delay time.
Hence triggering job -eval -force-reschedule does nothing.

Ideally we want to force restarting the pending task without bringing down the running task. An easy fix would be to reset the delay timer to 0 so it can restart the task immediately I guess.

[shoenig]

Thanks for filing this @SCYD, this opens up an interesting discussion around some very old internals of Nomad with regard to what "pending" covers. Currently a task is considered to be pending if it is starting up, as well as when it's in this cool-down period after failure. In either case, the fact that it's in pending is blocking the ability to do task restart. As you point out, it really doesn't make sense to block task restart-ability during the cool-down period.

We'd like to improve the task restart functionality to address this use-case, but doing it properly may involve splitting up the pending state into 2 discrete states - something like "startup" and "cool-down". A solution that keeps pending as-is but enables restarts during the cool-down period would be smaller in scope, but breaks the encapsulation of a task's state being informative of what the next state transitions are. We might do that anyway, just returning an error if the restart is sent during a startup.

[scyd-cb]

Thanks for accepting this change, do you have any idea of the timeline for this fix ? We will be using Nomad on Production and without that change, we would need to isolate each task in its own task group to prevent failing/pending tasks to affect their co-allocated tasks.

[johanforssell]

One year later and this is still a thing. A previously broken job will not be restarted because it's pending.

Any progress on this ticket?