Overwrite default-values

[fraenku]

Proposal

The restart and reschedule-stanza use some default-values depending on the type of the job (service, batch, etc).
The default service and system job restart policy is for example
restart { interval = "30m" attempts = 2 delay = "15s" mode = "fail" }
These default values are senseful in a lot of cases, but they may vary depending of the kind of jobs (critical jobs or experimental jobs) and the environment (testing, preprod, prod). Therefore it would be an interesting feature to have the possibility to overwrite these values.

Probably it would make sense for also other stanzas as well.

Use-cases

In Europe 'Energy' is quite an issue at the moment:
A lot of countries are afraid of not having enough energy (oil, gaz, elecricity) during the winter season.
Therefore a lot of companies/countries/household are thinking of ways to save energy.
In IT one way to save elecrtricty is to spare using resources like CPU. Therefore it could be interesting to have a reschedule-stanza with default-values unlimited = 'false' instead of unlimited = 'true'. This however is not possible today. The only possibility to change it, is to adapt all job-specification which can be quite rather laborious depending on the number of jobs a cluster is handling.

Attempted Solutions

Having default-values set in one of the configuration-files of nomad (/etc/nomad.d), for examle
restart { interval = "30m" attempts = 1 delay = "15s" mode = "fail" }

[jrasell]

Hi @fraenku and thanks for raising this issue. If I am correct in understanding, this would be similar to admission controllers in Kubernetes, whereby cluster operators could control default job specification injection values without the requirement for job developers to know about it.

I see a lot of potential in the idea but there would be a lot of complexity and size in any implementation. I'll mark this issue for further discussion internally, as it would need R&D time before accepting. Thanks again!

[fraenku]

I've no knowledge about Kubernetes, but yes, the idea would be that cluster operators could adapt the default job specification.

A concrete example:
The default-bevavior of the reschedule-stanza is to reschedule a job in a unlimited way. But if a job continues to fail after a few reschedule-attempts chances are high that rescheduling does not help and are therefore useless. As an operator I would like to adapt the behavior here, but I do not want to adapt all job specifaction one-by-one (which is a huge effort depending on the cluster-size), I prefer to adapt the default behavior.