Add distributed locks helper to jobspec

[mikenomitch]

Proposal

With Nomad adding basic CLI & API support for distributed locks in 1.7, it would be nice to automatically add something to the Nomad jobspec to set up a lock for the user. The lock could be set up on job run (if its not already) and then each allocation of the job could try to grab the lock. The result of the attempted lock-get could be thread thru to an env var and then templatized. Refreshing the lock could be done via the Nomad client or a separate process that Nomad spins up for the user.

Perhaps something like this:

job "application" {
  group "database" {
    count = 3
    lock {
  	path = "nomad/jobs/some-db/lock”
    }

    task "main" {
      driver = "docker"

       template {
	  data = <<EOF
# note sure how to actually use this ideally - could compare this to the current alloc to see if it is the leader?
LEADER_ALLOC = {{ with nomadVar "nomad/jobs/some-db/lock" }}{{ .alloc }}{{ end }}
   EOF
      }
    }
  }
}

Use-cases

This is probably most useful for leader election purposes. Running 1 job as a leader and N jobs as follower, then restarting a job as a leader if the old one goes down.