Description
Proposal
Follow Consul's lead by adding support for the experimental Raft WAL LogStore.
Use-cases
One of the most pathological failure conditions of Nomad today is when a user tries to add a new server to a busy cluster, and the server falls into an infinite loop of syncing snapshots. This can occur when so many changes occur during the syncing of the snapshot, that the new server cannot start streaming Raft log entries and must try to resync.
To avoid this we provide a configuration parameter to control how many logs to keep around: https://developer.hashicorp.com/nomad/docs/configuration/server#raft_trailing_logs
Disk space is cheap, so why not bump that default to like 1_000_000 (or at least recommend large clusters bump the default)?
Because the current (as of 1.7.0-dev) WAL backend is in BoltDB which introduces a new performance cliff to watch out for.
The experimental WAL LogStore is the purpose-built solution for this, and the intended future Raft WAL backend of all HashiCorp's Raft-backed projects.
Attempted Solutions
The easiest existing workaround is to reject job registration by setting -reject-job-registration during server operations to pause deployments and dramatically reduce the load on servers.
Not only is this not a panacea (node updates or other raft writes may be sufficient to cause the infinite snapshot loop problem), but it's also not without risks: adding new servers in busy clusters can take many minutes and is difficult to fully automate. Upgrading 5 servers for a large cluster could take 90 minutes. The longer scheduling remains paused, the greater the risk that a necessary deployment is unable to go out.
The risk is somewhat mitigated by the fact that rescheduling of failed/lost allocs _can still occur when job registration is paused.