reduce FSM backpressure from blocked evals queue

[tgross]

The coarse grained lock on the blocked evals queue can cause backpressure on the FSM when there are a large number of evals getting unblocked and there's contention from this lock from a large number of scheduler goroutines. The watchCapacity goroutine in the blocked evals queue has a large buffered channel for unblock operations, but it takes the same lock that's used by the unblock methods called from the FSM. Meanwhile, Eval.Reblock RPCs arriving from scheduler workers attempt to take this same lock, and we end up with a backlog waiting on this mutex.

This PR moves all the operations for the blocked evals queue onto a single goroutine that receives work from a large buffered channel. The Eval.Reblock RPCs and the Unblock methods called from the FSM push work onto this channel and immediately return. This prevents them from blocking except for during leader transitions where we flush the blocked evals queue, at which point we should not be making Unblock method calls from the FSM anyways.

This also allows us to move the tracking of stats into one goroutine so we no longer need to copy the stats on each update. This reduces memory allocation and GC pressure significantly.

Ref: https://hashicorp.atlassian.net/browse/NMD-1045

Testing & Reproduction steps

• WIP: I've got a good reproduction of the issue and the fix with benchmarking, but my test environment blew up with disk space issues so I've got to re-run that. Will wrap that up tomorrow.
• WIP: Looks like some of the worker and FSM tests are failing because we're not waiting for the future/channel that the Unblock methods are returning in the production code. It's going to be a bit of a pain to fix this in the tests.

Contributor Checklist

• Changelog Entry If this PR changes user-facing behavior, please generate and add a
  changelog entry using the make cl command.
• Testing Please add tests to cover any new functionality or to demonstrate bug fixes and
  ensure regressions will be caught.
• Documentation If the change impacts user-facing functionality such as the CLI, API, UI,
  and job configuration, please update the Nomad product documentation, which is stored in the
  web-unified-docs repo. Refer to the web-unified-docs contributor guide for docs guidelines.
  Please also consider whether the change requires notes within the upgrade
  guide. If you would like help with the docs, tag the nomad-docs team in this PR.

Reviewer Checklist

• Backport Labels Please add the correct backport labels as described by the internal
  backporting document.
• Commit Type Ensure the correct merge method is selected which should be "squash and merge"
  in the majority of situations. The main exceptions are long-lived feature branches or merges where
  history should be preserved.
• Enterprise PRs If this is an enterprise only PR, please add any required changelog entry
  within the public repository.

• If a change needs to be reverted, we will roll out an update to the code within 7 days.

Changes to Security Controls

Are there any changes to security controls (access controls, encryption, logging) in this pull request? If so, explain.