feat(cli): nomad job run: add --retry flag (default 3) with configurable backoff

[resmo]

Description

Adds a retry for errors when monitoring job status after a job run. Closes #12062

Testing & Reproduction steps

Unit tests added:
TBD

Links

N/A

Contributor Checklist

• Changelog Entry If this PR changes user-facing behavior, please generate and add a
  changelog entry using the make cl command.
• Testing Please add tests to cover any new functionality or to demonstrate bug fixes and
  ensure regressions will be caught.
• Documentation If the change impacts user-facing functionality such as the CLI, API, UI,
  and job configuration, please update the Nomad product documentation, which is stored in the
  web-unified-docs repo. Refer to the web-unified-docs contributor guide for docs guidelines.
  Please also consider whether the change requires notes within the upgrade
  guide. If you would like help with the docs, tag the nomad-docs team in this PR.

Reviewer Checklist

• Backport Labels Please add the correct backport labels as described by the internal
  backporting document.
• Commit Type Ensure the correct merge method is selected which should be "squash and merge"
  in the majority of situations. The main exceptions are long-lived feature branches or merges where
  history should be preserved.
• Enterprise PRs If this is an enterprise only PR, please add any required changelog entry
  within the public repository.

• If a change needs to be reverted, we will roll out an update to the code within 7 days.

Changes to Security Controls

No changes to security controls.

[hashicorp-cla-app]

All committers have signed the CLA.

[resmo]

Thank you for your submission! We require that all contributors sign our Contributor License Agreement ("CLA") before we can accept the contribution. Read and sign the agreement

Learn more about why HashiCorp requires a CLA and what the CLA includes

Have you signed the CLA already but the status is still pending? Recheck it.

I signed the CLA but the status is kept pending even though I clicked recheck. hmm...

[tgross]

I signed the CLA but the status is kept pending even though I clicked recheck. hmm...

That's because you're not the author of the PR, CoPilot is (see patch). CoPilot can't sign the CLA, only humans can. Read thru the AI policy and rebase your contribution once you've edited.

[resmo]

I signed the CLA but the status is kept pending even though I clicked recheck. hmm...

That's because you're not the author of the PR, CoPilot is (see patch). CoPilot can't sign the CLA, only humans can. Read thru the AI policy and rebase your contribution once you've edited.

I see, I didn't want to "hide" it, that is why I kept it and mention it in the PR. Anyways, recommited and pushed.

[tgross]

Hi @resmo! Unfortunately I don't this PR actually resolves the intended issue. There are a few high-level problems:

• The #12062 report is about the monitoring of the job. The RegisterOpts call gets an evaluation ID back on success, and then the monitor spins up to start polling for the deployment to appear (which is async), and then monitors progress of that deployment. This change does nothing to improve handling of errors that happen during monitoring.
• The PR doesn't retry only on transient errors but also on non-transient errors.
• The notion of retrying job registration is problematic, as this is not an idempotent operation. You could hypothetically write the new job to Raft and then have RPC forwarding fail. This would end up causing multiple Raft writes for the same job. But if you're using -check-index in that case, none of the retries will succeed.

I think there's value here but I think we'd want to avoid retrying the non-idempotent operations and focus on the monitoring that happens afterwards.