Address play and vr timeouts: ping first

[hjoliver]

Close #6846
Partially address #6261 (by documenting exactly what cylc vr does now, rather than changing its behaviour) [off-topic, punted to #6354]

We currently perform the detect_old_contact_file check before we resume a workflow (i.e. cylc play) and additionally forcylc vr.
With all other commands, we only perform this check if a network timeout has occurred. But in these situations we just do it anyway.
I don't think there is any good reason to perform these checks, we should be able to replace it with a simple cylc ping type check

The detect_old_contact_file check runs cylc psutil over ssh on a hard-wired 10 second timeout. With NFS latency issues on HPC, that may not be nearly long enough.

This change:

• do a (typically much faster) ping type check first, and only resort to detect_old_contact_file if that times out (resume: replace detect_old_contact_file check with a "ping" type check #6846)
• make the timeout configurable - it's still needed for orphaned contact files, and 10 sec ain't enough
• fix badly named exception ContactFileExists -> SchedulerAlive
  • (it is not raised if the contact file exists but the scheduler is not alive)
• (somewhat relatedly) remove the ability of cylc ping to detect running tasks (as opposed to running workflows)
  • this very old feature was untested and broken, so I can conclude no one was using it

Check List

• I have read CONTRIBUTING.md and added my name as a Code Contributor.
• Contains logically grouped changes (else tidy your branch by rebase).
• Does not contain off-topic changes (use other PRs for other changes).
• Applied any dependency changes to both setup.cfg (and conda-environment.yml if present).
• Tests are included (or explain why tests are not needed).
• Changelog entry included if this is a change that can affect users
• Cylc-Doc pull request opened if required at cylc/cylc-doc/pull/XXXX.
• If this is a bug fix, PR should be raised against the relevant ?.?.x branch.

[hjoliver]

This is somewhat-arguably not a bug fix; I can retarget to 8.6 if necessary, but it's needed pretty urgently (else I'll need patch each interim version installed at ESNZ, where we're running with this already).

[hjoliver]

(Changed description to not close #6261)

[hjoliver]

@oliver-sanders - any suggestions on how to test this?

[dwsutherland]

LGTM 👍 (also used in ESNZ operations)

[dwsutherland]

@oliver-sanders - any suggestions on how to test this?

The most you could do is unit test this, unless there's a way to make a schd/platform artificially non responsive?

[hjoliver]

Actually a good 'ol functional test is the easiest way to do this.

I got 100% patch coverage with one new one, tests/f/cylc-play/00-contact.t - check it out if you like, but probably no need (it passes, it's quick, it's only a test).

Merging with two approvals...