cat-log: infer --force-remote for --mode=list-dir if job.out not present

[ChrisPaulBennett]

This PR closes #6596.
There was an issue where if cat-log is ran in a small window of time between when a task finishes, but before the logs are copied locally, cat-log could display logs incorrectly. This change looks to see if the job.out log is present locally and if not, it defaults to remote retrieval.

Check List

• I have read CONTRIBUTING.md and added my name as a Code Contributor.
• Contains logically grouped changes (else tidy your branch by rebase).
• Does not contain off-topic changes (use other PRs for other changes).
• Applied any dependency changes to both setup.cfg (and conda-environment.yml if present).
• Tests are included (or explain why tests are not needed).
• Changelog entry included if this is a change that can affect users
• Cylc-Doc pull request opened if required at cylc/cylc-doc/pull/XXXX.
• If this is a bug fix, PR should be raised against the relevant ?.?.x branch.

[oliver-sanders]

This PR addresses the issue #6596.

FYI, if you change this message to closes #6596, then then GitHub magic will automatically link this PR to the issue and close it once the PR is merged.

[oliver-sanders]

LGTM

[ChrisPaulBennett]

Unless I've missed something, this code also closes #6597. Double check this though please.
I've also noticed that --mode=list-dir argument was missing from the test, so this has been added

[oliver-sanders]

FYI: The two issues are similar, but subtly different:

• cat-log: infer --force-remote for --mode=list-dir if job.out not present #6596 - is in relation to job log listing (if job.out not present).
• cat-log: infer --force-remote if log file not present locally #6597 - is in relation to job log viewing (if specified file not present).

[oliver-sanders]

I'm not sure this is actually going to the remote platform?

E.g, if I apply this diff:

diff --git a/cylc/flow/scripts/cat_log.py b/cylc/flow/scripts/cat_log.py
index e2ca4ad32..4af5e8536 100755
--- a/cylc/flow/scripts/cat_log.py
+++ b/cylc/flow/scripts/cat_log.py
@@ -611,6 +611,7 @@ def _main(
                 # (Ctrl-C while tailing)
                 # NOTE: This will raise NoHostsError if the platform is not
                 # contactable
+                LOG.critical("Goin' remote")
                 proc = remote_cylc_cmd(
                     cmd,
                     platform,

It says it is going to the remote platform, but it doesn't log the newly added message, suggesting that it doesn't actually go remote?

$ cylc cat remote//1/remote -f e --debug
DEBUG - Loading site/user config files
DEBUG - Reading file /home/users/metomi/apps/metomi-site/etc/cylc/flow/8/global.cylc
DEBUG - Processing with Jinja2
DEBUG - Setting Jinja2 template variables:
    + CYLC_TEMPLATE_VARS={'CYLC_VERSION': '8.6.0.dev', 'CYLC_TEMPLATE_VARS': {...}}
    + CYLC_VERSION=8.6.0.dev
DEBUG - Reading file /home/users/oliver.sanders/.cylc/flow/global.cylc
DEBUG - Processing with Jinja2
DEBUG - Setting Jinja2 template variables:
    + CYLC_TEMPLATE_VARS={'CYLC_VERSION': '8.6.0.dev', 'CYLC_TEMPLATE_VARS': {...}}
    + CYLC_VERSION=8.6.0.dev
DEBUG - Reading file /home/users/oliver.sanders/.cylc/flow/8/global.cylc
DEBUG - job.out not present, getting job log remotely
File not found: /home/users/oliver.sanders/cylc-run/remote/run1/log/job/1/remote/NN/job.err

[ChrisPaulBennett]

I'm not sure this is actually going to the remote platform?

E.g, if I apply this diff:

diff --git a/cylc/flow/scripts/cat_log.py b/cylc/flow/scripts/cat_log.py
index e2ca4ad32..4af5e8536 100755
--- a/cylc/flow/scripts/cat_log.py
+++ b/cylc/flow/scripts/cat_log.py
@@ -611,6 +611,7 @@ def _main(
                 # (Ctrl-C while tailing)
                 # NOTE: This will raise NoHostsError if the platform is not
                 # contactable
+                LOG.critical("Goin' remote")
                 proc = remote_cylc_cmd(
                     cmd,
                     platform,

It says it is going to the remote platform, but it doesn't log the newly added message, suggesting that it doesn't actually go remote?

$ cylc cat remote//1/remote -f e --debug
DEBUG - Loading site/user config files
DEBUG - Reading file /home/users/metomi/apps/metomi-site/etc/cylc/flow/8/global.cylc
DEBUG - Processing with Jinja2
DEBUG - Setting Jinja2 template variables:
    + CYLC_TEMPLATE_VARS={'CYLC_VERSION': '8.6.0.dev', 'CYLC_TEMPLATE_VARS': {...}}
    + CYLC_VERSION=8.6.0.dev
DEBUG - Reading file /home/users/oliver.sanders/.cylc/flow/global.cylc
DEBUG - Processing with Jinja2
DEBUG - Setting Jinja2 template variables:
    + CYLC_TEMPLATE_VARS={'CYLC_VERSION': '8.6.0.dev', 'CYLC_TEMPLATE_VARS': {...}}
    + CYLC_VERSION=8.6.0.dev
DEBUG - Reading file /home/users/oliver.sanders/.cylc/flow/8/global.cylc
DEBUG - job.out not present, getting job log remotely
File not found: /home/users/oliver.sanders/cylc-run/remote/run1/log/job/1/remote/NN/job.err

This is caused by this commit
a7d9fca

log_is_remote is always coming up as false and causing remote to not happen
Can you remind me what the logic flaw was?

[oliver-sanders]

Can you remind me what the logic flaw was?

There's a logical flaw?

I think the remote functionality is still working on master right?

[wxtim]

• Needs a changelog entry.
• I have questions about the logic
• I've left some style thoughts. None of them should block this PR>

[oliver-sanders]

Have converted to "draft" to assist with review prioritisation whilst Tim's comment above is addressed.

[wxtim]

You should undraft.