Issues with "late" events

[dpmatthews]

There appear to be a number of issues with late events.
I've tested with this workflow:

[scheduling]
    initial cycle point = now
    [[dependencies]]
        [[[ R1]]]
            graph = t1 => t2
[runtime]
    [[t1]]
        script = sleep 70
    [[t2]]
        script = true
        [[[events]]]
            late offset = PT1M
            mail events = late

Issues:

1. Using master, the late event is not triggered until t2 is triggered.

2021-01-25T10:41:53Z INFO - [t2.20210125T1040Z] -triggered off ['t1.20210125T1040Z']
2021-01-25T10:41:53Z WARNING - [t2.20210125T1040Z] -late (late-time=2021-01-25T10:41:00Z)

This is not surprising given spawn on demand but is not the desired behaviour (and not the behaviour using Cylc 7).

2. Using Cylc 7.8.7 I get a traceback in log/suite/log:

2021-01-25T09:47:01Z ERROR - [Errno 2] No such file or directory: '/../cylc-run/test.late/log/job/20210125T0946Z/t2/00/job-activity.log'
	Traceback (most recent call last):
	  File "/../cylc-7.8.7/lib/cylc/task_events_mgr.py", line 84, in log_task_job_activity
	    with open(job_activity_log, "ab") as handle:
	IOError: [Errno 2] No such file or directory: '/../cylc-run/test.late/log/job/20210125T0946Z/t2/00/job-activity.log'

(I don't understand why there would be any attempt to access the job activity log.)
There is no traceback when using master but that may be related to the late triggering of the late event.

3. The notification email contains:

late: 20210125T0946Z/t2/00

The reference to the job name (/00) makes no sense to me.

4. late offset is not documented in the Cylc 7 Suite.rc Reference. It is only mentioned in
   https://cylc.github.io/doc/build/7.8.7/singlehtml/index.html#late-events

[hjoliver]

Re behaviour on master (from Element chat):

late events will not work under spawn-on-demand (Cylc 8) because they require a waiting task proxy spawned ahead but not yet satisfied. So even under spawn-on-submit (Cylc 7) they don't work as some might expect. If a task is multiple cycles behind, only the current instance in the task pool will be detected as late, not subsequent instances that are technically also late already.
The user guide alludes to this, saying that the scheduler can only detect lateness in tasks that it is currently "aware of".

We might be able to do it in Cylc 8 by checking late offset for tasks in the n=1 datastore.

[hjoliver]

The reference to the job name (/00) makes no sense to me.

without looking, that's probably the job submit number, which hasn't reached 01 yet since the late task is still waiting. (So in this context, it should not be used!)