Feature Request: Adding execution timeout

[dgkf]

To support a safer failure mode for the execution engine (GitHub CI/CD), we thought that it might be useful for val.meter to make an effort to wrap up evaluation if it's getting close to the timeout for a GH runner.

We discussed that failing gracefully is a priority. Even if we are only able to derive partial information, it is more important that we have some data about a package so that end users can make downstream decisions about a cohort of packages in a snapshot of CRAN, as opposed to having to roll back to an earlier snapshot for specific packages that failed to generate metrics.

Some ideas:

Set individual metric limits (using setTimeLimit)

If we abort metrics that would evaluate beyond the runner timeout (probably with a small buffer), we could then reserve enough time for us to return NA for any remaining metrics and produce reported metrics that are as useful as possible.

We could then write out a PACKAGES file or .Rds as usual at the end of the session

Set session limits (using setSessionTimeLimit)

If instead we just wanted to abort the entire session when we are nearing the timeout, we then have a different problem - how do we record the metrics that were able to be evaluated?

We could stream to a PACKAGES file during evaluation. As metrics finish evaluating, they can write out to this file so that it builds over the course of metric evaluation. Even if interrupted, we'd have a partial PACKAGES file that would still be as useful as possible.

[llrs-roche]

Adding to the context: When pipelines fail the Github API doesn't provide any information about them (not even the time spend on a given step). The information is available on Github for some time until it is erased. Keeping track of failures could help with the validation itself. For example we are currently validating on ubuntu, maybe a package only works on Windows.

I like the second approach to keep updating a file. But this has some bigger technical implications for how tools relate as per default calculating metrics would require an associated file path (and that to be writable). We could use a temporary directory to save the file and then move it when the limit is reached (on.exit()).

The PACKAGES file works well as metrics can be appended one after the other without too much overhead. However, so far on the pipelines I've keep a .rds that keeps the whole object. We might need a reader to select a .rds or PACKAGES files to generate the matrix used on the CI to generate reports.