start_time/end_time reflects only the last execution segment for tests with reboots

[bgoncalv]

Summary

For tests that use tmt-reboot or restart_on_exit_code, the start_time and
end_time fields on the result are silently overwritten on each re-execution of
the test. The final values reflect only the last execution segment (after the
final reboot/restart), not the original start of the test.

Root cause

In tmt/steps/execute/internal.py, the _run_tests loop reuses the same
TestInvocation object across reboots/restarts (via continue without
incrementing the index). Each call to invoke_test() runs a new Stopwatch
and unconditionally overwrites self.start_time and self.end_time on the
invocation:

self.start_time = timer.start_time_formatted
self.end_time = timer.end_time_formatted

[LecrisUT]

@happz do you remember when the duration was calculated across reboot there was a similar question if it should count all tmt preparation steps booting, etc. How does that context fit into this? What should the duration indicate, total time of all test executions?

[happz]

@happz do you remember when the duration was calculated across reboot there was a similar question if it should count all tmt preparation steps booting, etc. How does that context fit into this? What should the duration indicate, total time of all test executions?

Yes, a duration of a test - both duration key as a limit and "duration" as the measured time - covers the whole period of the test, no matter how many reboots it performed, or how much time was consumed by push/pulls done by tmt on behalf of the test.

Similarly, a plan duration (we don't have it yet, but Soon (TM)) would measure the duration of plan, no matter how tests were intermixed with pushes and pulls or how many prepare phases ran.

I think we could introduce some form of timestamps, to record interesting points in test/plan/run lifetime, e.g. times when the guest was restarted. Sort of a list of events. But even with that, duration as a limit should be honored across reboots, and start_time/end_time should record points when the test (plan/run) started, and when it ended, respectively. Across reboots and restarts.

I have a rough draft in my git stash - not sure why I haven't made a PR out of it already - but it should address this problem. It's a known issue - to me, not to users, I was aware of this deficiency when I was fixing the duration handling over reboots. Now someone else noticed too...

Edit: the PR draft: #4928 - needs a test.