Switch to monotonic time when calculating the delayed precommit timeout

[rach-id]

After merging the temporary change in #2619, we should fix the core issue which is using the monotonic time instead of the wall one.

Wall clock is a subjective clock that can forward/backward in time depending on circumstances. Daylight saving for example, timezones, and sometimes even the wall clock can be delayed.

Now the issue is in comet we use the wall clock to make it easier to handle time:

celestia-core/types/time/time.go

Line 17 in ca3d1c3

return t.Round(0).UTC()

But when we're handling the delayed precommit, we're using: time.Until() which uses the canonical clock:

celestia-core/consensus/state.go

Line 2122 in ca3d1c3

waitTime := time.Until(precommitVoteTime)

This results in the difference between the values. Because when you apply any operation on a canonical clock and a wall clock, the canonical clock is stripped of the canonical part and only the wall part is left.

This was introduced in tendermint/tendermint#2203 (comment) because the way time was handled in comet is only when serializing/deserializing data. But our delay use case needs to have spot on clocks.

[evan-forbes]

the canonical clock is stripped of the canonical part and only the wall part is left.

Wall clock is a subjective clock that can forward/backward in time depending on circumstances. Daylight saving for example, timezones, and sometimes even the wall clock can be delayed.

I think in our rush here we're mixing up our terminology which is causing some confusion.

the wall clock has the monotonic bit, which means its safe for accurate comparisons. The wall clock also drifts and gets updated frequently from servers.

the canonical time (the time in the header) does not have the monotonic bit, therefore should not be used for comparisons since there is no way to account for drift.

Daylight saving for example

as mentioned in the slack we're reading from UTC, but the internal representation is still just nano seconds since 1985 or whatever so it has no consequence in any scenario. The issue we were seeing on testnets with the new timing mechanism is just due to natural drifts in wall clocks and not using the monotonic bit as it doesn't exist in the time in the header