Skip to content

Commit 5e58107

Browse files
noonionoonio
authored
Report drift on "how out of sync" the node is (#2453)
<!-- Describe your change here --> Follow-up on #2290 🔑 **Motivation** > Closes #2393 When the node is catching up with the chain, it may reject client inputs with a generic “chain out of sync” message. While technically correct, this message provides no indication of **how far** the node is behind, nor how long a client might need to wait before it becomes usable again. Hydra nodes previously tracked only the latest observed slot, without the corresponding chain time. As a result, it was not possible to compute or expose the drift between wall-clock time and the chain's current time, limiting observability and client feedback. Our goal is to improve the client-facing diagnostics and observability, so external systems (like TUI) can now make more informed decisions based on the reported drift. 🔄 **Changes** * `NodeSynced` and `NodeUnsynced` state-changed events, and their corresponding server outputs, now carry the observed `chainTime` and the measured `drift` in seconds. * The `NodeState` now tracks the **current chain time** (`UTCTime`) alongside the current slot and its drift. * Rejected client inputs during catch-up now include the **computed drift** in the error message, e.g.: ``` chain out of sync, drift: <duration> ``` --- <!-- Consider each and tick it off one way or the other --> * [X] CHANGELOG updated or not needed * [X] Documentation updated or not needed * [X] Haddocks updated or not needed * [X] No new TODOs introduced or explained herafter
2 parents 23e93d9 + 444e48c commit 5e58107

File tree

29 files changed

+4497
-5537
lines changed

29 files changed

+4497
-5537
lines changed

‎CHANGELOG.md‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ changes.
1616

1717
- **BREAKING** A Hydra node will now start rejecting both network and client inputs once its view of the chain has been out of sync for more than 50% of the configured `--contestation-period`, based on **system wall-clock time**.
1818
- Added `NodeUnsynced` and `NodeSynced` state events and server outputs.
19-
- Added `RejectedInput` client message.
19+
- Added `RejectedInputBecauseUnsynced` client message.
2020
- The `Checkpoint` event, and consequently the `EventLogRotated` server output, now carry the different `NodeState` variants: `NodeInSync` or `NodeCatchingUp`.
2121
- `Greetings` message now also contains the hydra-node synced status to the chain backend.
2222
- See [Issue #2286](https://github.com/cardano-scaling/hydra/issues/2286) and [PR #2290](https://github.com/cardano-scaling/hydra/pull/2290).
@@ -43,6 +43,12 @@ changes.
4343
- `POST /snapshot` now returns the specific side-load validation failure instead of timing out [#2462](https://github.com/cardano-scaling/hydra/issues/2462).
4444
- Fixed the internal wallet fee estimation, which was more often than not using maximum plutus execution units. This reduces costs for initializing, open, etc. of a head by a factor of ~4x [#2473](https://github.com/cardano-scaling/hydra/pull/2473).
4545
- Fixed another race-condition around incremental commits/decommits [#2500](https://github.com/cardano-scaling/hydra/issues/2500)
46+
- **BREAKING** Improved reporting of chain synchronization status by exposing the node's chain time and drift.
47+
- `NodeSynced` and `NodeUnsynced` state-changed events, and their corresponding server outputs, now include the observed chain time and drift.
48+
- `NodeState` now tracks the latest observed chan slot in addition to the chain time (`UTCTime`) and its drift measured in seconds.
49+
- The `EventLogRotated` and `Checkpoint` state-changed event schemas have been updated accordingly.
50+
- Client inputs rejected in `HeadLogic` (via `RejectedInputBecauseUnsynced`) during catch-up now report how far the node is out of sync (drift).
51+
- See [Issue #2393](https://github.com/cardano-scaling/hydra/issues/2393).
4652

4753
## [1.2.0] - 2025.11.28
4854

‎hydra-cluster/src/Hydra/Cluster/Scenarios.hs‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2363,8 +2363,7 @@ waitsForChainInSyncAndSecure tracer workDir backend hydraScriptsTxId = do
23632363
send n3 $ input "NewTx" ["transaction" .= tx]
23642364

23652365
waitMatch 5 n3 $ \v -> do
2366-
guard $ v ^? key "tag" == Just "RejectedInput"
2367-
guard $ v ^? key "reason" == Just "chain out of sync"
2366+
guard $ v ^? key "tag" == Just "RejectedInputBecauseUnsynced"
23682367

23692368
-- Carol API notifies the node is back on sync with the chain
23702369
-- note this is tuned based on how long it takes to sync

‎hydra-cluster/src/HydraNode.hs‎

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ import CardanoNode (cliQueryProtocolParameters)
99
import Control.Concurrent.Async (forConcurrently_)
1010
import Control.Concurrent.Class.MonadSTM (modifyTVar', readTVarIO)
1111
import Control.Exception (Handler (..), IOException, catches)
12-
import Control.Lens ((?~))
12+
import Control.Lens ((?~), (^?))
1313
import Control.Monad.Class.MonadAsync (forConcurrently)
1414
import Data.Aeson (Value (..), object, (.=))
1515
import Data.Aeson qualified as Aeson
@@ -448,8 +448,8 @@ withPreparedHydraNodeInSync tracer workDir hydraNodeId runOptions action =
448448
where
449449
action' client = do
450450
getHydraBackend >>= \case
451-
DirectBackendType -> waitForNodesSynced tracer 5 $ client :| []
452-
BlockfrostBackendType -> waitForNodesSynced tracer 10 $ client :| []
451+
DirectBackendType -> waitForNodesSynced 5 $ client :| []
452+
BlockfrostBackendType -> waitForNodesSynced 10 $ client :| []
453453
action client
454454

455455
-- | Run a hydra-node with given 'RunOptions'.
@@ -575,10 +575,10 @@ waitForNodesDisconnected tracer delay clients =
575575
waitFor tracer delay (toList clients) $
576576
output "NetworkDisconnected" []
577577

578-
waitForNodesSynced :: Tracer IO HydraNodeLog -> NominalDiffTime -> NonEmpty HydraClient -> IO ()
579-
waitForNodesSynced tracer delay clients = do
580-
waitFor tracer delay (toList clients) $
581-
output "NodeSynced" []
578+
waitForNodesSynced :: NominalDiffTime -> NonEmpty HydraClient -> IO ()
579+
waitForNodesSynced delay clients = do
580+
waitForAllMatch delay (toList clients) $ \v -> do
581+
guard $ v ^? key "tag" == Just "NodeSynced"
582582

583583
data HydraNodeLog
584584
= HydraNodeCommandSpec {cmd :: Text}

‎hydra-node/golden/ReasonablySized (NodeState (Tx ConwayEra)).json‎

Lines changed: 4046 additions & 5203 deletions
Large diffs are not rendered by default.

‎hydra-node/golden/ServerOutput/EventLogRotated.json‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,11 @@
22
"samples": [
33
{
44
"checkpoint": {
5-
"currentSlot": 1,
5+
"chainPointTime": {
6+
"currentChainTime": "1864-05-09T21:40:40.53348049031Z",
7+
"currentSlot": 1,
8+
"drift": 1
9+
},
610
"headState": {
711
"contents": {
812
"chainState": {

‎hydra-node/golden/ServerOutput/NodeSynced.json‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,9 @@
11
{
22
"samples": [
33
{
4+
"chainSlot": 0,
5+
"chainTime": "1864-05-08T05:28:53.992054083031Z",
6+
"drift": 1,
47
"tag": "NodeSynced"
58
}
69
],

‎hydra-node/golden/ServerOutput/NodeUnsynced.json‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,9 @@
11
{
22
"samples": [
33
{
4+
"chainSlot": 1,
5+
"chainTime": "1864-05-08T22:18:19.751420222935Z",
6+
"drift": 1,
47
"tag": "NodeUnsynced"
58
}
69
],

‎hydra-node/golden/StateChanged/Checkpoint.json‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,13 +2,17 @@
22
"samples": [
33
{
44
"state": {
5-
"currentSlot": 1,
5+
"chainPointTime": {
6+
"currentChainTime": "1864-05-10T20:19:35.748949974178Z",
7+
"currentSlot": 1,
8+
"drift": 0
9+
},
610
"headState": {
711
"contents": {
812
"chainState": {
913
"recordedAt": {
10-
"blockHash": "0100000101010000000000010000010101010101000000010100010101000001",
11-
"slot": 0,
14+
"blockHash": "0101010100010100000000010101010101000000000001010000000000000100",
15+
"slot": 1,
1216
"tag": "ChainPoint"
1317
},
1418
"spendableUTxO": {

‎hydra-node/golden/StateChanged/NodeSynced.json‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,9 @@
11
{
22
"samples": [
33
{
4+
"chainSlot": 1,
5+
"chainTime": "1864-05-08T12:36:04.0996257156Z",
6+
"drift": 0,
47
"tag": "NodeSynced"
58
}
69
],

‎hydra-node/golden/StateChanged/NodeUnsynced.json‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,9 @@
11
{
22
"samples": [
33
{
4+
"chainSlot": 1,
5+
"chainTime": "1864-05-10T08:21:45.260807303839Z",
6+
"drift": 0,
47
"tag": "NodeUnsynced"
58
}
69
],

0 commit comments

Comments
 (0)