GossipService Time difference between us and x too great #5436
Description
Describe the bug
A clear and concise description of what the bug is.
It seems that, after starting the cluster, GossipService starts giving errors stating that the server times are off.
Whenever I restart any of the 3 nodes the issue is gone for some time and then reoccurs.
What is especially interesting is that the date-time of peers (that are supposedly out of sync) are not changing over time - only after one of the 3 nodes is restarted.
To Reproduce
Steps to reproduce the behavior:
- Start the servers
- Tail the logs
- Restart one of the nodes
- Issue gone for some time.
Expected behavior
A clear and concise description of what you expected to happen.
Since al the servers clocks are ok this should not happen (synced with chrony)
Actual behavior
A clear and concise description of what actually happened.
After rebooting one of the nodes the Gossip Service seems to restart - now with different (unchanging) peer times.
Config/Logs/Screenshots
5515,40,16:00:16.385,ERR] GossipServiceBase Time difference between us and [10.0.2.13:2113/kurrentdb-cluster.kurrent.internal] is too great! UTC now: 2026-01-05 16:00:16.385, peer's time stamp: 2026-01-05 15:49:39.412.
[ 5515,39,16:00:20.383,ERR] GossipServiceBase Time difference between us and [10.0.2.11:2113/kurrentdb-cluster.kurrent.internal] is too great! UTC now: 2026-01-05 16:00:20.383, peer's time stamp: 2026-01-05 15:49:39.412.
[ 5515,39,16:00:22.385,ERR] GossipServiceBase Time difference between us and [10.0.2.13:2113/kurrentdb-cluster.kurrent.internal] is too great! UTC now: 2026-01-05 16:00:22.385, peer's time stamp: 2026-01-05 15:49:39.412.
[ 5515,44,16:00:34.391,ERR] GossipServiceBase Time difference between us and [10.0.2.11:2113/kurrentdb-cluster.kurrent.internal] is too great! UTC now: 2026-01-05 16:00:34.391, peer's time stamp: 2026-01-05 15:49:39.412.
[ 5515,44,16:00:40.393,ERR] GossipServiceBase Time difference between us and [10.0.2.13:2113/kurrentdb-cluster.kurrent.internal] is too great! UTC now: 2026-01-05 16:00:40.393, peer's time stamp: 2026-01-05 15:49:39.412.
[ 5515,40,16:00:48.396,ERR] GossipServiceBase Time difference between us and [10.0.2.11:2113/kurrentdb-cluster.kurrent.internal] is too great! UTC now: 2026-01-05 16:00:48.396, peer's time stamp: 2026-01-05 15:49:39.412.
[ 5515,41,16:00:50.398,ERR] GossipServiceBase Time difference between us and [10.0.2.13:2113/kurrentdb-cluster.kurrent.internal] is too great! UTC now: 2026-01-05 16:00:50.398, peer's time stamp: 2026-01-05 15:49:39.412.
[ 5515,41,16:00:52.397,ERR] GossipServiceBase Time difference between us and [10.0.2.11:2113/kurrentdb-cluster.kurrent.internal] is too great! UTC now: 2026-01-05 16:00:52.397, peer's time stamp: 2026-01-05 15:49:39.412.
[ 5515,41,16:00:56.400,ERR] GossipServiceBase Time difference between us and [10.0.2.13:2113/kurrentdb-cluster.kurrent.internal] is too great! UTC now: 2026-01-05 16:00:56.400, peer's time stamp: 2026-01-05 15:49:39.412.
[ 5515,38,16:00:58.400,ERR] GossipServiceBase Time difference between us and [10.0.2.13:2113/kurrentdb-cluster.kurrent.internal] is too great! UTC now: 2026-01-05 16:00:58.400, peer's time stamp: 2026-01-05 15:49:39.412.
[ 5515, 6,16:01:00.400,ERR] GossipServiceBase Time difference between us and [10.0.2.11:2113/kurrentdb-cluster.kurrent.internal] is too great! UTC now: 2026-01-05 16:01:00.400, peer's time stamp: 2026-01-05 15:49:39.412.
[ 5515, 6,16:01:02.401,ERR] GossipServiceBase Time difference between us and [10.0.2.11:2113/kurrentdb-cluster.kurrent.internal] is too great! UTC now: 2026-01-05 16:01:02.401, peer's time stamp: 2026-01-05 15:49:39.412.
[ 5515,38,16:01:08.408,ERR] GossipServiceBase Time difference between us and [10.0.2.11:2113/kurrentdb-cluster.kurrent.internal] is too great! UTC now: 2026-01-05 16:01:08.408, peer's time stamp: 2026-01-05 15:49:39.412.
[ 5515,41,16:01:12.410,ERR] GossipServiceBase Time difference between us and [10.0.2.11:2113/kurrentdb-cluster.kurrent.internal] is too great! UTC now: 2026-01-05 16:01:12.410, peer's time stamp: 2026-01-05 15:49:39.412.
[ 5515,44,16:01:14.411,ERR] GossipServiceBase Time difference between us and [10.0.2.11:2113/kurrentdb-cluster.kurrent.internal] is too great! UTC now: 2026-01-05 16:01:14.410, peer's time stamp: 2026-01-05 15:49:39.412.
[ 5515,44,16:01:16.412,ERR] GossipServiceBase Time difference between us and [10.0.2.12:2113/kurrentdb-cluster.kurrent.internal] is too great! UTC now: 2026-01-05 16:01:16.412, peer's time stamp: 2026-01-05 15:49:39.412.
Restart one of the nodes (notice the new - but stuck - peer timestamp below)
[ 5515,44,16:01:17.926,INF] EventStoreClusterClient View Change Proof Send Failed to Unspecified/kurrentdb-03.kurrent.internal:2113
System.AggregateException: One or more errors occurred. (Status(StatusCode="Unavailable", Detail="Error starting gRPC call. HttpRequestException: Connection refused (kurrentdb-03.kurrent.internal:2113) SocketException: Connection refused", DebugException="System.Net.Http.HttpRequestException: Connection refused (kurrentdb-03.kurrent.internal:2113)"))
---> Grpc.Core.RpcException: Status(StatusCode="Unavailable", Detail="Error starting gRPC call. HttpRequestException: Connection refused (kurrentdb-03.kurrent.internal:2113) SocketException: Connection refused", DebugException="System.Net.Http.HttpRequestException: Connection refused (kurrentdb-03.kurrent.internal:2113)")
---> System.Net.Http.HttpRequestException: Connection refused (kurrentdb-03.kurrent.internal:2113)
---> System.Net.Sockets.SocketException (111): Connection refused
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|285_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.AddHttp2ConnectionAsync(QueueItem queueItem)
at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
at Grpc.Net.Client.Internal.GrpcCall`2.RunCall(HttpRequestMessage request, Nullable`1 timeout)
--- End of inner exception stack trace ---
at EventStore.Core.Cluster.EventStoreClusterClient.SendViewChangeProofAsync(Guid serverId, EndPoint serverHttpEndPoint, Int32 installedView, DateTime deadline) in /home/runner/work/TrainStation/TrainStation/KurrentDB/src/KurrentDB.Core/Cluster/EventStoreClusterClient.Elections.cs:line 120
--- End of inner exception stack trace ---
[ 5515,39,16:01:18.412,ERR] GossipServiceBase Time difference between us and [10.0.2.11:2113/kurrentdb-cluster.kurrent.internal] is too great! UTC now: 2026-01-05 16:01:18.412, peer's time stamp: 2026-01-05 15:49:39.412.
[ 5515,41,16:01:20.414,INF] GossipServiceBase CLUSTER HAS CHANGED gossip received from [10.0.2.12:2113/kurrentdb-cluster.kurrent.internal]
Old:
["Priority: 0 VND {355db192-0743-4c7f-b7e8-4dbed3153271} <LIVE> [ReadOnlyReplica, 10.0.2.21:1112/kurrentdb-cluster.kurrent.internal, n/a, n/a, n/a, Unspecified/kurrentdb-replica.kurrent.internal:2113, (ADVERTISED: HTTP::0, TCP::0), Version: 25.1.0.2664] 3788/4152/4152/E6@3510:{37341e58-67a6-4b76-a5f9-4cba7b329d93} | 2026-01-05 16:01:15.053","Priority: 0 VND {0de61e59-ad02-4594-90a5-055b30f32d8d} <LIVE> [Follower, 10.0.2.13:1112/kurrentdb-cluster.kurrent.internal, n/a, n/a, n/a, Unspecified/kurrentdb-03.kurrent.internal:2113, (ADVERTISED: HTTP::0, TCP::0), Version: 25.1.0.2664] 3788/4152/4152/E6@3510:{37341e58-67a6-4b76-a5f9-4cba7b329d93} | 2026-01-05 16:01:15.811","Priority: 0 VND {d700b7bd-15ca-4bf8-a738-1293d34946d1} <LIVE> [Leader, 10.0.2.12:1112/kurrentdb-cluster.kurrent.internal, n/a, n/a, n/a, Unspecified/kurrentdb-02.kurrent.internal:2113, (ADVERTISED: HTTP::0, TCP::0), Version: 25.1.0.2664] 3788/4152/4152/E6@3510:{37341e58-67a6-4b76-a5f9-4cba7b329d93} | 2026-01-05 16:01:16.412","Priority: 0 VND {111ae1d8-11f8-4f55-a108-2563dbe549ae} <LIVE> [Follower, Unspecified/10.0.2.11:1112, n/a, n/a, n/a, Unspecified/kurrentdb-01.kurrent.internal:2113, (ADVERTISED: HTTP::0, TCP::0), Version: 25.1.0.2664] 3788/4152/4152/E6@3510:{37341e58-67a6-4b76-a5f9-4cba7b329d93} | 2026-01-05 16:01:20.412","MAN {00000000-0000-0000-0000-000000000000} <LIVE> [Manager, 10.0.2.13:2113/kurrentdb-cluster.kurrent.internal] | 2026-01-05 15:49:39.412","MAN {00000000-0000-0000-0000-000000000000} <LIVE> [Manager, 10.0.2.12:2113/kurrentdb-cluster.kurrent.internal] | 2026-01-05 15:49:39.412","MAN {00000000-0000-0000-0000-000000000000} <LIVE> [Manager, 10.0.2.11:2113/kurrentdb-cluster.kurrent.internal] | 2026-01-05 15:49:39.412"]
New:
["Priority: 0 VND {355db192-0743-4c7f-b7e8-4dbed3153271} <LIVE> [ReadOnlyReplica, 10.0.2.21:1112/kurrentdb-cluster.kurrent.internal, n/a, n/a, n/a, Unspecified/kurrentdb-replica.kurrent.internal:2113, (ADVERTISED: HTTP::0, TCP::0), Version: 25.1.0.2664] 3788/4152/4152/E6@3510:{37341e58-67a6-4b76-a5f9-4cba7b329d93} | 2026-01-05 16:01:19.508","Priority: 0 VND {76717cd4-a2fb-450f-856a-a1ff819887f0} <LIVE> [DiscoverLeader, 10.0.2.13:1112/kurrentdb-cluster.kurrent.internal, n/a, n/a, n/a, Unspecified/kurrentdb-03.kurrent.internal:2113, (ADVERTISED: HTTP::0, TCP::0), Version: 25.1.0.2664] 3788/4152/4152/E6@3510:{37341e58-67a6-4b76-a5f9-4cba7b329d93} | 2026-01-05 16:01:20.106","Priority: 0 VND {d700b7bd-15ca-4bf8-a738-1293d34946d1} <LIVE> [Leader, 10.0.2.12:1112/kurrentdb-cluster.kurrent.internal, n/a, n/a, n/a, Unspecified/kurrentdb-02.kurrent.internal:2113, (ADVERTISED: HTTP::0, TCP::0), Version: 25.1.0.2664] 3788/4152/4152/E6@3510:{37341e58-67a6-4b76-a5f9-4cba7b329d93} | 2026-01-05 16:01:20.414","Priority: 0 VND {111ae1d8-11f8-4f55-a108-2563dbe549ae} <LIVE> [Follower, Unspecified/10.0.2.11:1112, n/a, n/a, n/a, Unspecified/kurrentdb-01.kurrent.internal:2113, (ADVERTISED: HTTP::0, TCP::0), Version: 25.1.0.2664] 3788/4152/4152/E6@3510:{37341e58-67a6-4b76-a5f9-4cba7b329d93} | 2026-01-05 16:01:20.414","MAN {00000000-0000-0000-0000-000000000000} <LIVE> [Manager, 10.0.2.13:2113/kurrentdb-cluster.kurrent.internal] | 2026-01-05 16:01:20.422","MAN {00000000-0000-0000-0000-000000000000} <LIVE> [Manager, 10.0.2.12:2113/kurrentdb-cluster.kurrent.internal] | 2026-01-05 16:01:20.422","MAN {00000000-0000-0000-0000-000000000000} <LIVE> [Manager, 10.0.2.11:2113/kurrentdb-cluster.kurrent.internal] | 2026-01-05 16:01:20.422"]
[ 5515,44,16:01:20.546,INF] GossipServiceBase CLUSTER HAS CHANGED gossip received from [Unspecified/kurrentdb-03.kurrent.internal:2113]
Old:
["Priority: 0 VND {355db192-0743-4c7f-b7e8-4dbed3153271} <LIVE> [ReadOnlyReplica, 10.0.2.21:1112/kurrentdb-cluster.kurrent.internal, n/a, n/a, n/a, Unspecified/kurrentdb-replica.kurrent.internal:2113, (ADVERTISED: HTTP::0, TCP::0), Version: 25.1.0.2664] 3788/4152/4152/E6@3510:{37341e58-67a6-4b76-a5f9-4cba7b329d93} | 2026-01-05 16:01:19.508","Priority: 0 VND {76717cd4-a2fb-450f-856a-a1ff819887f0} <LIVE> [DiscoverLeader, 10.0.2.13:1112/kurrentdb-cluster.kurrent.internal, n/a, n/a, n/a, Unspecified/kurrentdb-03.kurrent.internal:2113, (ADVERTISED: HTTP::0, TCP::0), Version: 25.1.0.2664] 3788/4152/4152/E6@3510:{37341e58-67a6-4b76-a5f9-4cba7b329d93} | 2026-01-05 16:01:20.106","Priority: 0 VND {d700b7bd-15ca-4bf8-a738-1293d34946d1} <LIVE> [Leader, 10.0.2.12:1112/kurrentdb-cluster.kurrent.internal, n/a, n/a, n/a, Unspecified/kurrentdb-02.kurrent.internal:2113, (ADVERTISED: HTTP::0, TCP::0), Version: 25.1.0.2664] 3788/4152/4152/E6@3510:{37341e58-67a6-4b76-a5f9-4cba7b329d93} | 2026-01-05 16:01:20.414","Priority: 0 VND {111ae1d8-11f8-4f55-a108-2563dbe549ae} <LIVE> [Follower, Unspecified/10.0.2.11:1112, n/a, n/a, n/a, Unspecified/kurrentdb-01.kurrent.internal:2113, (ADVERTISED: HTTP::0, TCP::0), Version: 25.1.0.2664] 3788/4152/4152/E6@3510:{37341e58-67a6-4b76-a5f9-4cba7b329d93} | 2026-01-05 16:01:20.414","MAN {00000000-0000-0000-0000-000000000000} <LIVE> [Manager, 10.0.2.13:2113/kurrentdb-cluster.kurrent.internal] | 2026-01-05 16:01:20.422","MAN {00000000-0000-0000-0000-000000000000} <LIVE> [Manager, 10.0.2.12:2113/kurrentdb-cluster.kurrent.internal] | 2026-01-05 16:01:20.422","MAN {00000000-0000-0000-0000-000000000000} <LIVE> [Manager, 10.0.2.11:2113/kurrentdb-cluster.kurrent.internal] | 2026-01-05 16:01:20.422"]
New:
["Priority: 0 VND {355db192-0743-4c7f-b7e8-4dbed3153271} <LIVE> [ReadOnlyReplica, 10.0.2.21:1112/kurrentdb-cluster.kurrent.internal, n/a, n/a, n/a, Unspecified/kurrentdb-replica.kurrent.internal:2113, (ADVERTISED: HTTP::0, TCP::0), Version: 25.1.0.2664] 3788/4152/4152/E6@3510:{37341e58-67a6-4b76-a5f9-4cba7b329d93} | 2026-01-05 16:01:19.508","Priority: 0 VND {76717cd4-a2fb-450f-856a-a1ff819887f0} <LIVE> [Follower, 10.0.2.13:1112/kurrentdb-cluster.kurrent.internal, n/a, n/a, n/a, Unspecified/kurrentdb-03.kurrent.internal:2113, (ADVERTISED: HTTP::0, TCP::0), Version: 25.1.0.2664] 3788/4152/4152/E6@3510:{37341e58-67a6-4b76-a5f9-4cba7b329d93} | 2026-01-05 16:01:21.092","Priority: 0 VND {d700b7bd-15ca-4bf8-a738-1293d34946d1} <LIVE> [Leader, 10.0.2.12:1112/kurrentdb-cluster.kurrent.internal, n/a, n/a, n/a, Unspecified/kurrentdb-02.kurrent.internal:2113, (ADVERTISED: HTTP::0, TCP::0), Version: 25.1.0.2664] 3788/4152/4152/E6@3510:{37341e58-67a6-4b76-a5f9-4cba7b329d93} | 2026-01-05 16:01:20.545","Priority: 0 VND {111ae1d8-11f8-4f55-a108-2563dbe549ae} <LIVE> [Follower, Unspecified/10.0.2.11:1112, n/a, n/a, n/a, Unspecified/kurrentdb-01.kurrent.internal:2113, (ADVERTISED: HTTP::0, TCP::0), Version: 25.1.0.2664] 3788/4152/4152/E6@3510:{37341e58-67a6-4b76-a5f9-4cba7b329d93} | 2026-01-05 16:01:20.545","MAN {00000000-0000-0000-0000-000000000000} <LIVE> [Manager, 10.0.2.13:2113/kurrentdb-cluster.kurrent.internal] | 2026-01-05 16:01:20.422","MAN {00000000-0000-0000-0000-000000000000} <LIVE> [Manager, 10.0.2.12:2113/kurrentdb-cluster.kurrent.internal] | 2026-01-05 16:01:20.422","MAN {00000000-0000-0000-0000-000000000000} <LIVE> [Manager, 10.0.2.11:2113/kurrentdb-cluster.kurrent.internal] | 2026-01-05 16:01:20.422"]
[ 5515,41,16:02:20.437,ERR] GossipServiceBase Time difference between us and [10.0.2.11:2113/kurrentdb-cluster.kurrent.internal] is too great! UTC now: 2026-01-05 16:02:20.437, peer's time stamp: 2026-01-05 16:01:20.422.
[ 5515,38,16:02:26.440,ERR] GossipServiceBase Time difference between us and [10.0.2.11:2113/kurrentdb-cluster.kurrent.internal] is too great! UTC now: 2026-01-05 16:02:26.440, peer's time stamp: 2026-01-05 16:01:20.422.
[ 5515,38,16:02:32.443,ERR] GossipServiceBase Time difference between us and [10.0.2.12:2113/kurrentdb-cluster.kurrent.internal] is too great! UTC now: 2026-01-05 16:02:32.443, peer's time stamp: 2026-01-05 16:01:20.422.
[ 5515, 6,16:02:36.445,ERR] GossipServiceBase Time difference between us and [10.0.2.13:2113/kurrentdb-cluster.kurrent.internal] is too great! UTC now: 2026-01-05 16:02:36.445, peer's time stamp: 2026-01-05 16:01:20.422.
[ 5515,39,16:02:38.446,ERR] GossipServiceBase Time difference between us and [10.0.2.12:2113/kurrentdb-cluster.kurrent.internal] is too great! UTC now: 2026-01-05 16:02:38.446, peer's time stamp: 2026-01-05 16:01:20.422.
[ 5515,27,16:02:40.447,ERR] GossipServiceBase Time difference between us and [10.0.2.13:2113/kurrentdb-cluster.kurrent.internal] is too great! UTC now: 2026-01-05 16:02:40.447, peer's time stamp: 2026-01-05 16:01:20.422.
[ 5515, 6,16:02:42.447,ERR] GossipServiceBase Time difference between us and [10.0.2.12:2113/kurrentdb-cluster.kurrent.internal] is too great! UTC now: 2026-01-05 16:02:42.447, peer's time stamp: 2026-01-05 16:01:20.422.
[ 5515,41,16:02:48.450,ERR] GossipServiceBase Time difference between us and [10.0.2.12:2113/kurrentdb-cluster.kurrent.internal] is too great! UTC now: 2026-01-05 16:02:48.450, peer's time stamp: 2026-01-05 16:01:20.422.
[ 5515,41,16:02:50.451,ERR] GossipServiceBase Time difference between us and [10.0.2.12:2113/kurrentdb-cluster.kurrent.internal] is too great! UTC now: 2026-01-05 16:02:50.451, peer's time stamp: 2026-01-05 16:01:20.422.
[ 5515,39,16:02:52.451,ERR] GossipServiceBase Time difference between us and [10.0.2.11:2113/kurrentdb-cluster.kurrent.internal] is too great! UTC now: 2026-01-05 16:02:52.451, peer's time stamp: 2026-01-05 16:01:20.422.
[ 5515,27,16:03:04.456,ERR] GossipServiceBase Time difference between us and [10.0.2.11:2113/kurrentdb-cluster.kurrent.internal] is too great! UTC now: 2026-01-05 16:03:04.456, peer's time stamp: 2026-01-05 16:01:20.422.
[ 5515,27,16:03:14.460,ERR] GossipServiceBase Time difference between us and [10.0.2.11:2113/kurrentdb-cluster.kurrent.internal] is too great! UTC now: 2026-01-05 16:03:14.460, peer's time stamp: 2026-01-05 16:01:20.422.
[ 5515, 6,16:03:16.462,ERR] GossipServiceBase Time difference between us and [10.0.2.13:2113/kurrentdb-cluster.kurrent.internal] is too great! UTC now: 2026-01-05 16:03:16.462, peer's time stamp: 2026-01-05 16:01:20.422.
[ 5515,41,16:03:20.464,ERR] GossipServiceBase Time difference between us and [10.0.2.13:2113/kurrentdb-cluster.kurrent.internal] is too great! UTC now: 2026-01-05 16:03:20.464, peer's time stamp: 2026-01-05 16:01:20.422.
[ 5515,41,16:03:22.464,ERR] GossipServiceBase Time difference between us and [10.0.2.12:2113/kurrentdb-cluster.kurrent.internal] is too great! UTC now: 2026-01-05 16:03:22.464, peer's time stamp: 2026-01-05 16:01:20.422.
[ 5515, 9,16:03:30.466,ERR] GossipServiceBase Time difference between us and [10.0.2.12:2113/kurrentdb-cluster.kurrent.internal] is too great! UTC now: 2026-01-05 16:03:30.466, peer's time stamp: 2026-01-05 16:01:20.422.
[ 5515,41,16:03:32.467,ERR] GossipServiceBase Time difference between us and [10.0.2.12:2113/kurrentdb-cluster.kurrent.internal] is too great! UTC now: 2026-01-05 16:03:32.467, peer's time stamp: 2026-01-05 16:01:20.422.
[ 5515,41,16:03:34.467,ERR] GossipServiceBase Time difference between us and [10.0.2.13:2113/kurrentdb-cluster.kurrent.internal] is too great! UTC now: 2026-01-05 16:03:34.467, peer's time stamp: 2026-01-05 16:01:20.422.
[ 5515, 9,16:03:36.468,ERR] GossipServiceBase Time difference between us and [10.0.2.13:2113/kurrentdb-cluster.kurrent.internal] is too great! UTC now: 2026-01-05 16:03:36.468, peer's time stamp: 2026-01-05 16:01:20.422.
[ 5515, 9,16:03:38.468,ERR] GossipServiceBase Time difference between us and [10.0.2.13:2113/kurrentdb-cluster.kurrent.internal] is too great! UTC now: 2026-01-05 16:03:38.468, peer's time stamp: 2026-01-05 16:01:20.422.
[ 5515, 9,16:03:40.469,ERR] GossipServiceBase Time difference between us and [10.0.2.11:2113/kurrentdb-cluster.kurrent.internal] is too great! UTC now: 2026-01-05 16:03:40.469, peer's time stamp: 2026-01-05 16:01:20.422.
[ 5515,38,16:03:44.472,ERR] GossipServiceBase Time difference between us and [10.0.2.12:2113/kurrentdb-cluster.kurrent.internal] is too great! UTC now: 2026-01-05 16:03:44.471, peer's time stamp: 2026-01-05 16:01:20.422.
KurrentDB details
- KurrentDB server version:
Version: KurrentDB version 25.1.0.2664
- Operating system:
Ubuntu
- KurrentDB client library and version (if applicable):
Additional context
Add any other context about the problem here.
I'm aware of #4036 but the issue still exists (I need DNS resolution)