Failed to fetch the latest version of the routing table #243
Description
I'm running Olric (Embedded Client) in an application running in a Kubernetes StatefulSet. When I scale down the number of Pods, I see the node removed from the cluster, but continue to see errors like the following:
logger: cluster_iterator.go:276: [ERROR] Failed to fetch the latest version of the routing table: tcp 10.244.0.18:3320: i/o timeout
logger: cluster_client.go:763: [ERROR] Failed to fetch the latest version of the routing table: error while loading the routing table: tcp 10.244.0.18:3320: i/o timeout
Also the data within the DMaps is no longer there (or accessible)
Using Olric v0.5.4 and Olric Cloud Plugin v0.3.0-beta.4
More detail:
Log showing Olric node leaving the cluster when Pod is terminated (from point of view of the surviving Pod).
{"time":"2024-03-19T16:37:31.247446875Z","level":"INFO","msg":"Node left: 10.244.0.18:3320","library":"olric","timestamp":"2024-03-19T16:37:31"}
{"time":"2024-03-19T16:37:31.247598088Z","level":"INFO","msg":"Member: 10.244.0.18:3320 has been deleted from the primary owners list of PartID: 0","library":"olric","timestamp":"2024-03-19T16:37:31"}
{"time":"2024-03-19T16:37:31.247641869Z","level":"INFO","msg":"Member: 10.244.0.18:3320 has been deleted from the primary owners list of PartID: 1","library":"olric","timestamp":"2024-03-19T16:37:31"}
{"time":"2024-03-19T16:37:31.24768042Z","level":"INFO","msg":"Member: 10.244.0.18:3320 has been deleted from the primary owners list of PartID: 2","library":"olric","timestamp":"2024-03-19T16:37:31"}
...
{"time":"2024-03-19T16:37:31.262107697Z","level":"INFO","msg":"Member: 10.244.0.18:3320 has been deleted from the primary owners list of PartID: 266","library":"olric","timestamp":"2024-03-19T16:37:31"}
{"time":"2024-03-19T16:37:31.26236744Z","level":"INFO","msg":"Member: 10.244.0.18:3320 has been deleted from the primary owners list of PartID: 270","library":"olric","timestamp":"2024-03-19T16:37:31"}
{"time":"2024-03-19T16:37:31.262941421Z","level":"INFO","msg":"Routing table has been pushed by 10.244.0.17:3320","library":"olric","timestamp":"2024-03-19T16:37:31"}
logger: cluster_iterator.go:276: [ERROR] Failed to fetch the latest version of the routing table: context canceled
logger: cluster_iterator.go:276: [ERROR] Failed to fetch the latest version of the routing table: context canceled
logger: cluster_iterator.go:276: [ERROR] Failed to fetch the latest version of the routing table: context canceled
logger: cluster_iterator.go:276: [ERROR] Failed to fetch the latest version of the routing table: context canceled
logger: cluster_iterator.go:276: [ERROR] Failed to fetch the latest version of the routing table: context canceled
logger: cluster_iterator.go:276: [ERROR] Failed to fetch the latest version of the routing table: tcp 10.244.0.18:3320: i/o timeout
logger: cluster_iterator.go:276: [ERROR] Failed to fetch the latest version of the routing table: tcp 10.244.0.18:3320: i/o timeout
logger: cluster_iterator.go:276: [ERROR] Failed to fetch the latest version of the routing table: tcp 10.244.0.18:3320: i/o timeout
logger: cluster_iterator.go:276: [ERROR] Failed to fetch the latest version of the routing table: tcp 10.244.0.18:3320: i/o timeout
logger: cluster_iterator.go:276: [ERROR] Failed to fetch the latest version of the routing table: tcp 10.244.0.18:3320: i/o timeout
logger: cluster_iterator.go:276: [ERROR] Failed to fetch the latest version of the routing table: tcp 10.244.0.18:3320: i/o timeout
logger: cluster_iterator.go:276: [ERROR] Failed to fetch the latest version of the routing table: tcp 10.244.0.18:3320: i/o timeout
{"time":"2024-03-19T16:37:54.497092631Z","level":"INFO","msg":"Routing table has been pushed by 10.244.0.17:3320","library":"olric","timestamp":"2024-03-19T16:37:54"}
logger: cluster_client.go:763: [ERROR] Failed to fetch the latest version of the routing table: error while loading the routing table: tcp 10.244.0.18:3320: i/o timeout
logger: cluster_iterator.go:276: [ERROR] Failed to fetch the latest version of the routing table: tcp 10.244.0.18:3320: i/o timeout
logger: cluster_iterator.go:276: [ERROR] Failed to fetch the latest version of the routing table: tcp 10.244.0.18:3320: i/o timeout
logger: cluster_iterator.go:276: [ERROR] Failed to fetch the latest version of the routing table: tcp 10.244.0.18:3320: i/o timeout
logger: cluster_iterator.go:276: [ERROR] Failed to fetch the latest version of the routing table: tcp 10.244.0.18:3320: i/o timeout
logger: cluster_iterator.go:276: [ERROR] Failed to fetch the latest version of the routing table: tcp 10.244.0.18:3320: i/o timeout
logger: cluster_iterator.go:276: [ERROR] Failed to fetch the latest version of the routing table: tcp 10.244.0.18:3320: i/o timeout
logger: cluster_iterator.go:276: [ERROR] Failed to fetch the latest version of the routing table: tcp 10.244.0.18:3320: i/o timeout