Skip to content

in the latest versions, cluster creation "Waiting for the cluster to join" is infinite #1042

Closed
@KerimNar

Description

@KerimNar

in older versions
image: "ghcr.io/microsoft/garnet-alpine:sha-56394d8"

kubectl get pods
NAME                              READY   STATUS    RESTARTS   AGE
my-garnet-cluster-statefulset-0   1/1     Running   0          7m26s
my-garnet-cluster-statefulset-1   1/1     Running   0          7m18s
my-garnet-cluster-statefulset-2   1/1     Running   0          7m11s
redis-cli --cluster create my-garnet-cluster-statefulset-0.my-garnet-cluster-internal:6379 my-garnet-cluster-statefulset-1.my-garnet-cluster-internal:6379 my-garnet-cluster-statefulset-2.my-garnet-cluster-internal:6379 --cluster-replicas 0
>>> Performing hash slots allocation on 3 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
M: 56cc34429436823fa15ef5085111fc9234c2cfc0 my-garnet-cluster-statefulset-0.my-garnet-cluster-internal:6379
   slots:[0-5460] (5461 slots) master
M: e1c992c82e79c77f7b29c31667a7d5540c11aadb my-garnet-cluster-statefulset-1.my-garnet-cluster-internal:6379
   slots:[5461-10922] (5462 slots) master
M: fb5f9f68650acb903ede83e518b31cebd1143020 my-garnet-cluster-statefulset-2.my-garnet-cluster-internal:6379
   slots:[10923-16383] (5461 slots) master
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
.
>>> Performing Cluster Check (using node my-garnet-cluster-statefulset-0.my-garnet-cluster-internal:6379)
M: 56cc34429436823fa15ef5085111fc9234c2cfc0 my-garnet-cluster-statefulset-0.my-garnet-cluster-internal:6379
   slots:[0-5460] (5461 slots) master
M: e1c992c82e79c77f7b29c31667a7d5540c11aadb 10.243.121.187:6379
   slots:[5461-10922] (5462 slots) master
M: fb5f9f68650acb903ede83e518b31cebd1143020 10.243.120.40:6379
   slots:[10923-16383] (5461 slots) master
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
root@s50:/home/narov# redis-cli -h 10.0.1.220 -c CLUSTER NODES
fb5f9f68650acb903ede83e518b31cebd1143020 10.243.120.40:6379@16379,my-garnet-cluster-statefulset-2.my-garnet-cluster-internal.default.svc.cluster.local myself,master - 0 0 3 connected 10923-16383
56cc34429436823fa15ef5085111fc9234c2cfc0 10.243.119.28:6379@16379,my-garnet-cluster-statefulset-0.my-garnet-cluster-internal.default.svc.cluster.local master - 638757936731540401 638757936731538928 1 connected 0-5460
e1c992c82e79c77f7b29c31667a7d5540c11aadb 10.243.121.187:6379@16379,my-garnet-cluster-statefulset-1.my-garnet-cluster-internal.default.svc.cluster.local master - 638757936731541402 638757936731541026 2 connected 5461-10922

but latest versions,
image: "ghcr.io/microsoft/garnet-alpine:latest"

redis-cli --cluster create my-garnet-cluster-statefulset-0.my-garnet-cluster-internal:6379 my-garnet-cluster-statefulset-1.my-garnet-cluster-internal:6379 my-garnet-cluster-statefulset-2.my-garnet-cluster-internal:6379 --cluster-replicas 0
>>> Performing hash slots allocation on 3 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
M: 004641f75fd6f8e171a0a3ea1809dd070c926078 my-garnet-cluster-statefulset-0.my-garnet-cluster-internal:6379
   slots:[0-5460] (5461 slots) master
M: 39437f540f308aa3643123b46eb6f8bc18717abb my-garnet-cluster-statefulset-1.my-garnet-cluster-internal:6379
   slots:[5461-10922] (5462 slots) master
M: b238d43e9f60cba7c807d15ded367a8e2aa134b5 my-garnet-cluster-statefulset-2.my-garnet-cluster-internal:6379
   slots:[10923-16383] (5461 slots) master
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
..............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
root@s50:/home/narov# redis-cli -h 10.0.1.220 -c
10.0.1.220:6379> CLUSTER NODES
004641f75fd6f8e171a0a3ea1809dd070c926078 10.243.119.45:6379@16379,my-garnet-cluster-statefulset-0.my-garnet-cluster-internal.default.svc.cluster.local myself,master - 0 0 1 connected 0-5460
10.0.1.220:6379> 
root@s50:/home/narov# redis-cli -h 10.0.1.220 -c
10.0.1.220:6379> CLUSTER NODES
b238d43e9f60cba7c807d15ded367a8e2aa134b5 10.243.120.48:6379@16379,my-garnet-cluster-statefulset-2.my-garnet-cluster-internal.default.svc.cluster.local myself,master - 0 0 3 connected 10923-16383
10.0.1.220:6379> 
root@s50:/home/narov# redis-cli -h 10.0.1.220 -c
10.0.1.220:6379> CLUSTER NODES
39437f540f308aa3643123b46eb6f8bc18717abb 10.243.121.178:6379@16379,my-garnet-cluster-statefulset-1.my-garnet-cluster-internal.default.svc.cluster.local myself,master - 0 0 2 connected 5461-10922

garnet log:

2025-02-22T09:04:08.065010213+05:00     _________

2025-02-22T09:04:08.065043969+05:00    /_||___||_\      Garnet 1.0.57 64 bit; cluster mode

2025-02-22T09:04:08.065074187+05:00    '. \   / .'      Listening on: 0.0.0.0:6379

2025-02-22T09:04:08.065076723+05:00      '.\ /.'        https://aka.ms/GetGarnet

2025-02-22T09:04:08.065079016+05:00        '.'

2025-02-22T09:04:08.065086310+05:00 

2025-02-22T09:04:08.264288622+05:00 * Ready to accept connections

2025-02-22T09:04:14.679865287+05:00 04::04::14 fail: ClusterManager[0] Meet terminated with error System.FormatException: An invalid IP address was specified.  ---> System.Net.Sockets.SocketException (22): Invalid argument    --- End of inner exception stack trace ---    at System.Net.IPAddressParser.Parse(ReadOnlySpan`1 ipSpan, Boolean tryParse)    at Garnet.cluster.ClusterManager.TryMeetAsync(String address, Int32 port, Boolean acquireLock) in /src/libs/cluster/Server/Gossip.cs:line 164

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions