Node fails to rejoin the cluster after network reconnect when using etcd for service discovery

[pipiaha]

Description:

I encountered an issue when using etcd as the service discovery mechanism for a ProtoActor cluster. When a node loses connection to etcd (for example, due to network fluctuation or during breakpoint debugging), and the network is restored, A gocoroutine to call startKeepAlive to re-register the lease. While the lease is successfully renewed, the current node does not seem to be re-added to the members list in etcd.Provider. This causes the ActorSystem to remain active but the node is no longer part of the cluster.

Steps to Reproduce:

1. Start a node and use etcd for cluster service discovery. by default, keepAliveTTL=3s and retryInterval=1s.

provider, _ = etcd.NewWithConfig(b.Config.ClusterBaseKey, clientv3.Config{
			Endpoints: []string{"example.etcd.addr:2379"},
			Username:  "foo",
			Password:  "bar",
			//DialKeepAliveTime:    10 * time.Second,
			//DialKeepAliveTimeout: 10 * time.Second,
		})

3. Disconnect the node from etcd due to network fluctuations or debugging.
4. After network recovery, use the scheduled coroutine to call startKeepAlive and renew the lease.
5. Notice that the members list in etcd.Provider does not include the current node, and as a result, the node's ActorSystem does not rejoin the cluster.

Expected Behavior:

After network recovery, the node should successfully re-register itself via startKeepAlive and be added back to the etcd.Provider members list. The node's ActorSystem should then rejoin the cluster and function normally.

Current Behavior:

The node fails to rejoin the cluster. Even though the lease is renewed, the members list in etcd.Provider is not updated to include the node, which causes the node's ActorSystem to no longer participate in the cluster.

Environment:

• ProtoActor-Go version: v0.0.0-20240822202345-3c0e61ca19c9
• etcd version: v3
• Go version: go 1.22.7

Additional Information:

• Will it work if keepAliveTTL configuration can be customized?

I would appreciate assistance on how to ensure the node can properly rejoin the cluster after network recovery.

[pipiaha]

I reproduced this issue again using the examples/cluster-basic test case. First, I started nodes A and B, then forced node B to go offline in etcd by using a long breakpoint. After releasing the breakpoint and waiting for B to rejoin, the handleWatchResponse's Put event actually ignores the join logic for node B as "self," causing node B's Cluster MemberList to completely lose its own node.

If at this point node A restarts and rejoins the cluster, node B will continuously report Gossip errors to itself: Got gossip request from unknown member, even though this member is B itself.

I wondered whether ignoring "self" in the Delete event of handleWatchResponse—just as it does in the Put event—might resolve this issue. So, I tried PR #1141, and it appears to have fixed the problem. Looking forward to your reply (though this project hasn't seen much activity lately). @rogeralsing

[azambranogalbis]

Hi! I am facing a similar issue with the ZooKeeper provider. I kindly ask for support of the protoactor-go community of users.

Basically, due to network problems or high load on a particular protoactor node, we observe that the node leaves and re-registers itself in ZooKeeper. Nevertheless, the rest of the nodes of the protoactor cluster ignore it, even if the reconnection was successful.

In my case, after investigating everything boils down to the following line:

protoactor-go/cluster/member_list.go

Line 198 in 3c0e61c

blocked := ml.cluster.GetBlockedMembers().ToSlice()

Apparently, this behaviour is a design decission, which applies regardless the selected cluster provider. The node is marked as "blocked", and simply won't ever rejoin the protoactor cluster. I would like to understand the motivation of this decission, since it seems that removing it would make protoactor cluster operation more reliable.

Many thanks in advance for your comments!