Cluster unstable when running tcp example

[AntonWerenberg]

I've been experimenting with the raft node example code for some time, and one issue keeps being present: Instability in the cluster.

The nodes will often run fine in the beginning, but sooner or later they will start to go out of sync and fail. One node will show the tcp error warning:
"warn: DotNext.Net.Cluster.Consensus.Raft.Tcp.TcpServer[74022]
Request has timed out"
and others will start running elections, but will not reach consensus.

the behaviour can be seen in the attached image.

I'm running with default election timeout settings from the example.

I'm running with default election timeout settings from the example.

I'm not a great programmer and I'm really having trouble seeing which direction to go to get to the bottom of this?

My main concern is currently if this could be due to my own setup, something not configured correctly, or something like that.

I measured Broadcasttime using Metrics collector. It is showing broadcast times of around 3 ms.

[sakno]

Hi @AntonWerenberg ! Which version are you using? 3.x or 4.x?

[AntonWerenberg]

Hi @sakno! I'm running 4.0.0 beta 9. I have however seen similar problems when testing the 3.x fork at sakno/dotNext.
I will try to extract the logs today and see if they can shed light on the issue.

[AntonWerenberg]

That makes sense!

I think the latest update with a shorter ConnectTimeout has improved the stability of the system. However, on the real network, It still goes into the perpetual "Request has timed Out" error state (sometimes after an hour). Could it be because the connection is not properly reset after Timeout error on line 138 in TcpServer.cs?

[sakno]

I don't think so, exchage.Reset just resets the internal state of exchange controller. The connection is destroyed on timeout, so this object will not be reused across multiple connections.

[sakno]

What is your RequestTimeout? Maybe it is too large? You can try to set it smaller than lowerElectionTimeout.

[sakno]

There is another issue #82 that could affect cluster reliability. Internally, Raft implementation relies on synchronization for asynchronous code provided by DotNext.Threading. In some circumstances, it could lead to deadlocks. It explains why the node cannot respond in timely manner. The issue has been fixed in develop branch.

[AntonWerenberg]

My RequestTimeout was set to the default UpperElectionLimit / 2. I have changed it to UpperElectionLimit / 3 now and pulled the latest develop commit. So far the system appears stable.

I will cross my fingers and keep testing. ;)