Enhance Redundant Virtual Routers to 3 Routers, with RAFT Algorithm (or similar)

[btzq]

ISSUE TYPE

• Feature Idea

COMPONENT NAME

Redundant Virtual Router

CLOUDSTACK VERSION

N/A

CONFIGURATION

N/A

OS / ENVIRONMENT

N/A

SUMMARY

Idea To Be Able to Deploy A Total Of 3 Redundant Virtual Routers instead of 2.

In our own experiences, we've had bad experiences with Redundant VPC Networks because it creates total 2 Virtual Routers.

Although this was originally designed to offer redundancies, the real life experience of users of Cloudstack is that the setup is more likely to have cause operational issues instead. (myself and based on what ive read from other users experiences).

Some examples of theese issues are Master/Master Reundant Routers where both Routers are active at the same time. This usually ocurs due to a network issue where both routers are unable to communicate with each other (but are still alive). Hence, the passive becomes active, and the VPC becomes confused.

In our current set up, we had to change from a Redundant VPC (Dual Routers) to a Normal VPC (Single Router) because the redundant routers gave us more issues than benefits. Because of this, we had to accept a 5-7 minute downtime in the event a router goes down, so that it could failover to another node. This makes us less competitive than other options cloud providers, but provided the most predictability.

This then got me thinking.

What i've noticed from other technologies with a redundancy setup is that they use a minimum total of 3 members to ensure reliable redundancies. Examples are:

• Redis Sentinel (Uses Quorum = 2, resulting in total 3 nodes)
• Cockroach DB (Requires minimum 3 nodes)
• Apache Kafka (Requires minimum 3 Nodes)

I believe this has got to do with the RAFT Algorithm where there is 1 Leader, and 2 Followers. In the event that any 1 node fails, the other 2 remaining members need to agree that the service is down, before agreeing to promote any 1 of the 2 remaining nodes to be the Active Node.

As a Feature Idea, what if Cloudstack were to offer Redundant Virtual Routers with similar concepts? Where there would be 3 routers except of two, and an implementation of a voting/promotion algorithm similar to RAFT.

I can imagine that the greatest benefits to businesses using Clodstack would be being able to achieve much greater high availabilities than current implementations.

This is just a random night time idea, im not sure of the technical implementations or repurcussions, but happy to discuss this idea and hear what the community thinks.

STEPS TO REPRODUCE

N/A

EXPECTED RESULTS

Feature Idea - Idea To Be Able to Deploy A Total Of 3 Redundant Virtual Routers instead of 2.

ACTUAL RESULTS

Redundant Routers only has Total 2 Routers

[weizhouapache]

@btzq thank for your idea.
You missed etcd which is probably the most known implementation of raft algorithm.

To be honest, I do not think it is necessary to invest efforts on it.
The only difference between single VR and redundant VRs is the ip failover. By far keepalived is still the most popular and stable solution for IP failover. It is also used by some cloud providers and projects to implement features like elastic ip or vip.

Keepalived works very well if the network is configured correctly.
If redundant VRs do not work, please check the network configuration (not cloudstack network).

[btzq]

Hey @weizhouapache

i see, but i believe redundant virtual router also relies on oob and ipmi to be configured right? Otherwise, during a node failure, the original master router might still be alive, while the slave thinks something is wrong and promotes itself to master. This results in a master master issue?

But in our case, we are using linstor as the storage and linbits recommendation was to rely only on VM HA, and not configure any OOB/IPMI. They said it just works more reliably that way.

Not sure if this helps, but another idea i can think of is that 3 routers with a quorum can help reduce split brain scenarios since voting and consensus must happen before a router gets promoted. This doesnt happen in linstor cause linstor only allows single primary, but i can think this is beneficial for nfs storage environments.

[weizhouapache]

@btzq
the RVR only relies on the VRRP communication on the guest networks. This is why most master/master issues were caused by switch configurations (mlag/lacp/igmp, etc)

The feature is unrelated to host ha, vm ha and storage, etc.

In the past, there were some issues with vpc as the vpc tiers and public nics can be added/removed dynamically. It has improved in recent releases I think. if the vpc vrs are not started OOB , keepalived should work without issue.
If you have master/master issue with redundant vpc routers, please share the keepalived configurations and cloud logs, we can have a look.

[btzq]

Okay noted @weizhouapache , we havent had any Master/Master issues because we completely removed the redundancy as it gave us more predictability.

But are there any tests we can run to ensure our networks are configured correctly?

Also, what should i do with this feature request? Do i close it?

[weizhouapache]

Okay noted @weizhouapache , we havent had any Master/Master issues because we completely removed the redundancy as it gave us more predictability.

But are there any tests we can run to ensure our networks are configured correctly?

Everyone has different physical routers.
you can create a test isolated network with redundant VRs, and let it running for some days.

Also, what should i do with this feature request? Do i close it?

@btzq
I have added it to milestone "unplanned". let's keep it open

[rohityadavcloud]

The VR HA relies on L2/L3, all mentioned solutions are L4-L7 HA solutions and not appropriate for VRs. From an opensource PoV we only rely on keepalived that implements the VRRP protocol which is enough for most scenarios. You can also consider using shared networks in CloudStack with an upstream router config of your choice (say Pfsense, Cisco etc) or use VNF feature. Keepalived/VRRP doesn’t always guarantee zero RPO but offers automation to heal effectively with seconds in most scenarios but may require additional HA in the underlying network infra (for example a bad cable or faulty nic or switch connected to your hosts will not save your networks with VRRP).