[Testing guide] Redundant chain nodes

[rjan90]

We are currently testing the Raft Consenus / Redundant Lotus nodes internally, you can see the PR here: #9294. We plan to test this feature over the next couple of weeks/month, but thought it would be a good idea to release some setup steps for testing this in a local developer network if someone feels adventurous. 🔬🧪

N.B: this tutorial is itself a WIP/was written in one go, so it might jump som prerequisite steps, some steps might be redundant, and all steps might not be necessary in your case.

Setup a lotus network consisting of a single lotus node + lotus-miner

1. Start by following the Lotus node setup section for a local network in this guide.
2. When you come to step 3 in the tutorial, git checkout sbansal/nonce-coordination-and-consensus-for-chain-nodes instead of the master-branch, or a stable release.
3. Follow the rest of the steps in the Lotus node setup section.

Now that you’ve got everything setup, you can start the first lotus daemon and the lotus-miner.

1. Follow the steps in the Start the nodes section.
2. Follow all the steps until you have a lotus node and lotus-miner running on a local network.
3. We will need to change some configuration in the config.toml file, so we need to stop the local network. Before shutting things down, check that the lotus-miner is not running behind the chain with the lotus-miner info command.

lotus-miner info
Enabled subsystems (from miner API): [Mining Sealing SectorStorage Markets]
Enabled subsystems (from markets API): [Mining Sealing SectorStorage Markets]
Chain: [sync ok] [basefee 100 aFIL]
⚠ 1 Active alerts (check lotus-miner log alerts)

Stop the lotus-miner first, before you stop the lotus daemon node.
4. In the .lotus-local-net folder, uncomment and change the [API] ListenAddress and the [Libp2p] ListenAddress of the node:

[API]
ListenAddress = "/ip4/127.0.0.1/tcp/4567/http"
......
[Libp2p]
ListenAddresses = ["/ip4/0.0.0.0/tcp/2222", "/ip6/::/tcp/2222"]

5. Start the daemon again, as well as the lotus-miner and confirm the LibP2P multiaddress is similar to the one in the config.toml file when you run lotus net listen.

Setup the second Lotus node

1. In a new terminal, cd into the lotus-local-net folder.
2. Set a different Lotus path for the second lotus node with LOTUS_PATH=/Users/phi/.lotus-local-net2
3. Initialize the second lotus node with: ./lotus daemon --genesis=devgen.car. The node should initilize in the new path and you should see this error log:
   2022-10-07T15:21:31.980+0200 WARN peermgr peermgr/peermgr.go:205 no peers connected, and no bootstrappers configured
4. Stop the daemon and configure the [API] ListenAddress and the [Libp2p] ListenAddress in the config.toml file in the .lotus-local-net2 folder of this second node.

[API]
ListenAddress = "/ip4/127.0.0.1/tcp/5678/http"
......
[Libp2p]
ListenAddresses = ["/ip4/0.0.0.0/tcp/3333", "/ip6/::/tcp/3333"]

5. Restart this second node again with ./lotus daemon --genesis=devgen.car
6. Connect the second node to the first node with ./lotus net connect /ip4/127.0.0.1/tcp/2222/p2p/12D3xxxxxxxxxxsbD
7. Import the keys from node1 to node2.
8. Your second lotus node now should sync up with your local network. Wait for it to finish sync with lotus sync wait.

Setup the third Lotus node

1. Follow the same steps as setting up the second Lotus node, but set the path to LOTUS_PATH=/Users/phi/.lotus-local-net3 and configure the [API] ListenAddress and the [Libp2p] ListenAddress in the config.toml to this:

[API]
ListenAddress = "/ip4/127.0.0.1/tcp/6789/http"
......
[Libp2p]
ListenAddresses = ["/ip4/0.0.0.0/tcp/4444", "/ip6/::/tcp/4444"]

Configuring Raft Consensus / Redundant Chain nodes

As you may have noticed, there is now a new section in the config.toml file for the lotus node, called [Raft].

1. While your local network is running (consisting of three lotus nodes, and a single lotus-miner), configure the config.toml for the lotus daemon nodes like this:

You can get the multiaddress for your nodes by checking the outout of lotus net listen for all three daemons.

[Raft]
ClusterModeEnabled = true
.....
InitPeersetMultiAddr =   InitPeersetMultiAddr = ["/ip4/127.0.0.1/tcp/2222/p2p/12D3KooWHVawzGL5SG58rS1Ti8m3G8fA9NwEWkfnz1AcRLWq1deF","/ip4/127.0.0.1/tcp/3333/p2p/12D3KooWB2ikW3gvaQiwfdnD8HrFAqBd2Y54gdykLTFybUQsYrBG","/ip4/127.0.0.1/tcp/4444/p2p/12D3KooWHxNgWfmiJGf6sFXbjQhnBHudsXGz9WAuZB1H4LLwxx7V"]

2. On the lotus-miner unset any LOTUS_PATH enviroment variables, and add the full node api info for the three deamons: export FULLNODE_API_INFO=<node0_info>,<node1_info>,<node2_info>. You can get API-keys for each node by lotus auth create-token --perm admin. The format for each nodes info is like this: export FULLNODE_API_INFO=<api_token>:/ip4/<lotus_daemon_ip>/tcp/<lotus_daemon_port>/http

FULLNODE_API_INFO=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJBbGxvdyI6WyJyZWFkIiwid3JpdGUiLCJzaWduIiwiYWRtaW4iXX0.T_meWfWV-F_pX19EPZ1p0uLaRmX3kpE_KFE7nXx9ENs:/ip4/127.0.0.1/tcp/4567/http,eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJBbGxvdyI6WyJyZWFkIiwid3JpdGUiLCJzaWduIiwiYWRtaW4iXX0.lIygxCSIqdSeVvN73aVIme9mRdjOunFsn5eb8K8Q5R8:/ip4/127.0.0.1/tcp/5678/http,eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJBbGxvdyI6WyJyZWFkIiwid3JpdGUiLCJzaWduIiwiYWRtaW4iXX0.arVqeW93VujWC5JlIoumfbRFiHk8BtROp9rsdZPEaVk:/ip4/127.0.0.1/tcp/6789/http

2. Now we can restart the whole network. Turn the lotus-miner off first, then stop the daemons.
3. Start the first lotus node and the lotus-miner, before you start the other lotus nodes. You should now see a new log when starting the second and third node, confirming that the Raft Consensus (and leader) has been set properly:

2022-10-07T14:40:13.890+0200	INFO	raft	raft/raft.go:280	Current Raft Leader: 12D3KooWHcpyxqBMsRiUkCNnirNgYK4ueHQt3XU7tVofNUjHcsbD

You can also check who is the leader with: ./lotus-shed rpc RaftLeader

Thats it for now, I will come back with more later, until then; Happy Testing!! 👨‍🔬👩‍🔬

[dd45e640b42e6da7da96faee3996ef7c]

what exactly do people need to clone/merge to make this work?

[rjan90]

git clone https://github.com/filecoin-project/lotus lotus-local-net
cd lotus-local-net
git checkout sbansal/nonce-coordination-and-consensus-for-chain-nodes

[dd45e640b42e6da7da96faee3996ef7c]

it's not live network'able (yet)? calib? wallaby? mainnet?

[rjan90]

You can of course checkout the branch and make this for calibnet/wallaby, or mainnet (not adviced) if you want.

For now I have only been testing this on a local network for ease of setup, send a lot messages from the miner / shutdown/restart nodes / clean everything up, etc. We expect to encounter issues during testing, and will progressively test it in more production like enviroments once issues gets surfaced and fixed.

Just opened the steps for using this feature so one can familiarize with the new configs. Or if someone wants to test this early and find issues/surface potential enhancements before its pushed to a release candidate, we would highly appreciate that.