Older Nodes in cluster having higher CPU utilization

[subin94]

Description

The cluster setup initially consisted 2 nodes which was there for more than 6 months having more than 1500 databases and data size of more than 10 GiB. We now added a new node to the cluster. Now when monitoring the CPU utilization of the older nodes, we could see the beam.smp process eating up a lot of CPU (60-70%). But in the new node the CPU utilized by the beam.smp is less (<20%). Is there a way to reduce the CPU utilization in the older nodes?

Note

Traffic to this cluster is equally distributed to all 3 nodes via load balance

Environment

• CouchDB version used: 3.1.0
• Operating system and version: Ubuntu 16.04.5 LTS

Steps to Reproduce

Adding a new node to an existing cluster having huge data size.

Expected Behaviour

CPU utilization by beam.smp in older nodes should be less and almost equal to that of the new node.

Ready to provide any additional information if needed. Thanks in advance.

[janl]

Did you move any existing database shards from the existing node(s) to the new server after adding it to the cluster?

[subin94]

The new server was basically created from the Amazon Machine Image (AMI ) of an existing server. This means, couchdb data directory of the new server is an excat copy of an existing server. So Yes, the new server has database shards from an existing node.

[janl]

that is not how moving shards works, like at all ;) Did you do any of the steps outlined here? https://docs.couchdb.org/en/stable/cluster/sharding.html#moving-a-shard

If not, see where that leads you.

Just cloning a node will not give you a three node cluster, so you will have to have done a few more steps. Which were those?

[subin94]

Thanks a lot for the link. It is clear that we need to update the target node's cluster metadata for the load to be evenly distributed.
Please also guide us in the below pointers.

1. Instead of moving the shards to target node by removing it from the source node, can we just copy the shards to target node and update the cluster metadata of the target node? will this work ?
2. Is it okay to change the value of n and q now, in all node's config file?
3. Is there any tool or a script to help update the cluster metadata of the node safely?

[janl]

Instead of moving the shards to target node by removing it from the source node, can we just copy the shards to target node and update the cluster metadata of the target node? will this work ?

Ideally, let CouchDB handle this.

Is it okay to change the value of n and q now, in all node's config file?

Yes, but note that this only affects newly created databases. Existing databases retain the q settings they were created with. How to increase n, we showed above. To migrate a database’s q setting, in 3.x you can increase q, for reducing q in 3.x, or all changes to q in 2.x, we built this tool: https://github.com/neighbourhoodie/couch-continuum

Is there any tool or a script to help update the cluster metadata of the node safely?

Not really, you have to be very careful :)

Should you prefer professional assistance, do not hesitate to reach out: https://neighbourhood.ie/couchdb-support/