Autosharding tests: Outage & Rebalancing

[donhardman]

• Validate rebalancing during node outages while maintaining the replication factor.
• Ignore rebalancing when any part of the shards is missing.
• Ensure we are not performing a rebalance during a short-term node outage.
• Check shard rebalancing when a new node is added.
• Validate rebalancing in two cases: 3 nodes with 1 out, and 5 nodes with 2 out (no missing shards, rf=3).

[sanikolaev]

Ignore rebalancing when any part of the shards is missing.

This one is not clear.

[PavelShilin89]

Ignore rebalancing when any part of the shards is missing.

This one is not clear.

Do not rebalance when we lost some shards, for example 5 clusers, 2 node dead, rf=2, we lost data, no need rebalancing

[PavelShilin89]

@donhardman Testing is done in this PR - manticoresoftware/manticoresearch#3304
Please give feedback or approval to merge.

[PavelShilin89]

Bug: Shard rebalancing does not occur when a new node is added to the cluster

Description:

• A new node (6312) joined the cluster but did not receive any shards (shards: empty). This could indicate a potential issue with the rebalancing mechanism, where the new node is expected to participate in shard distribution to maintain a replication factor of 2, but fails to do so. Alternatively, this behavior might be intentional, aiming to avoid triggering a large-scale rebalance and excessive data movement all at once.

Expected behavior:

• At rf=2, each shard should have two copies distributed across nodes. A new node should receive at least one shard (e.g., shards: 0 or shards: 1) to balance the load.

You can see the result by running clt-test in my branch:

• Branch

test/autosharding-outage-rebalancing

• Running a test

/path/to/clt/clt test -d -t ./test/clt-tests/sharding/test-autosharding-outage-rebalancing.rec ghcr.io/manticoresoftware/manticoresearch:test-kit-fix-sharding-rebalacing-on-new-node

Current result:

––– input –––
sleep 60; export INSTANCE=6
––– output –––
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
Buddy started!
––– input –––
mysql -h0 -P6306 -e "join cluster c at '127.0.0.1:1312';"
––– output –––
––– input –––
sleep 300; timeout 120 bash -c 'until mysql -h0 -P1306 -e "select * from system.sharding_table\G" | grep -q "node: 127.0.0.1:6312"; do sleep 1; done'; mysql -h0 -P1306 -e "select * from system.sharding_table\G"
––– output –––
*************************** 1. row ***************************
     id: 75274028608126997
cluster: c
   node: 127.0.0.1:1312
  table: testrt
 shards: 0
*************************** 2. row ***************************
     id: 75274028608126998
cluster: c
   node: 127.0.0.1:2312
  table: testrt
 shards: 1
*************************** 3. row ***************************
     id: 75274028608126999
cluster: c
   node: 127.0.0.1:5312
  table: testrt
 shards: 0
*************************** 4. row ***************************
     id: 75274028608127000
cluster: c
   node: 127.0.0.1:3312
  table: testrt
 shards: 0,2
*************************** 5. row ***************************
     id: 75274028608127001
cluster: c
   node: 127.0.0.1:6312
  table: testrt
 shards:
*************************** 6. row ***************************
     id: 75274028608126996
cluster: c
   node: 127.0.0.1:4312
  table: testrt
 shards: 1,2
––– input –––
mysql -h0 -P6306 -e "show status like 'cluster_c_status'\G"
––– output –––
*************************** 1. row ***************************
Counter: cluster_c_status
  Value: primary
––– input –––
mysql -h0 -P6306 -e "show tables from system\G" | grep Table
––– output –––
Table: system.sharding_queue
Table: system.sharding_state
Table: system.sharding_table

@donhardman Let's discuss this problem.