[2025.1.1] Writes in heterogenous rack only go to 1 node in 1 of 3 racks

[cezarmoise]

The scenario is:
3 racks, RF=3, when adding or removing nodes it is done per rack.
Keyspaces and tables were created with default settings of cassandra-stress.
Start with a cluster of 3 i4i.large nodes, Write to 90%. (keyspace0)
Add 3 i4i.xlarge nodes, Remove the i4i.large nodes. Write to 90% again. (keyspace1)
Add 3 i4i.large nodes. Write to 90%. (keyspace2)
At this point the out of space failure occurs.

Argus Replicated once Replicated twice
Monitoring
Scylla was ran with --logger-log-level load_balancer=debug

Node elasticity-test-grow-l-8-db-node-fbf411a5-6 | 10.12.3.114 | i4i.xlarge does not get any data from writes in this keyspace, while the other node in the rack elasticity-test-grow-l-8-db-node-fbf411a5-9 | 10.12.3.71 | i4i.large gets more and ends up out of space. Other racks not affected.

This is the only case where writing began when racks had 2 different nodes, i4i.large and i4i.xlarge.

Previously, adding or removing nodes with existing keyspaces balances normally

Logs about keyspace2 in node 6

2025-04-04T20:24:55.736+00:00 elasticity-test-grow-l-8-db-node-fbf411a5-6     !INFO | scylla[5590]:  [shard 0: gms] schema_tables - Creating keyspace keyspace2
2025-04-04T20:24:56.550+00:00 elasticity-test-grow-l-8-db-node-fbf411a5-6     !INFO | scylla[5590]:  [shard 0:sl:d] migration_manager - Create new ColumnFamily: org.apache.cassandra.config.CFMetaData@0x600011334000[cfId=dfbe1840-1192-11f0-842f-be81d411025a,ksName==keyspace2,cfName=standard1,cfType=Standard,comparator=org.apache.cassandra.db.marshal.CompositeType(org.apache.cassandra.db.marshal.UTF8Type),comment=,tombstoneGcOptions={"mode":"repair","propagation_delay_in_seconds":"3600"},gcGraceSeconds=864000,minCompactionThreshold=4,maxCompactionThreshold=32,columnMetadata=[ColumnDefinition{name=key, type=org.apache.cassandra.db.marshal.BytesType, kind=PARTITION_KEY, componentIndex=0, droppedAt=-9223372036854775808}, ColumnDefinition{name=C0, type=org.apache.cassandra.db.marshal.BytesType, kind=REGULAR, componentIndex=null, droppedAt=-9223372036854775808}, ColumnDefinition{name=C1, type=org.apache.cassandra.db.marshal.BytesType, kind=REGULAR, componentIndex=null, droppedAt=-9223372036854775808}, ColumnDefinition{name=C2, type=org.apache.cassandra.db.marshal.BytesType, kind=REGULAR, componentIndex=null, droppedAt=-9223372036854775808}, ColumnDefinition{name=C3, type=org.apache.cassandra.db.marshal.BytesType, kind=REGULAR, componentIndex=null, droppedAt=-9223372036854775808}, ColumnDefinition{name=C4, type=org.apache.cassandra.db.marshal.BytesType, kind=REGULAR, componentIndex=null, droppedAt=-9223372036854775808}, ColumnDefinition{name=C5, type=org.apache.cassandra.db.marshal.BytesType, kind=REGULAR, componentIndex=null, droppedAt=-9223372036854775808}, ColumnDefinition{name=C6, type=org.apache.cassandra.db.marshal.BytesType, kind=REGULAR, componentIndex=null, droppedAt=-9223372036854775808}, ColumnDefinition{name=C7, type=org.apache.cassandra.db.marshal.BytesType, kind=REGULAR, componentIndex=null, droppedAt=-9223372036854775808}],compactionStrategyClass=class org.apache.cassandra.db.compaction.IncrementalCompactionStrategy,compactionStrategyOptions={enabled=true},compressionParameters={},bloo ...
2025-04-04T20:24:56.550+00:00 elasticity-test-grow-l-8-db-node-fbf411a5-6    !DEBUG | scylla[5590]:  [shard 0:sl:d] load_balancer - Creating tablets for keyspace2.standard1 id=dfbe1840-1192-11f0-842f-be81d411025a
2025-04-04T20:24:56.550+00:00 elasticity-test-grow-l-8-db-node-fbf411a5-6     !INFO | scylla[5590]:  [shard 0:sl:d] replication_strategy - Rounding up tablet count from 6 to 8 for table keyspace2.standard1
2025-04-04T20:24:56.550+00:00 elasticity-test-grow-l-8-db-node-fbf411a5-6     !INFO | scylla[5590]:  [shard 0: gms] schema_tables - Creating keyspace2.standard1 id=dfbe1840-1192-11f0-842f-be81d411025a version=dfbe1841-1192-11f0-842f-be81d411025a
2025-04-04T21:09:28.739+00:00 elasticity-test-grow-l-8-db-node-fbf411a5-6     !INFO | scylla[5590]:  [shard 0:strm] table - Setting split ready sequence number to 1 for table keyspace2.standard1

Logs about keyspace2 in node 9

2025-04-04T20:24:55.839+00:00 elasticity-test-grow-l-8-db-node-fbf411a5-9     !INFO | scylla[5548]:  [shard 0: gms] schema_tables - Creating keyspace keyspace2
2025-04-04T20:24:56.781+00:00 elasticity-test-grow-l-8-db-node-fbf411a5-9     !INFO | scylla[5548]:  [shard 0: gms] schema_tables - Creating keyspace2.standard1 id=dfbe1840-1192-11f0-842f-be81d411025a version=dfbe1841-1192-11f0-842f-be81d411025a

The a lot of logs with compaction - [Compact keyspace2.standard1 ... and sstable - Rebuilding bloom filter

[cezarmoise]

cc: @pehala @tgrabiec

[mykaul]

Are we sure it's not a loader (c-s / driver) issue? What version are you using?

[pehala]

Are we sure it's not a loader (c-s / driver) issue? What version are you using?

Are you using simulared racks or real AZ?

[cezarmoise]

Are we sure it's not a loader (c-s / driver) issue? What version are you using?

Are you using simulared racks or real AZ?

Simulated.

[pehala]

Are we sure it's not a loader (c-s / driver) issue? What version are you using?

Are you using simulared racks or real AZ?

Simulated.

In that case, please verify if loaders were not targetting wrong nodes. If so then this is a test issue and could be fixed by using rack_aware_loader: false (but that should be default I think)

[cezarmoise]

Are we sure it's not a loader (c-s / driver) issue? What version are you using?

3.17.3.
When writing it mentions all nodes.
cassandra-stress write no-warmup cl=QUORUM n=120000000 -schema keyspace=keyspace2 'replication(strategy=NetworkTopologyStrategy,replication_factor=3)' -mode cql3 native -rate 'threads=100 fixed=10000/s' -col 'size=FIXED(128) n=FIXED(8)' -pop seq=240000001..360000000 -node 10.12.2.232,10.12.3.165,10.12.3.114,10.12.2.12,10.12.0.103,10.12.3.71 -errors skip-unsupported-columns

In the loader logs there is nothing expect some connection refused errors to node9, but those can be explained by the fact that the node was overloaded since it was the only one in the rack receiving.

[cezarmoise]

Are we sure it's not a loader (c-s / driver) issue? What version are you using?

Are you using simulared racks or real AZ?

Simulated.

In that case, please verify if loaders were not targetting wrong nodes. If so then this is a test issue and could be fixed by using rack_aware_loader: false (but that should be default I think)

it is false.
each loader writes to all nodes

[tgrabiec]

This issue is a combination of things.

First, right after table is created you hit #23378, which puts all the tablets on the small nodes.

Then, it rebalances, but you have only 8 tablets (1 tablet per shard, initially 3x 2 shards = 6) and we have much more shards later, and equal per-table balance is not reached due to granularity/inaccuracy. That's the reason we recommend 10 tablets/shard now and it's computed dynamically after d99df7a (not in 2025.1).

So this issue should be fixed in master.

[cezarmoise]

This issue is a combination of things.

First, right after table is created you hit #23378, which puts all the tablets on the small nodes.

Then, it rebalances, but you have only 8 tablets (1 tablet per shard, initially 3x 2 shards = 6) and we have much more shards later, and equal per-table balance is not reached due to granularity/inaccuracy. That's the reason we recommend 10 tablets/shard now and it's computed dynamically after d99df7a (not in 2025.1).

So this issue should be fixed in master.

Something similar in master #23631