Upgrade tests fail on index creation on tablets enabled keyspace when starting upgrade from 2025.1.0

[soyacz]

Packages

Scylla version: 2025.1.0-20250325.9dca28d2b818 with build-id 3a706305ef8ac6f14575e4f3441968f67d29d38c

Kernel Version: 6.8.0-1026-gcp

Issue description

• This issue is a regression.
• It is unknown if this issue is a regression.

During upgrade test, at the beginning we try to fill db with data. Because tablets are enabled by default in 2025.1.0, when starting from this version this step fails due creating index in a tablets enabled keyspace:

2025-04-02 12:54:35.186: (TestFrameworkEvent Severity.ERROR) period_type=one-time event_id=e3b0d7a2-188a-48c3-8116-bdcef7cb24c2, source=UpgradeTest.test_rolling_upgrade (upgrade_test.UpgradeTest)() message=Traceback (most recent call last):
File "/home/ubuntu/scylla-cluster-tests/upgrade_test.py", line 629, in test_rolling_upgrade
self.prepare_keyspaces_and_tables()
File "/home/ubuntu/scylla-cluster-tests/sdcm/fill_db_data.py", line 3375, in prepare_keyspaces_and_tables
self.cql_create_tables(session)
File "/home/ubuntu/scylla-cluster-tests/sdcm/fill_db_data.py", line 3132, in cql_create_tables
session.execute(create_table)
File "/home/ubuntu/scylla-cluster-tests/sdcm/utils/common.py", line 1319, in execute_verbose
return execute_orig(*args, **kwargs)
File "cassandra/cluster.py", line 2762, in cassandra.cluster.Session.execute
File "cassandra/cluster.py", line 5166, in cassandra.cluster.ResponseFuture.result
cassandra.InvalidRequest: Error from server: code=2200 [Invalid query] message="Secondary indexes are not supported on base tables with tablets (keyspace 'keyspace_fill_db_data')"

Impact

Failing upgrade test

How frequently does it reproduce?

Describe the frequency with how this issue can be reproduced.

Installation details

Cluster size: 4 nodes (n2-highmem-32)

Scylla Nodes used in this run:

• rolling-upgrade--ubuntu-focal-db-node-f12caf90-0-4 (35.185.75.55 | 10.142.1.7) (shards: 30)
• rolling-upgrade--ubuntu-focal-db-node-f12caf90-0-3 (34.74.208.233 | 10.142.0.241) (shards: 30)
• rolling-upgrade--ubuntu-focal-db-node-f12caf90-0-2 (34.23.237.126 | 10.142.0.35) (shards: 30)
• rolling-upgrade--ubuntu-focal-db-node-f12caf90-0-1 (34.148.79.178 | 10.142.0.196) (shards: 30)

OS / Image: https://www.googleapis.com/compute/v1/projects/scylla-images/global/images/scylladb-2025-1-0 (gce: undefined_region)

Test: rolling-upgrade-gce-image-test
Test id: f12caf90-ee3e-44c9-a283-cd92e7b7b449
Test name: scylla-2025.1/rolling-upgrade/rolling-upgrade-gce-image-test
Test method: upgrade_test.UpgradeTest.test_rolling_upgrade
Test config file(s):

• rolling-upgrade.yaml

Logs and commands

• Restore Monitor Stack command: $ hydra investigate show-monitor f12caf90-ee3e-44c9-a283-cd92e7b7b449
• Restore monitor on AWS instance using Jenkins job
• Show all stored logs command: $ hydra investigate show-logs f12caf90-ee3e-44c9-a283-cd92e7b7b449

Logs:

• db-cluster-f12caf90.tar.zst - https://cloudius-jenkins-test.s3.amazonaws.com/f12caf90-ee3e-44c9-a283-cd92e7b7b449/20250402_125538/db-cluster-f12caf90.tar.zst
• sct-runner-events-f12caf90.tar.zst - https://cloudius-jenkins-test.s3.amazonaws.com/f12caf90-ee3e-44c9-a283-cd92e7b7b449/20250402_125538/sct-runner-events-f12caf90.tar.zst
• sct-f12caf90.log.tar.zst - https://cloudius-jenkins-test.s3.amazonaws.com/f12caf90-ee3e-44c9-a283-cd92e7b7b449/20250402_125538/sct-f12caf90.log.tar.zst
• loader-set-f12caf90.tar.zst - https://cloudius-jenkins-test.s3.amazonaws.com/f12caf90-ee3e-44c9-a283-cd92e7b7b449/20250402_125538/loader-set-f12caf90.tar.zst
• monitor-set-f12caf90.tar.zst - https://cloudius-jenkins-test.s3.amazonaws.com/f12caf90-ee3e-44c9-a283-cd92e7b7b449/20250402_125538/monitor-set-f12caf90.tar.zst
• ssl-conf-f12caf90.tar.zst - https://cloudius-jenkins-test.s3.amazonaws.com/f12caf90-ee3e-44c9-a283-cd92e7b7b449/20250402_125538/ssl-conf-f12caf90.tar.zst
• builder-f12caf90.log.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/f12caf90-ee3e-44c9-a283-cd92e7b7b449/upload_20250402_125702/builder-f12caf90.log.tar.gz

Jenkins job URL
Argus

[soyacz]

I investigated why it started to fail now and fix from #10193 didn't help and my conclusions are:

1. 6.2 does not reject MV/indexes and during upgrade we don't create them, only in final step 'complex profile' did - thus above fix worked
2. Now base version 2025.1 rejects all MV/indexes from the beginning of a test due tablets enabled by default and we have many index creation statements in all_verification_items list

@pehala @roydahan @fruch I propose to:

1. disable tablets by default for upgrade tests (quick fix by adding tablets = {'enabled': false} to keyspace create statements in prepare_keyspaces_and_tables)
2. enhance the test with tablets enabled keyspace that doesn't create MV/Indexes - this is longer one as requires design what we want to test and test it, so basically we would have 2 test keyspaces like keyspace_fill_db_data and keyspace_fill_db_data_tablets - both in one test run.
   IMHO, in future when MV/indexes are supported anyway we should test both tablets on and off.

WDYT?

[pehala]

What about having two tests instead, but have them part of the same job? I worry that having more logic in one upgrade test will make it harder to investigate why the specific tests fail and also make the test harder to understand (we should document how we test artifacts regardless, it is definitely not clear).