Can't restore schema if DC names of original and restored clusters are not the same

[mikliapko]

Preconditions:

• Backup snapshot created with Scylla 2024.2 and has the next keyspace cql_stmt in schema.json file:

"cql_stmt":"CREATE KEYSPACE \"10gb_sizetiered_2024_2\" WITH replication = {'class': 'org.apache.cassandra.locator.NetworkTopologyStrategy', 'eu-west': '3'} AND durable_writes = true AND tablets = {'enabled': false};"

where replication is defined as {'class': 'org.apache.cassandra.locator.NetworkTopologyStrategy', 'eu-west': '3'} (dc name is eu-west).

Steps:

1. Restore schema to the new cluster where datacenter name differs from the name in snapshot (for example, eu-west-1).
2. Monitor the restore task progress

Result:
Schema restore fails because of dc names mismatch - schema cql statement has the dc name eu-west whereas in the restore cluster it is eu-west-1:

Restore progress
Run:		711229c2-115c-11f0-8f78-0a7e0b70e983
Status:		ERROR (restoring backed-up data)
Cause:		restore data: create "10gb_sizetiered_2024_2" ("10gb_sizetiered_2024_2") with CREATE KEYSPACE "10gb_sizetiered_2024_2" WITH replication = {'class': 'org.apache.cassandra.locator.NetworkTopologyStrategy', 'eu-west': '3'} AND durable_writes = true AND tablets = {'enabled': false}: Unrecognized strategy option {eu-west} passed to org.apache.cassandra.locator.NetworkTopologyStrategy for keyspace 10gb_sizetiered_2024_2
Start time:	04 Apr 25 13:55:18 UTC
End time:	04 Apr 25 13:55:18 UTC
Duration:	0s
Progress:	0% | 0%
Snapshot Tag:	sm_20241203091417UTC
Bandwidth:
  - Download:    unknown
  - Load&stream: unknown

╭───────────────┬──────────┬────────────┬─────────┬────────────┬────────╮
│ Keyspace      │ Progress │       Size │ Success │ Downloaded │ Failed │
├───────────────┼──────────┼────────────┼─────────┼────────────┼────────┤
│ system_schema │  0% | 0% │ 393.173KiB │      0B │         0B │     0B │
╰───────────────┴──────────┴────────────┴─────────┴────────────┴────────╯

Expected result:
I believe, we should have an opportunity to handle such cases in Manager.
In recent release 3.5.0 we introduced the new flag --dc-mapping.
It works only in restore tables mode. Perhaps, we can enhance it and make to work in the scenario described above for schema restore.

[mikliapko]

cc: @karol-kokoszka @Michal-Leszczynski @VAveryanov8 Guys, WDYT?

[mikliapko]

related to #4308

[Michal-Leszczynski]

It works only in restore tables mode. Perhaps, we can enhance it and make to work in the scenario described above for schema restore.

This capability is also needed for the 1-1 restore and I advocated that it shouldn't be done by SM.

The reason for that is that SM stores schema in text (CQL) format, and we currently don't have a proper way of parsing it, so that we could replace the datacenters according to the --dc-mapping flag. In general, parsing is tricky and if we were to do it, then we should do it properly and keep on supporting it in the future, even when Scylla's CQL is extended.

So the idea is that if schema needs to be changed, it's user's responsibility to make those changes.
In order to make it simpler for the user, SM could simplify the process of getting schema from the backup (e.g. additional flag to --restore-schema which just prints the schema and nothing more, so that user can change it and apply manually, which should be easy).

[mikliapko]

So the idea is that if schema needs to be changed, it's user's responsibility to make those changes.
In order to make it simpler for the user, SM could simplify the process of getting schema from the backup (e.g. additional flag to --restore-schema which just prints the schema and nothing more, so that user can change it and apply manually, which should be easy).

@Michal-Leszczynski
It looks like we need to fix it for Nemesis ASAP.
Could you please share/describe the flow the Manager does while restoring the schema?
I believe we can do the same in the test via making direct CQL calls.