Automated backups are broken with ScyllaDB OS >=6.0 and Enterprise >=2024.2 when ScyllaClusters are set up with AuthN

[rzetelskik]

What happened?

As per https://manager.docs.scylladb.com/stable/sctool/backup.html#skip-schema:
For ScyllaDB versions starting at 6.0 and 2024.2, SM requires CQL credentials to back up the schema. This is the case since SM 3.3.2: scylladb/scylla-manager#4008.

We don't set up AuthN or AuthZ in our ScyllaDB Manager integration tests. When set up with AuthN and AuthZ, backup doesn't succeed: see e.g. https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator/2544/pull-scylla-operator-master-e2e-gke-parallel/1899420143805534208#1:test-build-log.txt%3A2435.

What did you expect to happen?

Automated backup procedure should work when AuthN and AuthZ are configured.

How can we reproduce it (as minimally and precisely as possible)?

See https://github.com/rzetelskik/scylla-operator/compare/b3c2a3871da6095c64bacb7986f960e10af042e1..45bd57c6e7c36ba475604c1c8c80d4d71071269e.

Scylla Operator version

master

Kubernetes platform name and version

n/a

Please attach the must-gather archive.

https://gcsweb.scylla-operator.scylladb.com/gcs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator/2544/pull-scylla-operator-master-e2e-gke-parallel/1899420143805534208/ (e2e collection failed here as CI is broken now)

Anything else we need to know?

Backups can be configured to skip schema backup. However, in that case, the restore procedure is impossible without an annoying workaround.

We have a few ways to approach this:

1. Update the documentation with information about providing credentials to SM server before scheduling a backup. We can't claim support for automated backups in that case though.
2. On cluster creation, create a dedicated user for operator/manager use through the maintenance socket and provide credentials to SM on cluster registration. This would require providing client certificate and key to SM for SM to use CQL over SSL (ref Configure scylla-manager with CQL over TLS #2398, most likely impossible with unmanaged SM Edit: possible, SM API takes cert/key pair in bytes despite the name indicating a filepath). In this case we should also switch to using HTTPS for manager-controller to SM API communication,
3. SM could provide a way for agent to use the maintenance socket to describe schema. This would require some effort from SM developers. In that case, SM agent would effectively serve as reverse proxy for the maintenance socket, which in the current state would be insecure with just the auth token. Edit: see Michal's comment below.

[rzetelskik]

fyi @mflendrich @zimnx

[Michal-Leszczynski]

3. SM could provide a way for agent to use the maintenance socket to describe schema. This would require some effort from SM developers. In that case, SM agent would effectively serve as reverse proxy for the maintenance socket, which in the current state would be insecure with just the auth token.

SM issue for using maintenance socket: scylladb/scylla-manager#3831
We haven't proceeded with it because it would solve CQL access issues only for scenarios where this socket is exposed (and I don't think that's the default Scylla behavior), but if it solves some problems for operator (and perhaps cloud), then we can include in our planning.

cc: @karol-kokoszka @tzach

[rzetelskik]

@Michal-Leszczynski what's the outlook on using the maintenance socket for restore and moving away from having to provide credentials to SM server altogether?

[Michal-Leszczynski]

It enables interaction with the node through CQL protocol without authentication. It gives full-permission access. The maintenance socket is available by Unix domain socket with file permissions 755, thus it is not accessible from outside of the node and from other POSIX groups on the node. It is created before the node joins the cluster.

Since maintenance socket grants full-permission access, it should be suitable for restore purposes. SM-agent wouldn't forward the CQL session to SM, as this would violate the only local exposure of maintenance socket, but it would have some additional endpoints which would basically just extend Scylla API with things like:

• DESC SCHEMA WITH INTERNALS
• get all views names
• alter table tombstone_gc mode (is it ok to expose such thing via API?)
• and more

On the other hand, there is a slight problem with schema restoration, as it applies some arbitrary CREATE and ALTER statements, which basically means that it would require SM-agent to expose an endpoint which allows for executing arbitrary CQL statemets, which would violate the only local exposure...

We could make it so those CQL statements are not passed to SM-agent in a request, but rather SM-agent would need to fetch the schema.cql file from the backup itself, which could provide more safety, but I'm still not sure if that's the way to go (e.g. access to backup location and SM allows for dropping all tables).

It seems a little bit hacky to use only locally exposed maintenance socked with full permission by SM-agent proxy. IMHO it makes more sense to use the maintenance socket to create a user with limited permissions that is going to be provided to SM. It also allows for disabling the maintenance socket in general, as user might not want to have it exposed at all. Having a possibility to configure CQL access for operator controller might also come in handy for other things related to configuring Scylla (if not now, then in the future) - especially that the general idea is to get rid of SM in some time.

Also, when maintenance socket is set to ignore, SM would still require the CQL username and password, which adds complexity in a way that sometimes SM uses CQL session for let's say altering table, and sometimes it uses SM-agent API for the same thing.

[rzetelskik]

We decided to provisionally address this as a part of #1896 by documenting the need for providing CQL credentials to SM server with AuthN configured.
We're postponing the decision on an approach to fixing this, although we're leaning towards resolving this on the Operator side.

[scylla-operator-bot]

The Scylla Operator project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 30d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out

/lifecycle stale