Have a command for purging orphaned snapshots

[dkropachev]

Currently s-m does not purge snapshots when backup job is deleted
In the sake of usability we need to have a way to purge these snapshots in sctool and have it properly documented
Another nice to have feature would be an option to delete snapshots when job is deleted

Tasks

Preview Give feedback

1. sctool backup delete to not require cluster to be currently managed by scylla manager #3467
   backup bug good first issue sctool +4
2. sctool backup list to require either --cluster-id or --all-clusters #3468
   backup bug good first issue sctool +4
3. Document how to find and remove snapshots with sctool. #3469
   backup documentation +2

[Michal-Leszczynski]

In the sake of usability we need to have a way to purge these snapshots in sctool and have it properly documented

We have sctool backup delete --cluster <id|name> --snapshot-tag <tag> for deleting specific backups and
sctool backup list --cluster <id|name> for listing all backups, so deleting orphaned snapshots can be achieved by combining those commands.

[karol-kokoszka]

All the commands are here.
Example:

➜  scylla-manager git:(docs_add_agent_config_change_prereq) ✗ ./sctool backup list --all-clusters -c 364e4cc0-6f69-4f2e-a83a-a35f603d3ac2 -L s3:backuptest-smoke
Cluster: 79eb5d3d-6648-441f-8f10-73fc4aecb5fa

backup/5aca45d0-9ccc-44cb-bf8e-88823621c1ed
Snapshots:
  - sm_20230630133111UTC (2.988G, 3 nodes)
  - sm_20230630133032UTC (2.988G, 3 nodes)
Keyspaces:
  - system_schema (13 tables)
  - backuptest_data (1 table)

Cluster: ea031891-0357-436c-acea-9fb7f1661a90

backup/0dc7c09b-d083-45e8-8c14-ad8ab7ca0f95
Snapshots:
  - sm_20230710220000UTC (4.008M, 6 nodes)
Keyspaces:
  - system_traces (5 tables)
  - system_distributed_everywhere (1 table)
  - system_distributed (4 tables)
  - system_schema (13 tables)
  - system_auth (4 tables)

backup/5c6803ee-9674-4de8-9414-13335eacd2e6
Snapshots:
  - sm_20230710221000UTC (3.488M, 6 nodes)
Keyspaces:
  - system_distributed_everywhere (1 table)
  - system_schema (13 tables)
  - system_auth (4 tables)
  - system_traces (5 tables)
  - system_distributed (4 tables)

https://manager.docs.scylladb.com/stable/sctool/backup.html#backup-list
https://manager.docs.scylladb.com/stable/sctool/backup.html#backup-delete

But sctool backup delete requires to pass cluster-id. If the cluster doesn't exist then it fails. Hehehe, so seems it's useless in the current state.

➜  scylla-manager git:(docs_add_agent_config_change_prereq) ✗ ./sctool backup delete -T sm_20230710221000UTC -L s3:backuptest-smoke 
Error: required flag(s) "cluster" not set