Error: create backup target: location is not accessible

[timtimb0t]

Packages

Scylla version: 6.3.0~dev-20250108.e51b2075dacc with build-id 1ffc83e51d7f78126ce77667ff1140f5f4913518

Kernel Version: 6.8.0-1021-aws

Issue description

During the disrupt_mgmt_backup_specific_keyspaces execution the following error occurred:

Traceback (most recent call last):
  File "/home/ubuntu/scylla-cluster-tests/sdcm/nemesis.py", line 5486, in wrapper
    result = method(*args[1:], **kwargs)
  File "/home/ubuntu/scylla-cluster-tests/sdcm/nemesis.py", line 3015, in disrupt_mgmt_backup_specific_keyspaces
    self._mgmt_backup(backup_specific_tables=True)
  File "/home/ubuntu/scylla-cluster-tests/sdcm/sct_events/group_common_events.py", line 534, in wrapper
    return decorated_func(*args, **kwargs)
  File "/home/ubuntu/scylla-cluster-tests/sdcm/sct_events/group_common_events.py", line 519, in inner_func
    return func(*args, **kwargs)
  File "/home/ubuntu/scylla-cluster-tests/sdcm/nemesis.py", line 3173, in _mgmt_backup
    mgr_task = mgr_cluster.create_backup_task(location_list=[location, ], keyspace_list=non_test_keyspaces)
  File "/home/ubuntu/scylla-cluster-tests/sdcm/mgmt/cli.py", line 671, in create_backup_task
    res = self.sctool.run(cmd=cmd, parse_table_res=False)
  File "/home/ubuntu/scylla-cluster-tests/sdcm/mgmt/cli.py", line 1208, in run
    raise ScyllaManagerError(f"Encountered an error on sctool command: {cmd}: {ex}") from ex
sdcm.mgmt.common.ScyllaManagerError: Encountered an error on sctool command: backup -c e4d8d68d-804e-418a-b109-86a08a23ebc7 --keyspace keyspace1,ks_truncate,mview  --location s3:manager-backup-tests-us-east-1 : Encountered a bad command exit code!

Command: 'sudo sctool backup -c e4d8d68d-804e-418a-b109-86a08a23ebc7 --keyspace keyspace1,ks_truncate,mview  --location s3:manager-backup-tests-us-east-1 '

Exit code: 1

Stdout:



Stderr:

Error: create backup target: location is not accessible
 10.4.23.157: agent [HTTP 400] operation put: s3 upload: 301 Moved Permanently: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint. (code:PermanentRedirect) - make sure the location is correct and credentials are set, to debug SSH to 10.4.23.157 and run "scylla-manager-agent check-location -L s3:manager-backup-tests-us-east-1 --debug"
Trace ID: Y6YlRSxsT52kKzvH4R0Hpw (grep in scylla-manager logs)

Error occurred during creation of backup task, not sure about the root cause of such behaviour

Impact

Backup task failure

How frequently does it reproduce?

Describe the frequency with how this issue can be reproduced.

Installation details

Cluster size: 6 nodes (i4i.4xlarge)

Scylla Nodes used in this run:

• longevity-tls-50gb-3d-master-db-node-2dcd4a4a-9 (34.251.4.141 | 10.4.22.20) (shards: 14)
• longevity-tls-50gb-3d-master-db-node-2dcd4a4a-8 (34.241.26.106 | 10.4.20.131) (shards: 14)
• longevity-tls-50gb-3d-master-db-node-2dcd4a4a-7 (3.254.126.153 | 10.4.22.124) (shards: -1)
• longevity-tls-50gb-3d-master-db-node-2dcd4a4a-6 (52.208.146.125 | 10.4.22.186) (shards: 14)
• longevity-tls-50gb-3d-master-db-node-2dcd4a4a-5 (54.247.14.114 | 10.4.22.170) (shards: 14)
• longevity-tls-50gb-3d-master-db-node-2dcd4a4a-4 (54.155.14.93 | 10.4.22.97) (shards: 14)
• longevity-tls-50gb-3d-master-db-node-2dcd4a4a-3 (54.194.32.152 | 10.4.20.122) (shards: 14)
• longevity-tls-50gb-3d-master-db-node-2dcd4a4a-2 (54.220.6.171 | 10.4.20.202) (shards: 14)
• longevity-tls-50gb-3d-master-db-node-2dcd4a4a-16 (54.76.98.223 | 10.4.23.157) (shards: 14)
• longevity-tls-50gb-3d-master-db-node-2dcd4a4a-15 (52.19.27.230 | 10.4.22.242) (shards: 14)
• longevity-tls-50gb-3d-master-db-node-2dcd4a4a-14 (54.76.156.247 | 10.4.21.108) (shards: 14)
• longevity-tls-50gb-3d-master-db-node-2dcd4a4a-13 (52.210.61.57 | 10.4.20.224) (shards: 14)
• longevity-tls-50gb-3d-master-db-node-2dcd4a4a-12 (52.212.247.74 | 10.4.21.143) (shards: 14)
• longevity-tls-50gb-3d-master-db-node-2dcd4a4a-11 (52.213.210.215 | 10.4.23.96) (shards: 14)
• longevity-tls-50gb-3d-master-db-node-2dcd4a4a-10 (54.171.113.110 | 10.4.22.53) (shards: 14)
• longevity-tls-50gb-3d-master-db-node-2dcd4a4a-1 (54.247.50.94 | 10.4.23.234) (shards: 14)

OS / Image: ami-0419ef0a7ad763693 (NO RUNNER: NO RUNNER)

Test: longevity-50gb-3days-test
Test id: 2dcd4a4a-e69e-492e-b62c-eb5f73fd311d
Test name: scylla-master/tier1/longevity-50gb-3days-test
Test method: longevity_test.LongevityTest.test_custom_time
Test config file(s):

• longevity-50GB-3days-authorization-and-tls-ssl.yaml

Logs and commands

• Restore Monitor Stack command: $ hydra investigate show-monitor 2dcd4a4a-e69e-492e-b62c-eb5f73fd311d
• Restore monitor on AWS instance using Jenkins job
• Show all stored logs command: $ hydra investigate show-logs 2dcd4a4a-e69e-492e-b62c-eb5f73fd311d

Logs:

• longevity-tls-50gb-3d-master-db-node-2dcd4a4a-7 - https://cloudius-jenkins-test.s3.amazonaws.com/2dcd4a4a-e69e-492e-b62c-eb5f73fd311d/20250111_040914/longevity-tls-50gb-3d-master-db-node-2dcd4a4a-7-2dcd4a4a.tar.gz
• longevity-tls-50gb-3d-master-db-node-2dcd4a4a-1 - https://cloudius-jenkins-test.s3.amazonaws.com/2dcd4a4a-e69e-492e-b62c-eb5f73fd311d/20250111_040914/longevity-tls-50gb-3d-master-db-node-2dcd4a4a-1-2dcd4a4a.tar.gz
• longevity-tls-50gb-3d-master-db-node-2dcd4a4a-9 - https://cloudius-jenkins-test.s3.amazonaws.com/2dcd4a4a-e69e-492e-b62c-eb5f73fd311d/20250111_040914/longevity-tls-50gb-3d-master-db-node-2dcd4a4a-9-2dcd4a4a.tar.gz
• longevity-tls-50gb-3d-master-db-node-2dcd4a4a-3 - https://cloudius-jenkins-test.s3.amazonaws.com/2dcd4a4a-e69e-492e-b62c-eb5f73fd311d/20250111_040914/longevity-tls-50gb-3d-master-db-node-2dcd4a4a-3-2dcd4a4a.tar.gz
• longevity-tls-50gb-3d-master-db-node-2dcd4a4a-2 - https://cloudius-jenkins-test.s3.amazonaws.com/2dcd4a4a-e69e-492e-b62c-eb5f73fd311d/20250111_040914/longevity-tls-50gb-3d-master-db-node-2dcd4a4a-2-2dcd4a4a.tar.gz
• longevity-tls-50gb-3d-master-db-node-2dcd4a4a-8 - https://cloudius-jenkins-test.s3.amazonaws.com/2dcd4a4a-e69e-492e-b62c-eb5f73fd311d/20250111_040914/longevity-tls-50gb-3d-master-db-node-2dcd4a4a-8-2dcd4a4a.tar.gz
• longevity-tls-50gb-3d-master-db-node-2dcd4a4a-4 - https://cloudius-jenkins-test.s3.amazonaws.com/2dcd4a4a-e69e-492e-b62c-eb5f73fd311d/20250111_040914/longevity-tls-50gb-3d-master-db-node-2dcd4a4a-4-2dcd4a4a.tar.gz
• longevity-tls-50gb-3d-master-db-node-2dcd4a4a-11 - https://cloudius-jenkins-test.s3.amazonaws.com/2dcd4a4a-e69e-492e-b62c-eb5f73fd311d/20250111_040914/longevity-tls-50gb-3d-master-db-node-2dcd4a4a-11-2dcd4a4a.tar.gz
• longevity-tls-50gb-3d-master-db-node-2dcd4a4a-15 - https://cloudius-jenkins-test.s3.amazonaws.com/2dcd4a4a-e69e-492e-b62c-eb5f73fd311d/20250111_040914/longevity-tls-50gb-3d-master-db-node-2dcd4a4a-15-2dcd4a4a.tar.gz
• longevity-tls-50gb-3d-master-db-node-2dcd4a4a-10 - https://cloudius-jenkins-test.s3.amazonaws.com/2dcd4a4a-e69e-492e-b62c-eb5f73fd311d/20250111_040914/longevity-tls-50gb-3d-master-db-node-2dcd4a4a-10-2dcd4a4a.tar.gz
• core.scylla-longevity-tls-50gb-3d-master-db-node-2dcd4a4a-2-2025-01-11_08-16-09.gz - https://storage.cloud.google.com/upload.scylladb.com/core.scylla.106.2b65d5bc72984b1fa3ab947ab44d01bd.5851.1736581275000000./core.scylla.106.2b65d5bc72984b1fa3ab947ab44d01bd.5851.1736581275000000.zst
• core.scylla-longevity-tls-50gb-3d-master-db-node-2dcd4a4a-6-2025-01-11_09-41-29.gz - https://storage.cloud.google.com/upload.scylladb.com/core.scylla.106.fed17cc1b03e4b92a20563e4c93248d8.5724.1736586273000000./core.scylla.106.fed17cc1b03e4b92a20563e4c93248d8.5724.1736586273000000.zst
• core.scylla-longevity-tls-50gb-3d-master-db-node-2dcd4a4a-5-2025-01-11_10-12-15.gz - https://storage.cloud.google.com/upload.scylladb.com/core.scylla.106.1a2c17d4326141168accae5a5dfb2b3a.5663.1736588211000000./core.scylla.106.1a2c17d4326141168accae5a5dfb2b3a.5663.1736588211000000.zst
• core.scylla-longevity-tls-50gb-3d-master-db-node-2dcd4a4a-5-2025-01-11_10-19-52.gz - https://storage.cloud.google.com/upload.scylladb.com/core.scylla.106.1a2c17d4326141168accae5a5dfb2b3a.16339.1736589551000000./core.scylla.106.1a2c17d4326141168accae5a5dfb2b3a.16339.1736589551000000.zst
• core.scylla-longevity-tls-50gb-3d-master-db-node-2dcd4a4a-2-2025-01-11_13-37-55.gz - https://storage.cloud.google.com/upload.scylladb.com/core.scylla.106.2b65d5bc72984b1fa3ab947ab44d01bd.14401.1736600232000000./core.scylla.106.2b65d5bc72984b1fa3ab947ab44d01bd.14401.1736600232000000.zst
• core.scylla-longevity-tls-50gb-3d-master-db-node-2dcd4a4a-12-2025-01-11_19-37-20.gz - https://storage.cloud.google.com/upload.scylladb.com/core.scylla.106.b762dab68f1d4f6a80cf267f4ecb4469.5630.1736622253000000./core.scylla.106.b762dab68f1d4f6a80cf267f4ecb4469.5630.1736622253000000.zst
• core.scylla-longevity-tls-50gb-3d-master-db-node-2dcd4a4a-5-2025-01-11_20-32-32.gz - https://storage.cloud.google.com/upload.scylladb.com/core.scylla.106.1a2c17d4326141168accae5a5dfb2b3a.16892.1736625353000000./core.scylla.106.1a2c17d4326141168accae5a5dfb2b3a.16892.1736625353000000.zst
• core.scylla-longevity-tls-50gb-3d-master-db-node-2dcd4a4a-6-2025-01-11_20-44-37.gz - https://storage.cloud.google.com/upload.scylladb.com/core.scylla.106.fed17cc1b03e4b92a20563e4c93248d8.16726.1736626121000000./core.scylla.106.fed17cc1b03e4b92a20563e4c93248d8.16726.1736626121000000.zst
• core.scylla-longevity-tls-50gb-3d-master-db-node-2dcd4a4a-10-2025-01-11_20-53-50.gz - https://storage.cloud.google.com/upload.scylladb.com/core.scylla.106.84dae8ea81be44e0a0a609dab28ea36b.1009.1736626860000000./core.scylla.106.84dae8ea81be44e0a0a609dab28ea36b.1009.1736626860000000.zst
• core.scylla-longevity-tls-50gb-3d-master-db-node-2dcd4a4a-12-2025-01-11_21-17-33.gz - https://storage.cloud.google.com/upload.scylladb.com/core.scylla.106.b762dab68f1d4f6a80cf267f4ecb4469.11889.1736627567000000./core.scylla.106.b762dab68f1d4f6a80cf267f4ecb4469.11889.1736627567000000.zst
• core.scylla-longevity-tls-50gb-3d-master-db-node-2dcd4a4a-14-2025-01-11_21-39-44.gz - https://storage.cloud.google.com/upload.scylladb.com/core.scylla.106.d32582065e4d4a359dff01da43258c12.5713.1736629066000000./core.scylla.106.d32582065e4d4a359dff01da43258c12.5713.1736629066000000.zst
• core.scylla-longevity-tls-50gb-3d-master-db-node-2dcd4a4a-13-2025-01-11_21-42-15.gz - https://storage.cloud.google.com/upload.scylladb.com/core.scylla.106.e5ccef9e9fb5427b9d5dc8fbc7570b91.5703.1736628285000000./core.scylla.106.e5ccef9e9fb5427b9d5dc8fbc7570b91.5703.1736628285000000.zst
• core.scylla-longevity-tls-50gb-3d-master-db-node-2dcd4a4a-6-2025-01-11_22-32-37.gz - https://storage.cloud.google.com/upload.scylladb.com/core.scylla.106.fed17cc1b03e4b92a20563e4c93248d8.37734.1736632686000000./core.scylla.106.fed17cc1b03e4b92a20563e4c93248d8.37734.1736632686000000.zst
• core.scylla-longevity-tls-50gb-3d-master-db-node-2dcd4a4a-5-2025-01-11_22-42-59.gz - https://storage.cloud.google.com/upload.scylladb.com/core.scylla.106.1a2c17d4326141168accae5a5dfb2b3a.46723.1736631950000000./core.scylla.106.1a2c17d4326141168accae5a5dfb2b3a.46723.1736631950000000.zst
• core.scylla-longevity-tls-50gb-3d-master-db-node-2dcd4a4a-12-2025-01-11_22-43-38.gz - https://storage.cloud.google.com/upload.scylladb.com/core.scylla.106.b762dab68f1d4f6a80cf267f4ecb4469.14110.1736633405000000./core.scylla.106.b762dab68f1d4f6a80cf267f4ecb4469.14110.1736633405000000.zst
• core.scylla-longevity-tls-50gb-3d-master-db-node-2dcd4a4a-13-2025-01-11_22-57-16.gz - https://storage.cloud.google.com/upload.scylladb.com/core.scylla.106.e5ccef9e9fb5427b9d5dc8fbc7570b91.14250.1736634125000000./core.scylla.106.e5ccef9e9fb5427b9d5dc8fbc7570b91.14250.1736634125000000.zst
• core.scylla-longevity-tls-50gb-3d-master-db-node-2dcd4a4a-14-2025-01-11_23-12-05.gz - https://storage.cloud.google.com/upload.scylladb.com/core.scylla.106.d32582065e4d4a359dff01da43258c12.13722.1736634869000000./core.scylla.106.d32582065e4d4a359dff01da43258c12.13722.1736634869000000.zst
• core.scylla-longevity-tls-50gb-3d-master-db-node-2dcd4a4a-16-2025-01-11_23-16-06.gz - https://storage.cloud.google.com/upload.scylladb.com/core.scylla.106.b0e52631767d4c5c8a5d727cfe66fe99.5688.1736635567000000./core.scylla.106.b0e52631767d4c5c8a5d727cfe66fe99.5688.1736635567000000.zst
• core.scylla-longevity-tls-50gb-3d-master-db-node-2dcd4a4a-12-2025-01-12_00-29-22.gz - https://storage.cloud.google.com/upload.scylladb.com/core.scylla.106.b762dab68f1d4f6a80cf267f4ecb4469.16778.1736639818000000./core.scylla.106.b762dab68f1d4f6a80cf267f4ecb4469.16778.1736639818000000.zst
• core.scylla-longevity-tls-50gb-3d-master-db-node-2dcd4a4a-13-2025-01-12_02-51-03.gz - https://storage.cloud.google.com/upload.scylladb.com/core.scylla.106.495c22596c9a4253bd8300c12d92457e.964.1736648291000000./core.scylla.106.495c22596c9a4253bd8300c12d92457e.964.1736648291000000.zst
• db-cluster-2dcd4a4a.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/2dcd4a4a-e69e-492e-b62c-eb5f73fd311d/20250112_044103/db-cluster-2dcd4a4a.tar.gz
• sct-runner-events-2dcd4a4a.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/2dcd4a4a-e69e-492e-b62c-eb5f73fd311d/20250112_044103/sct-runner-events-2dcd4a4a.tar.gz
• sct-2dcd4a4a.log.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/2dcd4a4a-e69e-492e-b62c-eb5f73fd311d/20250112_044103/sct-2dcd4a4a.log.tar.gz
• loader-set-2dcd4a4a.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/2dcd4a4a-e69e-492e-b62c-eb5f73fd311d/20250112_044103/loader-set-2dcd4a4a.tar.gz
• monitor-set-2dcd4a4a.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/2dcd4a4a-e69e-492e-b62c-eb5f73fd311d/20250112_044103/monitor-set-2dcd4a4a.tar.gz
• ssl-conf-2dcd4a4a.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/2dcd4a4a-e69e-492e-b62c-eb5f73fd311d/20250112_044103/ssl-conf-2dcd4a4a.tar.gz

Jenkins job URL
Argus

[timtimb0t]

@VAveryanov8 , @Michal-Leszczynski could you please take a look at this issue? maybe some additional information is needed?

[timtimb0t]

Packages

Scylla version: 6.3.0~dev-20250108.e51b2075dacc with build-id 1ffc83e51d7f78126ce77667ff1140f5f4913518

Kernel Version: 6.8.0-1021-aws

Installation details

Cluster size: 4 nodes (i4i.4xlarge)

Scylla Nodes used in this run:

• longevity-50gb-12h-master-db-node-bf0c7579-7 (63.34.7.22 | 10.4.8.238) (shards: 13)
• longevity-50gb-12h-master-db-node-bf0c7579-6 (34.245.207.49 | 10.4.11.63) (shards: 12)
• longevity-50gb-12h-master-db-node-bf0c7579-5 (18.201.25.47 | 10.4.10.152) (shards: 12)
• longevity-50gb-12h-master-db-node-bf0c7579-4 (3.255.173.221 | 10.4.11.204) (shards: 13)
• longevity-50gb-12h-master-db-node-bf0c7579-3 (54.76.108.105 | 10.4.11.166) (shards: 12)
• longevity-50gb-12h-master-db-node-bf0c7579-2 (3.250.88.220 | 10.4.11.229) (shards: 12)
• longevity-50gb-12h-master-db-node-bf0c7579-1 (54.217.164.238 | 10.4.10.126) (shards: 8)

OS / Image: ami-0419ef0a7ad763693 (NO RUNNER: NO RUNNER)

Test: longevity-150gb-asymmetric-cluster-12h-test
Test id: bf0c7579-e48d-4814-9625-8a3f1193c8d7
Test name: scylla-master/tier1/longevity-150gb-asymmetric-cluster-12h-test
Test method: longevity_test.LongevityTest.test_custom_time
Test config file(s):

• longevity-150GB-12h-autorization-LimitedMonkey.yaml

Logs and commands

• Restore Monitor Stack command: $ hydra investigate show-monitor bf0c7579-e48d-4814-9625-8a3f1193c8d7
• Restore monitor on AWS instance using Jenkins job
• Show all stored logs command: $ hydra investigate show-logs bf0c7579-e48d-4814-9625-8a3f1193c8d7

Logs:

• longevity-50gb-12h-master-db-node-bf0c7579-5 - https://cloudius-jenkins-test.s3.amazonaws.com/bf0c7579-e48d-4814-9625-8a3f1193c8d7/20250111_041403/longevity-50gb-12h-master-db-node-bf0c7579-5-bf0c7579.tar.gz
• longevity-50gb-12h-master-db-node-bf0c7579-6 - https://cloudius-jenkins-test.s3.amazonaws.com/bf0c7579-e48d-4814-9625-8a3f1193c8d7/20250111_041403/longevity-50gb-12h-master-db-node-bf0c7579-6-bf0c7579.tar.gz
• longevity-50gb-12h-master-db-node-bf0c7579-2 - https://cloudius-jenkins-test.s3.amazonaws.com/bf0c7579-e48d-4814-9625-8a3f1193c8d7/20250111_041403/longevity-50gb-12h-master-db-node-bf0c7579-2-bf0c7579.tar.gz
• core.scylla-longevity-50gb-12h-master-db-node-bf0c7579-3-2025-01-11_21-01-53.gz - https://storage.cloud.google.com/upload.scylladb.com/core.scylla.106.e413da9d671c48d682abad5758f58d45.9780.1736626496000000./core.scylla.106.e413da9d671c48d682abad5758f58d45.9780.1736626496000000.zst
• db-cluster-bf0c7579.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/bf0c7579-e48d-4814-9625-8a3f1193c8d7/20250111_211355/db-cluster-bf0c7579.tar.gz
• sct-runner-events-bf0c7579.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/bf0c7579-e48d-4814-9625-8a3f1193c8d7/20250111_211355/sct-runner-events-bf0c7579.tar.gz
• sct-bf0c7579.log.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/bf0c7579-e48d-4814-9625-8a3f1193c8d7/20250111_211355/sct-bf0c7579.log.tar.gz
• loader-set-bf0c7579.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/bf0c7579-e48d-4814-9625-8a3f1193c8d7/20250111_211355/loader-set-bf0c7579.tar.gz
• monitor-set-bf0c7579.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/bf0c7579-e48d-4814-9625-8a3f1193c8d7/20250111_211355/monitor-set-bf0c7579.tar.gz
• ssl-conf-bf0c7579.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/bf0c7579-e48d-4814-9625-8a3f1193c8d7/20250111_211355/ssl-conf-bf0c7579.tar.gz

Jenkins job URL
Argus

[timtimb0t]

Packages

Scylla version: 6.3.0~dev-20250108.e51b2075dacc with build-id 1ffc83e51d7f78126ce77667ff1140f5f4913518

Kernel Version: 6.8.0-1021-aws

Installation details

Cluster size: 6 nodes (i4i.4xlarge)

Scylla Nodes used in this run:

• longevity-tls-50gb-3d-master-db-node-2dcd4a4a-9 (34.251.4.141 | 10.4.22.20) (shards: 14)
• longevity-tls-50gb-3d-master-db-node-2dcd4a4a-8 (34.241.26.106 | 10.4.20.131) (shards: 14)
• longevity-tls-50gb-3d-master-db-node-2dcd4a4a-7 (3.254.126.153 | 10.4.22.124) (shards: -1)
• longevity-tls-50gb-3d-master-db-node-2dcd4a4a-6 (52.208.146.125 | 10.4.22.186) (shards: 14)
• longevity-tls-50gb-3d-master-db-node-2dcd4a4a-5 (54.247.14.114 | 10.4.22.170) (shards: 14)
• longevity-tls-50gb-3d-master-db-node-2dcd4a4a-4 (54.155.14.93 | 10.4.22.97) (shards: 14)
• longevity-tls-50gb-3d-master-db-node-2dcd4a4a-3 (54.194.32.152 | 10.4.20.122) (shards: 14)
• longevity-tls-50gb-3d-master-db-node-2dcd4a4a-2 (54.220.6.171 | 10.4.20.202) (shards: 14)
• longevity-tls-50gb-3d-master-db-node-2dcd4a4a-16 (54.76.98.223 | 10.4.23.157) (shards: 14)
• longevity-tls-50gb-3d-master-db-node-2dcd4a4a-15 (52.19.27.230 | 10.4.22.242) (shards: 14)
• longevity-tls-50gb-3d-master-db-node-2dcd4a4a-14 (54.76.156.247 | 10.4.21.108) (shards: 14)
• longevity-tls-50gb-3d-master-db-node-2dcd4a4a-13 (52.210.61.57 | 10.4.20.224) (shards: 14)
• longevity-tls-50gb-3d-master-db-node-2dcd4a4a-12 (52.212.247.74 | 10.4.21.143) (shards: 14)
• longevity-tls-50gb-3d-master-db-node-2dcd4a4a-11 (52.213.210.215 | 10.4.23.96) (shards: 14)
• longevity-tls-50gb-3d-master-db-node-2dcd4a4a-10 (54.171.113.110 | 10.4.22.53) (shards: 14)
• longevity-tls-50gb-3d-master-db-node-2dcd4a4a-1 (54.247.50.94 | 10.4.23.234) (shards: 14)

OS / Image: ami-0419ef0a7ad763693 (NO RUNNER: NO RUNNER)

Test: longevity-50gb-3days-test
Test id: 2dcd4a4a-e69e-492e-b62c-eb5f73fd311d
Test name: scylla-master/tier1/longevity-50gb-3days-test
Test method: longevity_test.LongevityTest.test_custom_time
Test config file(s):

• longevity-50GB-3days-authorization-and-tls-ssl.yaml

Logs and commands

• Restore Monitor Stack command: $ hydra investigate show-monitor 2dcd4a4a-e69e-492e-b62c-eb5f73fd311d
• Restore monitor on AWS instance using Jenkins job
• Show all stored logs command: $ hydra investigate show-logs 2dcd4a4a-e69e-492e-b62c-eb5f73fd311d

Logs:

• longevity-tls-50gb-3d-master-db-node-2dcd4a4a-7 - https://cloudius-jenkins-test.s3.amazonaws.com/2dcd4a4a-e69e-492e-b62c-eb5f73fd311d/20250111_040914/longevity-tls-50gb-3d-master-db-node-2dcd4a4a-7-2dcd4a4a.tar.gz
• longevity-tls-50gb-3d-master-db-node-2dcd4a4a-1 - https://cloudius-jenkins-test.s3.amazonaws.com/2dcd4a4a-e69e-492e-b62c-eb5f73fd311d/20250111_040914/longevity-tls-50gb-3d-master-db-node-2dcd4a4a-1-2dcd4a4a.tar.gz
• longevity-tls-50gb-3d-master-db-node-2dcd4a4a-9 - https://cloudius-jenkins-test.s3.amazonaws.com/2dcd4a4a-e69e-492e-b62c-eb5f73fd311d/20250111_040914/longevity-tls-50gb-3d-master-db-node-2dcd4a4a-9-2dcd4a4a.tar.gz
• longevity-tls-50gb-3d-master-db-node-2dcd4a4a-3 - https://cloudius-jenkins-test.s3.amazonaws.com/2dcd4a4a-e69e-492e-b62c-eb5f73fd311d/20250111_040914/longevity-tls-50gb-3d-master-db-node-2dcd4a4a-3-2dcd4a4a.tar.gz
• longevity-tls-50gb-3d-master-db-node-2dcd4a4a-2 - https://cloudius-jenkins-test.s3.amazonaws.com/2dcd4a4a-e69e-492e-b62c-eb5f73fd311d/20250111_040914/longevity-tls-50gb-3d-master-db-node-2dcd4a4a-2-2dcd4a4a.tar.gz
• longevity-tls-50gb-3d-master-db-node-2dcd4a4a-8 - https://cloudius-jenkins-test.s3.amazonaws.com/2dcd4a4a-e69e-492e-b62c-eb5f73fd311d/20250111_040914/longevity-tls-50gb-3d-master-db-node-2dcd4a4a-8-2dcd4a4a.tar.gz
• longevity-tls-50gb-3d-master-db-node-2dcd4a4a-4 - https://cloudius-jenkins-test.s3.amazonaws.com/2dcd4a4a-e69e-492e-b62c-eb5f73fd311d/20250111_040914/longevity-tls-50gb-3d-master-db-node-2dcd4a4a-4-2dcd4a4a.tar.gz
• longevity-tls-50gb-3d-master-db-node-2dcd4a4a-11 - https://cloudius-jenkins-test.s3.amazonaws.com/2dcd4a4a-e69e-492e-b62c-eb5f73fd311d/20250111_040914/longevity-tls-50gb-3d-master-db-node-2dcd4a4a-11-2dcd4a4a.tar.gz
• longevity-tls-50gb-3d-master-db-node-2dcd4a4a-15 - https://cloudius-jenkins-test.s3.amazonaws.com/2dcd4a4a-e69e-492e-b62c-eb5f73fd311d/20250111_040914/longevity-tls-50gb-3d-master-db-node-2dcd4a4a-15-2dcd4a4a.tar.gz
• longevity-tls-50gb-3d-master-db-node-2dcd4a4a-10 - https://cloudius-jenkins-test.s3.amazonaws.com/2dcd4a4a-e69e-492e-b62c-eb5f73fd311d/20250111_040914/longevity-tls-50gb-3d-master-db-node-2dcd4a4a-10-2dcd4a4a.tar.gz
• core.scylla-longevity-tls-50gb-3d-master-db-node-2dcd4a4a-2-2025-01-11_08-16-09.gz - https://storage.cloud.google.com/upload.scylladb.com/core.scylla.106.2b65d5bc72984b1fa3ab947ab44d01bd.5851.1736581275000000./core.scylla.106.2b65d5bc72984b1fa3ab947ab44d01bd.5851.1736581275000000.zst
• core.scylla-longevity-tls-50gb-3d-master-db-node-2dcd4a4a-6-2025-01-11_09-41-29.gz - https://storage.cloud.google.com/upload.scylladb.com/core.scylla.106.fed17cc1b03e4b92a20563e4c93248d8.5724.1736586273000000./core.scylla.106.fed17cc1b03e4b92a20563e4c93248d8.5724.1736586273000000.zst
• core.scylla-longevity-tls-50gb-3d-master-db-node-2dcd4a4a-5-2025-01-11_10-12-15.gz - https://storage.cloud.google.com/upload.scylladb.com/core.scylla.106.1a2c17d4326141168accae5a5dfb2b3a.5663.1736588211000000./core.scylla.106.1a2c17d4326141168accae5a5dfb2b3a.5663.1736588211000000.zst
• core.scylla-longevity-tls-50gb-3d-master-db-node-2dcd4a4a-5-2025-01-11_10-19-52.gz - https://storage.cloud.google.com/upload.scylladb.com/core.scylla.106.1a2c17d4326141168accae5a5dfb2b3a.16339.1736589551000000./core.scylla.106.1a2c17d4326141168accae5a5dfb2b3a.16339.1736589551000000.zst
• core.scylla-longevity-tls-50gb-3d-master-db-node-2dcd4a4a-2-2025-01-11_13-37-55.gz - https://storage.cloud.google.com/upload.scylladb.com/core.scylla.106.2b65d5bc72984b1fa3ab947ab44d01bd.14401.1736600232000000./core.scylla.106.2b65d5bc72984b1fa3ab947ab44d01bd.14401.1736600232000000.zst
• core.scylla-longevity-tls-50gb-3d-master-db-node-2dcd4a4a-12-2025-01-11_19-37-20.gz - https://storage.cloud.google.com/upload.scylladb.com/core.scylla.106.b762dab68f1d4f6a80cf267f4ecb4469.5630.1736622253000000./core.scylla.106.b762dab68f1d4f6a80cf267f4ecb4469.5630.1736622253000000.zst
• core.scylla-longevity-tls-50gb-3d-master-db-node-2dcd4a4a-5-2025-01-11_20-32-32.gz - https://storage.cloud.google.com/upload.scylladb.com/core.scylla.106.1a2c17d4326141168accae5a5dfb2b3a.16892.1736625353000000./core.scylla.106.1a2c17d4326141168accae5a5dfb2b3a.16892.1736625353000000.zst
• core.scylla-longevity-tls-50gb-3d-master-db-node-2dcd4a4a-6-2025-01-11_20-44-37.gz - https://storage.cloud.google.com/upload.scylladb.com/core.scylla.106.fed17cc1b03e4b92a20563e4c93248d8.16726.1736626121000000./core.scylla.106.fed17cc1b03e4b92a20563e4c93248d8.16726.1736626121000000.zst
• core.scylla-longevity-tls-50gb-3d-master-db-node-2dcd4a4a-10-2025-01-11_20-53-50.gz - https://storage.cloud.google.com/upload.scylladb.com/core.scylla.106.84dae8ea81be44e0a0a609dab28ea36b.1009.1736626860000000./core.scylla.106.84dae8ea81be44e0a0a609dab28ea36b.1009.1736626860000000.zst
• core.scylla-longevity-tls-50gb-3d-master-db-node-2dcd4a4a-12-2025-01-11_21-17-33.gz - https://storage.cloud.google.com/upload.scylladb.com/core.scylla.106.b762dab68f1d4f6a80cf267f4ecb4469.11889.1736627567000000./core.scylla.106.b762dab68f1d4f6a80cf267f4ecb4469.11889.1736627567000000.zst
• core.scylla-longevity-tls-50gb-3d-master-db-node-2dcd4a4a-14-2025-01-11_21-39-44.gz - https://storage.cloud.google.com/upload.scylladb.com/core.scylla.106.d32582065e4d4a359dff01da43258c12.5713.1736629066000000./core.scylla.106.d32582065e4d4a359dff01da43258c12.5713.1736629066000000.zst
• core.scylla-longevity-tls-50gb-3d-master-db-node-2dcd4a4a-13-2025-01-11_21-42-15.gz - https://storage.cloud.google.com/upload.scylladb.com/core.scylla.106.e5ccef9e9fb5427b9d5dc8fbc7570b91.5703.1736628285000000./core.scylla.106.e5ccef9e9fb5427b9d5dc8fbc7570b91.5703.1736628285000000.zst
• core.scylla-longevity-tls-50gb-3d-master-db-node-2dcd4a4a-6-2025-01-11_22-32-37.gz - https://storage.cloud.google.com/upload.scylladb.com/core.scylla.106.fed17cc1b03e4b92a20563e4c93248d8.37734.1736632686000000./core.scylla.106.fed17cc1b03e4b92a20563e4c93248d8.37734.1736632686000000.zst
• core.scylla-longevity-tls-50gb-3d-master-db-node-2dcd4a4a-5-2025-01-11_22-42-59.gz - https://storage.cloud.google.com/upload.scylladb.com/core.scylla.106.1a2c17d4326141168accae5a5dfb2b3a.46723.1736631950000000./core.scylla.106.1a2c17d4326141168accae5a5dfb2b3a.46723.1736631950000000.zst
• core.scylla-longevity-tls-50gb-3d-master-db-node-2dcd4a4a-12-2025-01-11_22-43-38.gz - https://storage.cloud.google.com/upload.scylladb.com/core.scylla.106.b762dab68f1d4f6a80cf267f4ecb4469.14110.1736633405000000./core.scylla.106.b762dab68f1d4f6a80cf267f4ecb4469.14110.1736633405000000.zst
• core.scylla-longevity-tls-50gb-3d-master-db-node-2dcd4a4a-13-2025-01-11_22-57-16.gz - https://storage.cloud.google.com/upload.scylladb.com/core.scylla.106.e5ccef9e9fb5427b9d5dc8fbc7570b91.14250.1736634125000000./core.scylla.106.e5ccef9e9fb5427b9d5dc8fbc7570b91.14250.1736634125000000.zst
• core.scylla-longevity-tls-50gb-3d-master-db-node-2dcd4a4a-14-2025-01-11_23-12-05.gz - https://storage.cloud.google.com/upload.scylladb.com/core.scylla.106.d32582065e4d4a359dff01da43258c12.13722.1736634869000000./core.scylla.106.d32582065e4d4a359dff01da43258c12.13722.1736634869000000.zst
• core.scylla-longevity-tls-50gb-3d-master-db-node-2dcd4a4a-16-2025-01-11_23-16-06.gz - https://storage.cloud.google.com/upload.scylladb.com/core.scylla.106.b0e52631767d4c5c8a5d727cfe66fe99.5688.1736635567000000./core.scylla.106.b0e52631767d4c5c8a5d727cfe66fe99.5688.1736635567000000.zst
• core.scylla-longevity-tls-50gb-3d-master-db-node-2dcd4a4a-12-2025-01-12_00-29-22.gz - https://storage.cloud.google.com/upload.scylladb.com/core.scylla.106.b762dab68f1d4f6a80cf267f4ecb4469.16778.1736639818000000./core.scylla.106.b762dab68f1d4f6a80cf267f4ecb4469.16778.1736639818000000.zst
• core.scylla-longevity-tls-50gb-3d-master-db-node-2dcd4a4a-13-2025-01-12_02-51-03.gz - https://storage.cloud.google.com/upload.scylladb.com/core.scylla.106.495c22596c9a4253bd8300c12d92457e.964.1736648291000000./core.scylla.106.495c22596c9a4253bd8300c12d92457e.964.1736648291000000.zst
• db-cluster-2dcd4a4a.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/2dcd4a4a-e69e-492e-b62c-eb5f73fd311d/20250112_044103/db-cluster-2dcd4a4a.tar.gz
• sct-runner-events-2dcd4a4a.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/2dcd4a4a-e69e-492e-b62c-eb5f73fd311d/20250112_044103/sct-runner-events-2dcd4a4a.tar.gz
• sct-2dcd4a4a.log.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/2dcd4a4a-e69e-492e-b62c-eb5f73fd311d/20250112_044103/sct-2dcd4a4a.log.tar.gz
• loader-set-2dcd4a4a.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/2dcd4a4a-e69e-492e-b62c-eb5f73fd311d/20250112_044103/loader-set-2dcd4a4a.tar.gz
• monitor-set-2dcd4a4a.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/2dcd4a4a-e69e-492e-b62c-eb5f73fd311d/20250112_044103/monitor-set-2dcd4a4a.tar.gz
• ssl-conf-2dcd4a4a.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/2dcd4a4a-e69e-492e-b62c-eb5f73fd311d/20250112_044103/ssl-conf-2dcd4a4a.tar.gz

Jenkins job URL
Argus

[VAveryanov8]

It looks to me that:

1. Scylla manager agent was running in eu-west region
2. I assume from bucket name manager-backup-tests-us-east-1 that it's created in us-east-1 region ( correct me if I'm wrong)
3. scylla-manager-agent.yaml regarding s3 look empty -

auth_token: ***
prometheus: :5090
s3: {}
tls_cert_file: null
tls_key_file: null

Accordingly to https://manager.docs.scylladb.com/stable/backup/setup-amazon-s3.html#config-file

If the S3 bucket is not running in the same region as the AWS EC2 instance uncomment and set the region to the S3 bucket’s region.

I think changing scylla-manager-agent.yaml to smth like:

auth_token: ***
prometheus: :5090
s3: 
  region: us-east-1
tls_cert_file: null
tls_key_file: null

should fix the issue.

In the meantime, I think we need to discuss with @Michal-Leszczynski and @karol-kokoszka if it's possible to handle this redirect automatically and do we want to do it.

[Michal-Leszczynski]

In the meantime, I think we need to discuss with @Michal-Leszczynski and @karol-kokoszka if it's possible to handle this redirect automatically and do we want to do it.

I think it's possible, but I would give it a low priority.