SM 3.5.0 DTest failures

[Michal-Leszczynski]

A few DTests have failed when testing against SM master (as it is a candidate for SM 3.5.0), but it looks like all of the failures need to be handled by adjusting the tests. Note that those DTests were ran against master Scylla. Here is a list of them with some explanations:

dtest-enterprise-no-tablets:

• test_repairing_a_downed_node - asserted error message needs to be updated:

>           assert "One of the nodes is down. Task cannot be created." in err.args[0], f"Unexpected error: {err.args[0]}"
E           AssertionError: Unexpected error: Encountered an error on sctool command: ['repair', '--keyspace', 'ks', '--cluster', '2b331ee9-eee1-484d-ad75-aee6a62220a7']: ('/jenkins/workspace/manager-master/dtest-enterprise-no-tablets/scylla-enterprise/.dtest/dtest-_m0wud56/test/scylla-manager/bin/sctool --api-url http://127.0.41.1:5080/api/v1 repair --keyspace ks --cluster 2b331ee9-eee1-484d-ad75-aee6a62220a7', 1, '', "Error: create repair target: repairing DCs with down nodes won't be successful: [127.0.41.3]. Please either exclude down nodes from being repaired with --ignore-down-hosts flag, or filter out DCs with down nodes with --dc flag\nTrace ID: ZlqAHyIpRAG32qc93I39SQ (grep in scylla-manager logs)\n\n")
E           assert 'One of the nodes is down. Task cannot be created.' in 'Encountered an error on sctool command: [\'repair\', \'--keyspace\', \'ks\', \'--cluster\', \'2b331ee9-eee1-484d-ad75-aee6a62220a7\']: (\'/jenkins/workspace/manager-master/dtest-enterprise-no-tablets/scylla-enterprise/.dtest/dtest-_m0wud56/test/scylla-manager/bin/sctool --api-url http://127.0.41.1:5080/api/v1 repair --keyspace ks --cluster 2b331ee9-eee1-484d-ad75-aee6a62220a7\', 1, \'\', "Error: create repair target: repairing DCs with down nodes won\'t be successful: [127.0.41.3]. Please either exclude down nodes from being repaired with --ignore-down-hosts flag, or filter out DCs with down nodes with --dc flag\\nTrace ID: ZlqAHyIpRAG32qc93I39SQ (grep in scylla-manager logs)\\n\\n")'

• test_ignore_down_hosts_with_one_down_host - asserted error message needs to be updated (same as above)

dtest-oss-with-tablets:

• test_restore_after_deleting_file_from_s3: deleting files from system_schema (which is not restored) does not result in error:

16:34:41,494 653     urllib3.connectionpool         DEBUG    connectionpool.py   :475  | test_restore_after_deleting_file_from_s3: http://172.17.0.11:9000 "GET /backup-bucket?prefix=backup%2Fsst%2Fcluster%2F4d11ea55-423b-4785-ad72-8331d0c01431%2F&encoding-type=url HTTP/1.1" 200 271693
16:34:41,537 653     manager_backup_tests           INFO     manager_backup_tests.py:170  | test_restore_after_deleting_file_from_s3: Removing file backup/sst/cluster/4d11ea55-423b-4785-ad72-8331d0c01431/dc/datacenter1/node/10489698-fc7e-4039-92ea-7cbd5d309d00/keyspace/system_schema/table/functions/96489b7980be3e14a70166a0b9159450/me-3gp1_19yr_1kl1d2sqmwo4i08arn-big-Statistics.db from bucket backup-bucket

• test_repairing_a_downed_node - asserted error message needs to be updated (same as above)

• test_ignore_down_hosts_with_one_down_host - asserted error message needs to be updated (same as above)

• test_small_table_threshold_parameter - this test looks for the logs from the old repair API when the tablet repair API is used, but more importantly, small_table_threshold is ignored when tablet repair API is used, which is expected, so this test should be for vnodes only:

>       assert table_name in table_names, f"The '{repair_log_message}' message for keyspace '{keyspace_name}.{table_name}' not found!\nThe following logs are found {pformat(logs)} "
E       AssertionError: The 'starting user-requested repair for keyspace keyspace1, repair id' message for keyspace 'keyspace1.cf1' not found!
E         The following logs are found [] 
E       assert 'cf1' in []

@mikliapko Please take a look and validate if my assumptions makes sense.