(send|receive)_files API of RemoteCmdRunner is failing on ubuntu24 when scp is used as transport

[dimakr]

send_files, receive_files API of RemoteCmdRunner object is failing on ubuntu24 when scp is used for transfering files, with the error like:

Traceback (most recent call last):
  File "/home/dmitriy/Work/Scylla/sct_serialize_stress_results/sdcm/tester.py", line 194, in wrapper
    return method(*args, **kwargs)
  File "/home/dmitriy/Work/Scylla/sct_serialize_stress_results/sdcm/utils/decorators.py", line 121, in inner
    res = func(*args, **kwargs)
  File "/home/dmitriy/Work/Scylla/sct_serialize_stress_results/performance_regression_test.py", line 64, in setUp
    super().setUp()
  File "/home/dmitriy/Work/Scylla/sct_serialize_stress_results/sdcm/tester.py", line 194, in wrapper
    return method(*args, **kwargs)
  File "/home/dmitriy/Work/Scylla/sct_serialize_stress_results/sdcm/utils/decorators.py", line 121, in inner
    res = func(*args, **kwargs)
  File "/home/dmitriy/Work/Scylla/sct_serialize_stress_results/sdcm/tester.py", line 1062, in setUp
    future.result()
  File "/home/dmitriy/.pyenv/versions/3.10.0/lib/python3.10/concurrent/futures/_base.py", line 438, in result
    return self.__get_result()
  File "/home/dmitriy/.pyenv/versions/3.10.0/lib/python3.10/concurrent/futures/_base.py", line 390, in __get_result
    raise self._exception
  File "/home/dmitriy/.pyenv/versions/3.10.0/lib/python3.10/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/dmitriy/Work/Scylla/sct_serialize_stress_results/sdcm/tester.py", line 1006, in _create_db_clusters
    self.init_nodes(db_cluster=db_cluster)
  File "/home/dmitriy/Work/Scylla/sct_serialize_stress_results/sdcm/tester.py", line 761, in init_nodes
    db_cluster.wait_for_init(node_list=db_cluster.nodes)
  File "/home/dmitriy/Work/Scylla/sct_serialize_stress_results/sdcm/cluster.py", line 4013, in wrapper
    verify_node_setup_or_startup(start_time, setup_queue, setup_results)
  File "/home/dmitriy/Work/Scylla/sct_serialize_stress_results/sdcm/cluster.py", line 3961, in verify_node_setup_or_startup
    raise NodeSetupFailed(
sdcm.cluster.NodeSetupFailed: [Node elasticity-test-VERSION-DESC-db-node-a8236ba5-2 [100.27.33.122 | 10.12.0.230]] NodeSetupFailed: Encountered a bad command exit code!

Command: 'scp -r -o StrictHostKeyChecking=no -o BatchMode=yes -o ConnectTimeout=300 -o ServerAliveInterval=300 -o UserKnownHostsFile=/tmp/tmpcllrllaf -P 22 -i /home/dmitriy/.ssh/scylla_test_id_ed25519  /tmp/sct4ule3i3v/scylla.yaml \'scyllaadm@[100.27.33.122]:"/tmp/tmp.qaq9PnAghO"\''

Exit code: 1
Stdout:
Stderr:

scp: dest open ""/tmp/tmp.qaq9PnAghO"": No such file or directory
scp: failed to upload file /tmp/sct4ule3i3v/scylla.yaml to "/tmp/tmp.qaq9PnAghO"

The issue is likely caused by the fact that on ubuntu24 the scp switched to using sftp protocol under the hood. And this is likely started to cause problems in the places where we encode multiple remote paths for send/receive_files API:

scylla-cluster-tests/sdcm/remote/remote_base.py

Line 401 in 93e7488

return '%s@[%s]:"%s"' % (self.user, self.hostname, " ".join(paths))

Seems like sftp is not processing double quotes within single quotes the same way as it was previously in scp.

[fruch]

@dimakr where did you seen this issue ? and why we don't have rsync available ?

[dimakr]

@dimakr where did you seen this issue ? and why we don't have rsync available ?

I started seeing it regularly when running tests from local machine (using it as sct-runner), after I upgraded from u22 to u24.
I checked it during runtime - some nodes of a basic pr-provision-test cluster don't have rsync installed (don't remember exactly the role of a node, but seems like that was a DB one).

[dimakr]

After small research it seems like many were having similar issues after upgrading to ubuntu24, and had to add this backward compatibility option in their systems.

[fruch]

isn't you installing rsync on you machine solves the issue ?

[dimakr]

isn't you installing rsync on you machine solves the issue ?

I have rsync installed. The RemoteCmdRunnerBase._check_rsync check was failing for remote host.

[fruch]

isn't you installing rsync on you machine solves the issue ?

I have rsync installed. The RemoteCmdRunnerBase._check_rsync check was failing for remote host.

on which node type ?

[fruch]

the code is uber weird, why it checks the host, when the rsync command is gonna be executed locally ?

[fruch]

it's since this was introduced in:
5b4ea12

after reading again, we need rsync on both way, to be able to send data in both directions

anyhow we should install rsync on all of our nodes, if we have situation it's missing