number of the threads has no effect

[amozaffari]

I am running the cfip on the virtual machine with Ubuntu ( 18.04.3 ) with 3 cpu available for VM. I am testing to copy files 30 ( each 100MB) between two directories, and regardless of how many threats is defined, it takes more or less same amount to rsync them.

python copy-files-in-parallel -j 16 /home/amirpasha/Move_test/Source /home/amirpasha/Move_test/Receiver

Could you please take a look and see if I am using it right or not? Thanks.

log_cfip.txt

[perfecto25]

Im seeing same issue, on centos 7 w 4 CPU cores and 16gb ram ,

copying 6 files, 4 files at 500MB + 2 files at 1GB each

using plain rsync is same as using this wrapper, -j makes no difference

using plain rsync

time rsync -azP /opt/backups qbinfra02:/opt/

sending incremental file list
backups/
backups/testfile1
    524,288,000 100%  102.52MB/s    0:00:04 (xfr#1, to-chk=5/7)
backups/testfile2
    524,288,000 100%  107.27MB/s    0:00:04 (xfr#2, to-chk=4/7)
backups/testfile3
    524,288,000 100%  118.57MB/s    0:00:04 (xfr#3, to-chk=3/7)
backups/testfile4
    524,288,000 100%  130.96MB/s    0:00:03 (xfr#4, to-chk=2/7)
backups/testfile5
  1,048,576,000 100%  126.60MB/s    0:00:07 (xfr#5, to-chk=1/7)
backups/testfile6
  1,048,576,000 100%  124.92MB/s    0:00:08 (xfr#6, to-chk=0/7)
real	0m55.829s
user	0m27.978s
sys	0m0.714s

using wrapper

time prsync -j 8 /opt/backups qbinfra02:/opt/backups
Copying 6 files using 1 jobs!
tmp_dir is /tmp/copy-files-in-parallel/0b6296d8-8b8c-4dd3-8ea9-0b28afd55e02
Computers / CPU cores / Max jobs to run
1:local / 4 / 1
Computer:jobs running/jobs completed/%of started jobs/Average seconds to complete
ETA: 0s Left: 1 AVG: 0.00s  local:1/0/100%/0.0s 
ETA: 0s Left: 0 AVG: 0.00s  local:0/1/100%/55.0s 

real	0m55.175s
user	0m32.100s
sys	0m7.839s

[jochenklar]

Hi, this repo is more or less unsupported, thus the long delay.

We used the script mainly when we copied millions of files, with different sizes, over an infinband network. We had some improvement over plain rsync there, but I don't have any numbers anymore.