Add parallel rsync sessions with fpsync

[ruaridhg]

Towards #197

This PR:

• rsync.py strips the trailing / to handle the user specifying the rsync paths with or without a trailing / and then adds a trailing / to rsync_source since it is required for the rsync command
• rsync.py replaces simple rsync command with fpsync which runs 8 parallel rsync sessions

[HChughtai]

fpsync looks useful, but a couple of comments. Have you seen improvements in practice using it?

[HChughtai]

Why 8 parallel jobs? Will this depend on the number of physical and logical CPU cores, and so need to be something that is different dependning on the machine running it?

[ruaridhg]

Yeah it will depend on physical and logical CPU cores so maybe having 8 as a default max and I could have it as an optional input i.e. parallel_jobs to the cli rsync command

Then within the cli command:

@app.command
def rsync(
    source: str,
    source_rsync: str,
    destination_rsync: str,
    log_dir: pathlib.Path = pathlib.Path("logs"),
    source_projects: list[str] | None = None,
    parallel_jobs: int | None = None,
) -> None:
    if parallel_jobs is None:
          parallel_jobs = min(8, os.cpu_count() or 1)
    
     run_rsync(
        destination_rsync,
        destination_proj,
        source_rsync,
        source_proj,
        parallel_jobs,
        )

What do you think @HChughtai ?

[HChughtai]

Yeah, looks like a sensible approach. Only thing I would say is to add some handling so that you can't pass in numbers less than 1

[ruaridhg]

Sounds good, I'll do those changes and ask you to re-review after

[ruaridhg]

fpsync looks useful, but a couple of comments. Have you seen improvements in practice using it?

Yep definitely improvements using it
7.7G (~7.5 minutes, 5,119 files) /data/xnat/archive/EPISURG (using fpsync ~90s so ~5x faster)
https://github.com/UCL-MIRSG/MIRSG/issues/354#issuecomment-4841335684