remote repo writes extremely slow

[laur89]

Been backing up against the same set of local&remote (rsync.net's) repositories for the past 7 or so years. Lately write operations against remote repo have become crazy bad compared to local. Example of a delete from this morning:

[2024-10-05 03:25:25] [delete-34]       INFO  => starting delete operation on local repo [/backups/]...
[2024-10-05 03:25:25] [delete-34]       INFO  => starting delete operation on remote repo [[email protected]:backups/]...

------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
Deleted data:              -40.99 GB            -37.09 GB            -36.29 GB
All archives:               46.19 GB             26.34 GB              2.60 GB

                       Unique chunks         Total chunks
Chunk index:                   17236               153410
------------------------------------------------------------------------------
terminating with success status, rc 0
[2024-10-05 03:25:44] [delete-34]       INFO  => local repo delete succeeded in 19s


------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
Deleted data:              -40.99 GB            -37.09 GB            -36.56 GB
All archives:               46.19 GB             26.34 GB              2.58 GB

                       Unique chunks         Total chunks
Chunk index:                   17196               152933
------------------------------------------------------------------------------
terminating with success status, rc 0
[2024-10-05 10:35:29] [delete-34]       INFO  => remote repo delete succeeded in 7h:10m:4s

And compact following the delete above:

[2024-10-05 11:37:08] [compact-88]      INFO  => local compact succeeded in 4m:14s
[2024-10-05 13:14:42] [compact-88]      INFO  => remote compact succeeded in 1h:41m:48s

Check w/ --verify-data is not as bad:

[2024-10-05 13:15:55] [check-128]       INFO  => local repo check succeeded in 1m:13s
[2024-10-05 13:23:59] [check-128]       INFO  => remote repo check succeeded in 8m:4s

Note the discrepancy here - local delete took sub-minute, remote 7h. Both deleted the same single archive targeted by the glob param.

Should I just nuke the repo and start from scratch? Is there any value of collecting debug data for the project?

---

Executing borg from an alpine container. Local & remote both running 1.2.8

[ThomasWaldmann]

Could be latency to the remote repo or remote repo performance?

[laur89]

Doubt it's latency. Rsync.net seems ok and my connection speed is the usual 2-300M symmetrical. As for remote repo perfromance - that's my guess; as in something has caused it to degrade. Unsure how to go about making sense of it though?

[ThomasWaldmann]

You could ask rsync.net support if they know something. Maybe they ran some zfs maintenance while you used your repo?

[laur89]

You were right - it were rsync doing some maintenance!