Preferred Backup Synchronisation Tool Poll

[astrom0]

In short, this is a poll to gain an understanding of synchronisation tools applied by Greyhole users, the pros/cons of a given tool and to share helpful tips in their use.

For context (the long story), see below for what inspired asking the question.

I might have missed it but I haven't found a lot of guidance around synchronisation strategies when using Greyhole to back up file storage.

Possibly that's because the assumption is that Greyhole can be configured to store multiple files across multiple discs (so using Greyhole as I am is not typical).

My use case is different to this. How I'm working is that I've got a NAS that's designed for routine access and is configured with a RAID strategy to provide some level of data storage resilience. I've set up Greyhole simply (for context, Greyhole is loaded onto a 5 disc Thecus NAS that I've hacked that is loaded with a lightweight Ubuntu flavour) where it's configured to store 1 copy of the files that it receives from the NAS during a sync, somewhere across one of the 5 discs. For my situation, I consider that to be satisfactory to give me back up in depth.

With the sync tool, my plan is to sync the NAS with the Greyhole LZ using a simple "mirror" strategy, so that if a file is deleted on the NAS, then Greyhole will mirror this, and delete its copy as well, during a sync. I'm running the Greyhole box as cold storage, with the plan that every week or so I'll boot the box into gear and run a sync, fsck and then balance.

Seemed pretty simple to me but the challenges that I've worked through are the nuances associated with symbolic links and having those links intelligently treated by the sync tool.

I started out using FreeFileSync (FFS) purely out of familiarity of using it in a Windows environment. The tool is FOSS and is multi-platform so it can be run on Linux, as well. FFS can be configured to follow symlinks and it does this competently such that it will follow the link to the target file and compare it with the source. If the source file has been deleted, then it appears to delete the target without issue (I'm testing this right now, so hopefully I'll confirm this in short order).

I'm using FFS in 1-on-1 mode via its GUI with a plan to use its batch operation capability to set up the tool in command line mode, once the bugs are ironed out and the sync strategy is bedded in. Having the GUI is useful for a low tech person like myself because it provides visual clarity of the sync job via its comparison function that allows the user to confirm the sync configuration before executing it.

From what I can make out (and my understanding may be flawed), FFS is not the most intelligent when dealing with moved files. If a file has been moved at the source, FFS will delete the original target file and then copy the source file from its new location to a new location on the target. Not the best strategy in terms of operation time, especially if the relocated file is large.

Because of this shortcoming, I thought that rsync would be a better tool. However, what I've found out is that rsync doesn't work well (for me anyway) with symlinks. If I configure rsync to mirror (i.e. set the --del option to delete files in the target that have disappeared from the source) rsync won't honour the symlink. Setting the option --keep-dirlinks doesn't make any difference because rsync doesn't appear to have the AI awareness to follow the link on the target. So, what happens is that rsync deletes the symlink on the target (because there is no matching symlink on the source) and then copies the source file across to the LZ, which defeats the point of Greyhole's clever method of distributing files across its span of discs.

As a result, I've decided to park rsync and stick with FFS. As a result of my journey, I thought I'd check to see how others deal with this and compare notes. I look forward to improving my understanding and possibly finding a better way to achieve my goal.

[uSlackr]

I would agree with your first observation, you are using Greyhole in a largely non-standard way. I have no need for a file sync tool as I use greyhole to do the sync.
That said, take a look at syncthing

[gboudreau]

Why don't you mount the Greyhole (Samba) shares remotely and then use whatever tool you'd like to sync your local files with the Samba files?
You wouldn't need to fsck or balance. The Greyhole daemon would take care of handling new & removed files.

Of note: I doubt you'll find a tool that could identify a large file that was renamed, and rename it on the target, instead of copying it again. Some backup tools (eg. duplicacy) would allow you to "deduplicate" files, and thus not have to re-copy files to a remote if the same data is already there because of other files, but I'm pretty sure no "sync" tool supports this feature.