Remote Sync Enhancements

[HexF]

Have you searched for an existing feature request?

• Yes, I tried searching

Brief Summary

The current remote sync implementation only allows for two simple actions - uploading and downloading a database. It does not allow for file locking, or progress to be reflected in the UI.

My proposal is to abstract the remote sync interface to introduce the concept of providers which would give support for merging more than simple upload/download commands.

I propose a RemoteSyncProvider base class with methods for:

• Start Synchronisation (this could be used to lock the remote file)
• Download Database (to a byte-stream in memory - no need to hit the disk)
• Upload Database
• Finish Synchronisation (either abort, or finished cleanly)
  Signals for:
• Reporting progress as a percentage
• Reporting logging messages

On top of this RemoteSyncProvide, I propose 2 providers:

• External tool (as we have currently, with additional scripts for start/finish)
• WebDAV

It would also be nice if we could recall credentials contained within the open database when synchronising. I propose doing this through nominating credentials in the UI, which would be passed through environment variables

I would also like to see a sync-on-save option, so every time I save my database it will be synchronised if possible.

Example

This feature ideally should allow me to configure a remote path to synchronise my database to which would be housed on my NextCloud server.

Each time I save my database after making changes, I should have my database merged automagically with the file stored on my configured remote server.
I would get progress bars for all of downloading, merging and re-uploading my database, with logging output from the provider, see the mockup below:

I could also foresee additional providers for other cloud storage solutions such as Google Drive, OneDrive and others.

Context

Issues such as #11854 and #9409 seem to provide minimal additional reasoning, and get closed with 'mount the WebDAV folder natively'.
I sometimes work in locations without internet access, so mounting the WebDAV folder doesn't work in my case.
When using tools such as the Nextcloud Desktop client, if I modify entries in two places, the synchronisation this provides is not aware of how to merge two Keepass databases, and will end up losing changes.

While simply using a tool such as rclone may work, I have run into a situation at least once where two devices have still managed to overwrite each other because the file was not locked between the download and upload phases of the merge. Providing a native way to synchronise with WebDAV could overcome this issue.

I synchronise with Nextcloud, for which my credentials are stored in the database.
I found no good way to get credentials out of my database when synchronising, instead resorting to hard-coding an access token into my script, which is not ideal.
The CLI last I used it was not interfacing with the GUI application, and required me to unlock my database with my master password - which would be even worse to store on disk.

[HexF]

Some reason the feature request template wasn't letting me submit - throwing some cryptic, so none of the labels are attached to this issue

[droidmonkey]

I like this idea, and it's a good transition from the current implementation, which is decidedly clunky.

[HexF]

@t-h-e - Any thoughts around this?

[droidmonkey]

Also need to make the sequence a little more explicit to the end user: #7222 (comment)

[matsmcp]

Current keeshare has (in my eyes) a number of shortcomings

• it doesn't support folder structures making it unusable if you have many Entries (1000+)
• it may fail to sync without telling the user. (SMB share gets disconnected. The change gets stored in the local database only and doesn't give an error message - the user goes home and the next user tries the old password from the keeshare which doesn't work)

Add to this that MS has deprecated webdav (https://learn.microsoft.com/en-us/windows/whats-new/deprecated-features)
SMB shouldn't be used over internet

I would like to see

• A reliable sync so that we do not loose passwords/changes
• Preserved folder structure
• A sync that works cross plattform and is "internet-safe" (maybe using SSH/SFTP with some kind of locking and versioning)

[t-h-e]

The current remote sync implementation only allows for two simple actions - uploading and downloading a database.

It already does a merge of the current database and the downloaded one

It does not allow for file locking, or progress to be reflected in the UI.

True, file locking is missing, but the progress is reflected in the UI. Bottom right corner similar to the "Clearing the clipboard in X seconds" progress bar.

concept of providers which would give support for merging more than simple upload/download commands.

Having a concept of providers was my initial suggestion, which was shut down due to potential maintenance overhead. Instead a more general approach was chosen to let the user run any command.

Download Database (to a byte-stream in memory - no need to hit the disk)

I like the idea of having the downloaded database only in memory 👍

It would also be nice if we could recall credentials contained within the open database when synchronising.

Yes, definitely. It was on my ToDo list, but I did not get around to it.

I would also like to see a sync-on-save option, so every time I save my database it will be synchronised if possible.

Yes, definitely would be nice as well.

---

@HexF regarding your use case of syncing via nextcloud. This is possible as long as your system provides a tool like curl.
See Accessing files using cURL from the nextcloud documentation.
You simply have to set a curl command to download a file from nextcloud and the corresponding upload command in the upload tab. Merging happens automatically.
Additionally, the merge via the sync implementation does not require you to enter your password again for merging, as long as both databases use the same key.

I completely agree that the sync functionality is somewhat clunky, especially the little information given to the user if something goes wrong is a pain which may lead to unnecessary complaints. Additionally further improvements can be made as you suggested, like:

• file locking or keeping the downloaded database in memory
• being able to use credentials from the open database
• maybe have a more explicit dialog as in your mock up

The reason I have not worked further on polishing the user experience is because I already have three open PRs that are awaiting merging and I did not want to put more work in until those get merged.

• Fix open remote database #11682
• Add KeePass header check for testing remote download (#10910) #10915
• Use secure temp directory for remote sync (#10911) #10914

Don't get me wrong, I'm happy to help improve KeepassXC and the sync implementation and I understand that reviewing and managing an open source project like this is a lot of work. Thank you very much @droidmonkey.

[droidmonkey]

Having a concept of providers was my initial suggestion, which was shut down due to potential maintenance overhead. Instead a more general approach was chosen to let the user run any command

Perhaps we can meet this half way and offer known templates for providers: sftp, curl, webdav, etc. Something like common commands and their common parameters / input.

I am making progress on clearing PR's. My time has been very limited lately.

[t-h-e]

Great. Sounds like a plan. I will look into having templates/providers as well as the other improvements

[moritzthecat]

for a bi-directional sync to get offline-first approach working to my understanding

• a sync on open and
• a sync on close (if things changed)
  is necessary.
  So far I am using the the KeePass triggers for this, which I think is a really flexible method. Maybe worth to look at.