feat: Simplify the SFTP configuration

[Kirari04]

For my personal use I implemented the features to automatically add a repo to known_hosts file and also define an identity file and custom sftp port.
I know that you could just create a ssh config file but when adding multiple repos this becomes an hassle and that's why I preferred to have a simple option inside the ui.

I'm creating this pr in case there is any interest to merge this feature :)
I'm open for any suggestions for improvements.

[CLAassistant]

All committers have signed the CLA.

[garethgeorge]

Hi -- thanks for the interest in contributing. I really like the idea of this feature-- I think SFTP configuration is one of the more significant pain points for new users setting up backrest.

Reading through your impl -- I think a lot is quite reasonable, but something I want to avoid is the new repo properties that are special cased to sftp. The philosophy here being that things being that any new properties need to be handled (i.e. in this case merged into flags or env vars) correctly in perpituity, which makes them expensive for me to maintain.

But changes either to AddRepo, CheckRepoExists, or the UI are relatively cheap-- so I'm wondering if we can make this a UI / CheckRepoExists only change and rely entirely on existing fields of the repo struct.

My initial design ideas are

1. What if we use a magic path to know when we should setup known_hosts or a keyfile for the user? My proposal would be filepath.Join(env.DataDir(), ".ssh")

2. In the CheckRepoExists flow, we check if the repo type is sftp and, if it is, we search for strings that look like a keyfile or known host path in filepath.Join(env.DataDir(), ".ssh") , and we automatically initialize these if present.

   1. in the case of a known_host file, we do the keyscan operation and append the key if not present.
   2. in the case of any other file, we initialize it as an SSH key if it is not present.

The UI change then can be to update the flags management UI to make it easy to add a known_host file or a ssh key to the flags section defaulting to that directory. For an initial impl I think it'd be fine to assume this path is ~/.local/share/backrest though users can technically override this, I can help with the changes needed to handle that properly.

wdyt?

[Kirari04]

Hi @garethgeorge
Switched to the more simplistic approach by generating the flags in the frontend and only adding the host to known_hosts file during check repo function.
Lmk if you like this better.

[basldfalksjdf]

Just chiming in here. Tried to get sftp setup with backrest. It's a major pain and just ended up giving up. This PR sounds awesome if it simplifies that process.

[Kirari04]

Hi @garethgeorge, have you had the time to take a look at the changes?

[garethgeorge]

Sorry about the late review, I've been distracted from backrest for a while with the holidays in the states. This change looks great. I'm happy to go ahead and merge.

One thing I'm noticing you dropped but felt was a cool feature was the ability to automatically generate a key. If you're interested to chase that down, happy to take a followup on that and merge this now. Elsewise it's something I'll consider building on your change down the line.

[BLu85]

Hi,

I am new to backrest.
I haven't been able to run it yet as I get an error when trying to create a repository via SFTP.
in Linux, from the command line, I can create a restic repository running:

restic -o sftp.args="-i /home/remoteuser/.ssh/myprivatekey -pNNNN" -r sftp:[email protected]:/myuser/ init

or, having configured the user and the server in the .ssh/config file, I can do the same with:

restic -r sftp:configUsername:/myuser/ init

However, from the backrest GUI the same commands return errors.

Are the errors due to the fact that backrest needs SSH access before attempting an SFTP connection? I am asking this because the mysuer is chroot-ed and has SFTP access only and no SSH access.
My understanding is that the GUI is primarily a wrapper for the restic commands (other than a backup orchestrator). Is this correct?

Are the errors I am having related to this thread?

Thanks,
Luca

[Kirari04]

@garethgeorge, I’ve been working on some UX improvements, such as adding confirmation prompts.

Regarding automatic SSH-key generation, I plan to implement it using a flow similar to this confirmation modal. On the first connection, it would ask the user to trust the host. On the second connection attempt, it would prompt for a username and password; we would then use those credentials to generate an SSH key, save it to the .ssh folder, and add a flag to use that specific key.

I'll follow up with these changes as soon as I have more time to finish them.

[Kirari04]

@garethgeorge, here is the PoC with both features implemented.

• Configuration is saved via flags (sftp.args).
• SSH keys (generated via username & password) are saved in the .ssh directory.
• known_hosts is updated in the .ssh directory as well.

Workflow Demo:

Enter Credentials: Add the standard information (using Hetzner storage in this example), open the bootstrap options, and enter the SSH credentials.

Test Configuration: Run "Test Config". (might need to do 2 times to confirm host key too)

Confirm Installation: Confirm the installation of the new SSH key.

Completion: The SSH key is installed on the server, saved locally to the .ssh folder, and automatically added to the configuration (using sftp args).

Final Result: This is how the final configuration looks.

[Kirari04]

Are the errors due to the fact that backrest needs SSH access before attempting an SFTP connection? I am asking this because the mysuer is chroot-ed and has SFTP access only and no SSH access. My understanding is that the GUI is primarily a wrapper for the restic commands (other than a backup orchestrator). Is this correct?

@BLu85 you should check the logs, usually it's just a missing SSH key or a "host key verification" issue because .ssh is not mounted inside the docker container. You can look it up in the docs: https://garethgeorge.github.io/backrest/cookbooks/ssh-remote

[BLu85]

Hi @Kirari04,

Thanks for your answer :)

I am not using the docker I am running the executable in Linux.
I am just surprised that the same restic command with the sftp.args added doesn't work, either if the path and port are explicited or if I use the Host name in the config file.

Any idea?
Thanks

[Kirari04]

@BLu85 What are you passing as flags?

[BLu85]

I'm passing
-o sftp.args="-i /home/remoteuser/.ssh/myprivatekey -pNNNN"

When I explicit them.

I don't add any flags if I use the host name in the .ssh/config where the key and the port are defined.

In both cases it doesn't work. I don't have access to my laptop at the moment and can't report the error, sorry.

[BLu85]

This is what I get if I put sftp:resticUserA:/folderA in Repository URI:

[unknown] failed to init repo: command "/usr/bin/restic init --json -o sftp.args=-oBatchMode=yes" failed: exit status 1
Output:
subprocess ssh: ssh: Could not resolve hostname resticUserA: Temporary failure in name resolution
{"message_type":"exit_error","code":1,"message":"Fatal: create repository at sftp:resticUserA:/folderA failed: Fatal: unable to open repository at sftp:resticUserA:/folderA: unable to start the sftp session, error: error receiving version packet from server: server unexpectedly closed connection: unexpected EOF\n"}

Note that in a shell the command restic -r sftp:resticUserA:/folderA init works fine.

I get a SSH key error message when I explicit the user, the server IP, and the args as:

Repository URI: sftp:[email protected]:/folderA
Flags: --option=sftp.args="-i /home/myuser/.ssh/key_UserA -p 22222"

[unknown] failed to init repo: command "/usr/bin/restic init --json --option=sftp.args=-i /home/myuser/.ssh/key_UserA -p 22222" failed: exit status 1
Output:
subprocess ssh: Host key verification failed.
{"message_type":"exit_error","code":1,"message":"Fatal: create repository at sftp:[email protected]:/folderA failed: Fatal: unable to open repository at sftp:[email protected]:/folderA: unable to start the sftp session, error: error receiving version packet from server: server unexpectedly closed connection: unexpected EOF\n"}

The myuser .ssh/known_hosts file is populated with the host key information.

What am I doing wrong?