SSH experiences / braindump

[jspaaks]

So this issue is a bit of a mess but I needed a way to record some strange things I noticed while preparing materials for a xenon tutorial https://github.com/xenon-middleware/xenon-tutorial/tree/2b00c8d7625cf2966fdfe7c3e2aaeb6046fd7823. With the below items, we at least have concrete things to follow up on. From this collection, we should probably spawn new, singular issues.

1. ssh config options not used xenon-docker-image's Dockerfile because ssh(d) uses the first option it finds in a file even when there are repeated entries, and we echo options at the bottom. E.g. https://github.com/xenon-middleware/xenon-docker-images/blob/a87f1ef876055e19cedcccf4bae4012db2953258/slurm-ssh/Dockerfile#L10-L13. I'd prefer to copy a complete ssh_config file from the Docker context anyway, that way it's easier to see what the options are. Example setup here https://github.com/xenon-middleware/xenon-tutorial/tree/ed69661cdc7125ea15863fa2360fb8fd23309d10/containers

2. docker images for client side and server side, should be separate. e.g. slurm-ssh should be server side since it is a parent to all slurm images, but then the Dockerfile also has https://github.com/xenon-middleware/xenon-docker-images/blob/a87f1ef876055e19cedcccf4bae4012db2953258/slurm-ssh/Dockerfile#L10-L13

3. ssh client docker image should have different configurations, for example

   • with wrong permissions on directories, xenon should complain
   • with and without a ~/.ssh directory
   • with and without a ~/.ssh/config file
   • with various options in ~/.ssh/config, such as
     • hashKnownHosts
     • strictHostKeyChecking
   • with and without a known_hosts file
   • with a known_hosts file, with various types of entries in there

4. Xenon does not respect certain combinations of configuration parameters, specifically when inpuit argument knownHosts is false (see Notes xenon-tutorial#122), so by setting knownHosts to falseI can disobey input parameter stricHostCheck

5. my ssh understands strictHostKeyChecking=ask, xenon doesn't

6. in xenon, the config state is not assembled correctly from system config, user config, command line config

7. in xenon, you can't ask xenon what it thinks the current config state is, like you can with ssh -G xenon@localhost and sshd -T

8. in xenon, there is no way to control fallback behavior. ssh first tries public-private key, then interactive, then username-password (or something) this makes it more difficult to know that something is not working, and more difficult what is going wrong if you realize that something is going wrong

9. rethink options in SSHUtil

   xenon/src/main/java/nl/esciencecenter/xenon/adaptors/shared/ssh/SSHUtil.java

   Lines 187 to 188 in c24fba2

   	public static SshClient createSSHClient(boolean useKnownHosts, boolean loadSSHConfig, boolean stricHostCheck, boolean useSSHAgent,
   	boolean useAgentForwarding) {

   1. useAgentForwarding is not used and only in the way
   2. stricHostCheck typo plus should be strictHostKeyChecking
   3. stricHostCheck option is unused if I set useKnownHosts false
   4. loadSSHConfig ambiguous which SSHConfig you're talking about: system, user, commandline, or the combination of these

10. xenon-docker-images have multiple key pairs, would be better to have one per image, then you know how you got inside a container, or if it doesn't work, you know why (see https://github.com/xenon-middleware/xenon-docker-images/tree/07db6c0f12621e51bd109a5325fe08b1c0e9123f/slurm-ssh/.ssh)

11. docker images client side ssh has authorized_keys like https://github.com/xenon-middleware/xenon-docker-images/blob/a87f1ef876055e19cedcccf4bae4012db2953258/slurm-ssh/Dockerfile#L27, not sure that's needed if it is only client

12. xenon docker images (slurm:17 image): not sure what the value of PasswordAuthentication is):

    from ``/etc/ssh/sshd_config``
    # The strategy used for options in the default sshd_config shipped with
    # OpenSSH is to specify options with their default value where
    # possible, but leave them commented.  Uncommented options change a
    # default value.
    

    and

    #PasswordAuthentication no
    

    seems to suggest password authentication is disabled by default. Stands to reason, since it
    is not as secure as keypairs. However, if I do sshd -T, I see
    passwordauthentication yes. Not sure where that's coming from.

[jmaassen]

Regarding point 9. This is indeed the place where we need to figure out what to do and what not to do.

i. useAgentForwarding was supported in the past, but has not been implemented yet when we moved from JSch to SSHD. Some (advanced) ssh users use this to forward keys to other servers when using ssh to traverse multiple hops. Will be implemented again at some point. Lets ignore this for now.

ii. stricHostCheck will rename

iii. when not using known_hosts, a strict host check does not make sense since there is nothing to check against. The command lines ssh always checks known_hosts, but ignores any problems it encounters when -o StrictHostKeyChecking=no. We could mimic that behavour if we wanted to (see below).

iv. loadSSHConfig tells SSHD to use the default action to read the config(s). This may actually do nothing, as we just repeat the default? Maybe we could drop this.

We can probably clean this up quite a bit.

If I comment out this code completely and don't tell SSHD to do anything special it seems to accept all connections and not add anything to known_hosts. This is the same behavour as setting useKnownHost=false.

It we set useKnownHost=true and stricHostCheck=false, we get the same behaviour as -o StrictHostKeyChecking=no, which will not complain about new or different keys, but just add everything to known_hosts.

The alternative, useKnownHost=true and stricHostCheck=true is similar to -o StrictHostKeyChecking=yes. It will refuse to connect when a key has changed or an new server is
connected to. This is regarded as being annoying (it says so in the ssh documentation ;-) because it requires the user to properly maintain the known_host manually.

Unfortunately, setting it to ask is not really an option in a library like xenon, since we do not assume there is a user interactively doing stuff.

Additionally, the DefaultKnownHostsServerKeyVerifier of SSD which we use to implement this has a parameter boolean strict which tells it to be fussy about the access rights to the .ssh folder. Its currently set to true, which may explain the problems you had earlier.