git ssh credentials

[prl900]

Git commands over ssh have stopped working on my DEA Sandbox instance. For example, I can't clone a public github repo:

jovyan@jupyter-prl900:~/tmp$ git clone git@github.com:ANU-WALD/dea_course.git
Cloning into 'dea_course'...
git@github.com: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

I have tried adding a key under the .ssh directory but the system doesn't seem to like that:

$ git clone git@github.com:ANU-WALD/dea_course.git
Cloning into 'dea_course'...
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions 0660 for '/home/jovyan/.ssh/id_rsa' are too open.
It is required that your private key files are NOT accessible by others.
This private key will be ignored.
Load key "/home/jovyan/.ssh/id_rsa": bad permissions

I'm not sure if the problem is specific to my instance or there is a more generic problem affecting other instances. I'd be happy to reset my instance to the initial config and contents if that solves the problem.

Thanks,
Pablo

[benjimin]

I have experienced the same issue. (Tested on dev sandbox.)

Apparently every time the sandbox is restarted, all files under the home directory have their permissions changed, to be permissive of read/write by both owner and other members of the group. This includes all contents of the hidden .ssh folder. Git complains that it is unacceptable to use a private key that other group members can see.

You can fix this by deleting and re-generating your SSH key pair (and re-certifying your new public key to the remote host) each time. (Alternatively, it might be sufficient to use chmod to lock down the private key again after each restart, but this seems like a terribly questionable security practice.)

I'm not sure whether or not this is a true security hole (noting that all sandbox users seem to be presented the same unix username on sandbox, so the unix file permissions are kind of irrelevant anyway, but possibly your own home directory gets mounted from somewhere that many other ordinary users actually cannot easily access for other reasons, I'm not too sure). I think it would help if the sandbox did not change the permissions of users files, and if there was some security advice provided on the splash page.

[benjimin]

Likely to affect more users since GitHub deprecated password authentication (because retyping full account passwords into terminals is not considered best practice for security) as of mid Nov 2020.

This issue seems likely intertwined to #79 (specific practices reducing isolation between private data and public history).

[benjimin]

I initially assumed that our custom startup scrips were responsible (deliberately resetting permissions to prevent git sync failure), but, that ~/.no_sync does not avoid the issue suggests git-clone and rsync might not be relevant.

An alternate possibility could be a docker misconfiguration (e.g. mismatched UID or GID between inside and outside of the container) semi-related to #57 (GID 40000 not in /etc/group). Particularly note the gid=0.

$ id jovyan
uid=1000(jovyan) gid=100(users) groups=100(users)
$ id
uid=1000(jovyan) gid=0(root) groups=0(root),100(users),40000
$ grep 1000 /etc/passwd
jovyan:x:1000:100::/home/jovyan:/bin/bash
$ groups jovyan
jovyan : users
$ groups
root users 40000
groups: cannot find name for group ID 40000
$ egrep ':0|100|40000' /etc/group
root:x:0:
users:x:100:

Another observation, by running stat * | grep Change, is that every file (excluding those outside of the home directory mount) reports a metadata change occurring during the server startup but not all instantaneously. The slightly different timestamps suggest that some startup process traverses the home directory, updating one inode at a time, granting user and group readwrite permission while simultaneously changing back the group designation (from root or 40000) to users.

These timestamps could be used to narrow down which process is responsible? (Something after AttachVolume.Attach succeeded and before Created container notebook..)

[benjimin]

Actually this seems to be documented behaviour (or known problem) of kubernetes. When preparing to mount a volume for a pod (e.g. before starting the containers), kubernetes recursively changes the ownership and permissions inside that mount. We should probably try explicitly setting:

securityContext:
  runAsUser: 1000
  runAsGroup: 100
  fsGroup: 100
  fsGroupChangePolicy: "OnRootMismatch"

(Although it may also require --feature-gates="ConfigurableFSGroupPolicy=true" if we haven't yet upgraded to version 1.20)

The change policy should prevent permissions being changed on successive logins.