Add information about using NFS with z2jh

[yuvipanda]

NFS is still a very popular storage setup, and is a good fit for use with z2jh in several cases:

1. When you are supporting a large number of users
2. When you are running on baremetal and NFS is your only option
3. When your utilization % (% of total users active at any time) is very low, causing you to spend more on storage than compute.

While we don't want to be on the hook for teaching users to setup and maintain NFS servers, we should document how to use an NFS serer that already exists.

[cam72cam]

@yuvipanda I'd like to be a guinea pig on this. I am trying to setup a persistent EFS volume and use that as the user storage.

So far I've created and applied:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: efs-persist
spec:
  capacity:
    storage: 123Gi
  accessModes:
    - ReadWriteMany
  nfs:
    server: <fs-id>.efs.us-east-1.amazonaws.com
    path: "/"

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: efs-persist
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 11Gi

After that I added the following to my config.yaml:

singleuser:
  storage:
    static:
      pvc-name: efs-persist

I am pretty sure I am missing a few key ideas here

Edit:
first change was to add "type: static" to the storage section in config
changed pvc-name to pvcName

[yuvipanda]

w00t, thanks for volunteering :)

The other two things to keep in mind are:

1. subPath (https://github.com/jupyterhub/zero-to-jupyterhub-k8s/blob/master/jupyterhub/values.yaml#L138). This specifies where inside the share the user's homedirectory should go. Although it defaults to just {username}, I would recommend something like home/{username}
2. User permissions. This can be a little tricky, since IIRC when kubelet creates a directory for subPath mounting it makes it uid 0 / gid 0, which is problematic for our users (with uid 1000 by default). The way we've worked around it right now is by using anongid / anonuid properties in our NFS share, but that's not a good long term solution. I've been working on http://github.com/yuvipanda/nfs-flex-volume as another option here. Is anongid / anonuid an option with EFS?

[cam72cam]

1. I saw that, thanks for the clearer explanation
2. I don't think that anonuid/gid is an option on EFS

I'll take a look through the nfs-flex-volume repo

[yuvipanda]

@cam72cam another way to check that everything works right now except for permissions is to set:

singleuser:
  uid: 0
  fsGid: 0
  cmd: 
    - jupyterhub-singleuser
    - --allow-root

If you can launch servers with that, then we can confirm that the uid situation is the only problem.

[cam72cam]

I am currently getting the following response:

 HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Pod in version \"v1\" cannot be handled as a Pod: v1.Pod: Spec: v1.PodSpec: Containers: []v1.Container: v1.Container: VolumeMounts: []v1.VolumeMount: v1.VolumeMount: SubPath: ReadString: expects \" or n, parsing 184 ...ubPath\": {... at {\"kind\": \"Pod\", \"spec\": {\"containers\": [{\"imagePullPolicy\": \"IfNotPresent\", \"lifecycle\": {}, \"ports\": [{\"containerPort\": 8888, \"name\": \"notebook-port\"}], \"volumeMounts\": [{\"subPath\": {\"username\": null}, \"name\": \"home\", \"mountPath\": \"/home/jovyan\"}, {\"readOnly\": true, \"name\": \"no-api-access-please\", \"mountPath\": \"/var/run/secrets/kubernetes.io/serviceaccount\"}], \"env\": [{\"name\": \"JUPYTERHUB_HOST\", \"value\": \"\"}, {\"name\": \"JUPYTERHUB_CLIENT_ID\", \"value\": \"user-efs4\"}, {\"name\": \"JUPYTERHUB_API_TOKEN\", \"value\": \"355dc09aca1143f580ee0435339cc18d\"}, {\"name\": \"JUPYTERHUB_USER\", \"value\": \"efs4\"}, {\"name\": \"EMAIL\", \"value\": \"efs4@local\"}, {\"name\": \"GIT_AUTHOR_NAME\", \"value\": \"efs4\"}, {\"name\": \"JUPYTERHUB_ADMIN_ACCESS\", \"value\": \"1\"}, {\"name\": \"JUPYTERHUB_SERVICE_PREFIX\", \"value\": \"/user/efs4/\"}, {\"name\": \"JPY_API_TOKEN\", \"value\": \"355dc09aca1143f580ee0435339cc18d\"}, {\"name\": \"JUPYTERHUB_API_URL\", \"value\": \"http://100.65.96.26:8081/hub/api\"}, {\"name\": \"JUPYTERHUB_BASE_URL\", \"value\": \"/\"}, {\"name\": \"JUPYTERHUB_OAUTH_CALLBACK_URL\", \"value\": \"/user/efs4/oauth_callback\"}, {\"name\": \"GIT_COMMITTER_NAME\", \"value\": \"efs4\"}, {\"name\": \"MEM_GUARANTEE\", \"value\": \"1073741824\"}], \"image\": \"jupyterhub/k8s-singleuser-sample:v0.5.0\", \"resources\": {\"limits\": {}, \"requests\": {\"memory\": 1073741824}}, \"args\": [\"jupyterhub-singleuser\", \"--ip=\\\"0.0.0.0\\\"\", \"--port=8888\"], \"name\": \"notebook\"}], \"securityContext\": {\"runAsUser\": 1000, \"fsGroup\": 1000}, \"volumes\": [{\"persistentVolumeClaim\": {\"claimName\": \"efs-persist\"}, \"name\": \"home\"}, {\"emptyDir\": {}, \"name\": \"no-api-access-please\"}], \"initContainers\": []}, \"metadata\": {\"labels\": {\"hub.jupyter.org/username\": \"efs4\", \"heritage\": \"jupyterhub\", \"component\": \"singleuser-server\", \"app\": \"jupyterhub\"}, \"name\": \"jupyter-efs4\"}, \"apiVersion\": \"v1\"}","reason":"BadRequest","code":400}

I suspect it has to do with:

              'volumes': [{'name': 'home',
                           'persistentVolumeClaim': {'claimName': 'efs-persist'}},
                          {'aws_elastic_block_store': None,
                           'azure_disk': None,
                           'azure_file': None,
                           'cephfs': None,
                           'cinder': None,
                           'config_map': None,
                           'downward_api': None,
                           'empty_dir': {},
                           'fc': None,
                           'flex_volume': None,
                           'flocker': None,
                           'gce_persistent_disk': None,
                           'git_repo': None,
                           'glusterfs': None,
                           'host_path': None,
                           'iscsi': None,
                           'name': 'no-api-access-please',
                           'nfs': None,
                           'persistent_volume_claim': None,
                           'photon_persistent_disk': None,
                           'portworx_volume': None,
                           'projected': None,
                           'quobyte': None,
                           'rbd': None,
                           'scale_io': None,
                           'secret': None,
                           'storageos': None,
                           'vsphere_volume': None}]},

It should have the nfs option set there if I understand correctly

Actually:

'volume_mounts': [{'mountPath': '/home/jovyan',
                                                 'name': 'home',
                                                 'subPath': {'username': None}},
                                                {'mount_path': '/var/run/secrets/kubernetes.io/serviceaccount',
                                                 'name': 'no-api-access-please',
                                                 'read_only': True,
                                                 'sub_path': None}],

it appears that subpath is not being set correctly

[yuvipanda]

So it turns out specifying subPath manually of home/{username} was required, so we should investigate why.

[yuvipanda]

The PVC needs to be in the same namespace as JupyterHub, so the pods can find it.

[yuvipanda]

The PVC needs to be told how to find the PV to match, and this is done by using:

1. Labels to match PVC and PV
2. Setting storageclass of PVC to '' (so kubernetes does not try to create a PV for it)

So

apiVersion: v1
kind: PersistentVolume
metadata:
  name: efs-persist
  labels:
    <some-label1-key>: <some-label1-value>
    <some-label2-key>: <some-label2-value>
spec:
  capacity:
    storage: 123Gi
  accessModes:
    - ReadWriteMany
  nfs:
    server: <fs-id>.efs.us-east-1.amazonaws.com
    path: "/"
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: efs-persist
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: ""
  selector:
    matchLabels:
       <some-label1-key>: <some-label1-value>
       <some-label2-key>: <some-label2-value>
  resources:
    requests:
      storage: 11Gi

Things to note:

1. Specify labels that uniquely identify this PV in your entire k8s cluster.
2. The size requests bits are ignored both in the PV and PVC for EFS specifically, since it grows as you use it.

[cam72cam]

Ok, I've got the mount working. I did not do the label stuff yet, simply set 'storageClassName: ""' in the claim. That seemed to work just fine.

I ran into a speed bump where I had to change the security groups to allow access from EC2 to EFS. As a temporary measure I added both the EFS volume and the EC2 instances to the "default" security group. Eventually part of the initial kops config should add the correct security groups.

I am now getting a permission error:
PermissionError: [Errno 13] Permission denied: '/home/jovyan/.jupyter'

I am going to try to change the permissions on the EFS drive first, and if that does not work try the root hack that @yuvipanda mentioned

EDIT: A manual chown on the EFS volume to 1000:1000 seems to have worked!

[cam72cam]

EFS Success!

Process:

Setup an EFS volume. It must be in the same VPC as your cluster. This can be changed in the AWS settings after it has been created.
The EFS volume will be created in the default security group in the VPC. As a temporary hack around, add your cluster master and nodes to the default VPC group so they can access the EFS volume. Eventually we will setup proper security groups as part of this process.

Created test_efs.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: efs-persist
spec:
  capacity:
    storage: 123Gi
  accessModes:
    - ReadWriteMany
  nfs:
    server: fs-$EFS_ID.efs.us-east-1.amazonaws.com
    path: "/"

Created test_efs_claim.yaml

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: efs-persist
spec:
  storageClassName: ""
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 11Gi

kubectl --namespace=cmesh-test apply -f test_efs.yaml
kubectl --namespace=cmesh-test apply -f test_efs_claim.yaml

The sizes in these files don't mean what you think. There is no quota enforced with EFS **. In the future we want to set the efs PersistentVolume size to something ridiculously large like 8Ei and the PersistentVolumeClaim to 10GB (neither matters AFAICT). This is my rough understanding and could be incorrect.

A PersistentVolume defines a service which can perform a mount inside of a container. The PersistentVolumeClaim is a way of reserving a portion of the PersistentVolume and potentially locking access to it.

The storageClassName setting looks innocuous, but it is incredibly critical. The only non storage class PV in the cluster is the one we defined above. In the future we should tag different PV's and use tag filters in the PVC instead of relying on a default of "".

We are going to configure jupyterhub to use the same "static" claim among all of the containers*** . This means that all of our users will be using the same EFS share which should be able to scale as high as we need.

We now add the following to config.yaml

singleuser:
  storage:
    type: "static"
    static:
      pvcName: "efs-persist"
      subPath: 'home/{username}'

type static tells jh not to use a storage class and instead use a PVC defined below.
pvcName matches the claim name we specified before
subPath tells where on the supplied storage the mount point should be. In this case it will be "$EFS_ROOT/home/{username}"

It turns out there is a bug in jupyterhub where the default subPath does not work, and setting the subPath to "{username}" breaks in the same way.

At this point if we tried to start our cluster, it would fail. The directory created on the mount at subPath will be created with uid:0 and gid:0. This means that when jupyter hub is launched it won't be able to create any files, will complain, then self destruct.

What we need to do is tell the container to run our jupyterhub setup as root, then switch to the Jovian user before starting the jupyterhub process. When we are running as root we can do our own chown to adjust the created directory permissions.

First we merge the following to our config.yaml

singleuser:
  uid: 0
  fsGid: 0
  cmd: "start-singleuser.sh"

This tells jupyterhub to enter the container as root and run the start-singleuser script. Start-singleuser calls a helper start.sh script which we will use later on.

This will get jupyter hub to provision the container and attempt to start, but the process will still fail as the chmod has not taken place.

In order for us to have a properly chowned directory in /home/Jovian mounted from $EFS_ROOT/home/{username}, we need to create our own docker container***.

Here are some terse steps:
Create a docker account
Create a docker repo
Create a directory to store the build file
Create Dockerfile inside that directory

FROM jupyter/base-notebook:281505737f8a

# pin jupyterhub to match the Hub version
# set via --build-arg in Makefile
ARG JUPYTERHUB_VERSION=0.8
RUN pip install --no-cache jupyterhub==$JUPYTERHUB_VERSION

USER root
RUN sed -i /usr/local/bin/start.sh -e 's,# Handle username change,chown 1000:1000 /home/$NB_USER \n # Handle username change,'
RUN cat /usr/local/bin/start.sh
USER $NB_USER

The base dockerfile came from 967b2d2#diff-aed8b29ee8beb1247469956c481040c2
Notice that we are using the older revision. The newer revision is broken in some awesome way that Yuvi needs to fix.
This script is fragile and should be done better in the future...
The first parts of the docker setup are done as $NB_USER
The rest of the file is done as ROOT since we need to modify the start.sh script which will be run as root when the container is started.
Many of the files referenced can be found in https://github.com/jupyter/docker-stacks/tree/master/base-notebook
sudo yum install docker
sudo docker login
sudo docker build ${directory_containing_dockerfile}
sudo docker tag ${image_id_in_cmd_output} $docker_username/$docker_repo # can also be found by sudo docker images
sudo docker push $docker_username/$docker_repo

Merge the following into config.yaml

singleuser:
  image:
   name: $docker_username/$docker_repo
    tag: latest

You may be able to do a helm upgrade, but I ended up purging and reinstalling via helm just to be safe.

At this point you should be all set with a semi-fragile (but functional) EFS backed jupyterhub setup

Debugging tools: (all with --namespace=)

• kubectl get pods # list pods
• kubectl logs $podname # get a pod log, may not be anything if the crash happens soon enough
• kubectl describe pod $podname # dumps a bunch of useful info about the pod
• kubectl get pod $podname -o yaml # dumps the args used to create the pod. This stuff is the container creation stuff

** fuse layer for fs quota
*** We may run into issues with a hundred containers all hitting the same EFS volume.  I suspect that AWS can more than handle that, but I have been wrong before.  If it can't handle that Yuvi has a WIP nfs server sharding system partially built that we could use.
**** I hope that the changes I made to the base container will be adopted by the project as it seems relatively harmless to have in the start script.  Even if it is harmful to others, I would still like it in there as a config option (if possible).

[yuvipanda]

Thank you for getting this working, @cam72cam!

To summarize, this mostly works, except for the issue of permissions:

1. When using subPath, Kubernetes creates this directory when it doesn't exist if it needs to
2. However, this will always be created as root:root
3. Since we want our users to run as non-root, this won't work for us and we have to use hacks to do chowns.

It'll be great if we can fix EFS or Kubernetes to have options around 'what user / group / mode should this directory be created as?'

[cam72cam]

Could we add the chown hack I put in my image to the start.sh script in the stacks repo?
https://github.com/jupyter/docker-stacks/blob/master/base-notebook/start.sh#L19

Would that break any existing users setups?