Document k8s cluster prerequisites

[consideRatio]

We have got a lot of questions about issues associated with k8s clusters that does not meet the prerequisites for using the helm chart. I see two main aspects to verify for anyone using a bare metal cluster for example.

1. Kubernetes StorageClass, is there a default one in the cluster, and can it be used to provision a PV to bind to a PVC?

   • ref: PVC stays pending indefinitely #1413
   • ref: hub pod stuck in pending #1315

2. Kubernetes Service's spec.type: LoadBalancer, can they get an ExternalIP?
   MetalLB describes the issue well in their initial documentation under the title Why?.

   NAME           TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
   proxy-public   LoadBalancer   100.70.36.147   <pending>     80:31165/TCP   5h
   

   • ref: proxy-public service stalled/pending #641 (comment)
   • ref: Proxy-public external ip is pending  #880
   • ref: hub pod stuck in pending #1315 (again)

I think it would make sense and be plausible to have a small section on verifying the prerequisites of the k8s cluster, or even better, link to somewhere where this is done already.

[clkao]

I wonder if the proxy-public's default type should be LoadBalancer since it's not always available. also when ingress is configured, you might end up with a public ip from ingress, and another public ip from LoadBalancer svc that is not actually used.

[alcfftools]

Working on a bare metal k8s installation, and so no LoadBalancer solution for services by default.

What are the recommended options?

• Is there a way to run jupyterhub without the LoadBalancer proxypublic? Can I stay at the ingress level to grant access to my jupyterhub? If so, any guideline would be really appreciated!
• I've no experience with MetaLB, but it seems it might be the solution to bring LoadBalancer capabilities to the cluster.

Any experience/hints/documentation on where tow go from here?

[alcfftools]

Working on a bare metal k8s installation, and so no LoadBalancer solution for services by default.

What are the recommended options?

• Is there a way to run jupyterhub without the LoadBalancer proxypublic? Can I stay at the ingress level to grant access to my jupyterhub? If so, any guideline would be really appreciated!
• I've no experience with MetaLB, but it seems it might be the solution to bring LoadBalancer capabilities to the cluster.

Any experience/hints/documentation on where tow go from here?

I made it work with ingress as suggested here: https://zero-to-jupyterhub.readthedocs.io/en/latest/administrator/advanced.html#ingress

Maybe it would be worth it to document this solution in your initial post?

Thanks!

[manics]

@alcfftools thanks for following up, as you found out a Kubernetes ingress works well. If you have any suggestions on how to make this clearer please let us know, or open a PR against the docs 😄

[alcfftools]

In fact, I am able to access now through ingress the jupyterhub via http, but the websockets connection is rejected (http 503), and so the kernel is not able to connect... So I don't have still a fully working solution...

Any hint?

[manics]

It's most likely a configuration issue with your ingress controller. I've used nginx ingress successfully
https://kubernetes.github.io/ingress-nginx/
But other implementations may not have websocket proxying enabled by default, so this is something to add to the docs. It's also possible you've got an institutional firewall or web proxy that blocks websockets or other http requests which could be an issue.

[alcfftools]

Actually I have configured proxy-public as a NodePort, and directly accessing to that (no ingress in between) the websocket connection still fails.

Therefore I think it shall not be connected to ingress, right?

[manics]

Yes, so perhaps it's an institutional proxy that restricts websocket connections, or local restrictions on your machine enforced by your institutions IT department? Could you investigate further? If it's not a simple answer could you please start a discussion on the Jupyter Community forum? https://discourse.jupyter.org

This is so that we can keep this issue focused on the docs. It also means any information you discover that's not directly related to this issue will still be findable and accessible on the forum to others, since when this issue is closed the discussions will be less visible. Thanks!