jupyter web app is giving me 404s when hitting the backend.

[jlewi]

/kind bug

What steps did you take and what happened:

The jupyter web app is giving me 404s when hitting the backend. Specifically, the URLs
https://${DOMAIN}/api/namespaces
https://${DOMAIN}/dashboard_lib.bundle.js

Give me 404s. I think this is a problem with the jupyter web app front end not properly working behind a reverse proxy. The jupyter web app is running behind the URL
https://${DOMAIN}/jupyter/ and it looks like the correct URLs should be
https://${DOMAIN}/jupyter/api/namespaces
https://${DOMAIN}/jupyter/dashboard_lib.bundle.js

Other requests are correctly using the reverse proxy e.g.
https://${DOMAIN}/jupyter/api/namespaces/kubeflow-user/notebooks

What did you expect to happen:
Expected it to work behind the reverse proxy.

Anything else you would like to add:
I'm setting the environment variable "APP_PREFIX" to "/jupyter/"

Environment:

• Kubeflow version: (version number can be found at the bottom left corner of the Kubeflow dashboard):
  Not quite sure. I'm not deploying the dashboard. My manifests were using "docker.io/kubeflownotebookswg/jupyter-web-app:latest". But checking the logs and it looks like at least one of the images pulled had sha
  sha256:55e02d75bf2921ec61b4bd3fc728f299e37051fe0340a166e936fecabee8fb0f

• kfctl version: (use kfctl version):

• Kubernetes platform: (e.g. minikube)

• Kubernetes version: (use kubectl version):

• OS (e.g. from /etc/os-release):

[kimwnasptd]

@jlewi thanks for the detailed description. I'm almost sure that, as you mentioned, the frontend is not properly working behind a proxy. And most probably because it's using hardcoded paths in one place.

First of, the https://${DOMAIN}/dashboard_lib.bundle.js request path is expected. The JWA will try to fetch the common library of the CentralDashboard, which is used for passing the selected namespace information from the dashboard to the JWA. This library is not served by the JWA, but by the CentralDashboard. You can find more details in kubeflow/dashboard#41.

The TL;DR is this:

1. the apps will try to fetch this library
2. If the library exists (and is loaded correctly in the app's window) then the apps will detect and use that for getting namespace information
3. Else they'll show a custom namespace selector.

So the dashboard request is OK. The problematic one though is the https://${DOMAIN}/api/namespaces which should indeed be https://${DOMAIN}/jupyter/api/namespaces

[kimwnasptd]

By taking a more detailed look I'm almost sure the bug is in the frontend logic that we have which decides when to show a namespace selector element, in case the CentralDashboard library was not found.

Specifically in this line:
https://github.com/kubeflow/kubeflow/blob/a6b53f02890ab248f62cdfe3c8008a3ed49803a8/components/crud-web-apps/jupyter/frontend/src/app/pages/index/index-default/index-default.component.html#L5

That part is where we conditionally check if the dashboard did not get connected and if not which route path to use for fetching the namespaces. But this path should actually be api/namespaces, instead of /api/namespaces.

@jlewi could you try to update this line and rebuild the JWA docker image to verify if this is indeed the case?

[kimwnasptd]

And let me expose a last bit of info, regarding how the apps are expected to work behind a proxy (for future reference as well):

1. The frontend should always use relative paths whenever it needs to make a request
2. The index.html file should have a <base href="<prefix>"> element https://developer.mozilla.org/en-US/docs/Web/HTML/Element/base
3. The backends will be setting this element in the index.html files when serving them (APP_PREFIX env var) https://github.com/kubeflow/kubeflow/blob/master/components/crud-web-apps/common/backend/kubeflow/kubeflow/crud_backend/serving.py#L26

Note though that the backend routes always work under /. So the proxy will need to remove the prefix when forwarding the requests to the backend. In the KF mode this is done by the VirtualService (Istio ingressgateway is our proxy in this case)
https://github.com/kubeflow/kubeflow/blob/master/components/crud-web-apps/jupyter/manifests/overlays/istio/virtual-service.yaml#L18

[jlewi]

Thanks @kimwnasptd; I'm busy with other stuff at the moment but I'll give that a try when I get back to Kubeflow.

[andreyvelich]

/transfer notebooks

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity.
It will be closed if no further activity occurs.
Thank you for your contributions.

Members may comment /lifecycle frozen to prevent this issue from being marked as stale.