Inject PyPI information (URL, username, credentials) automatically into the workbench

[jiridanek]

Checks

• I have searched the existing issues.
• My request is related to one of the components in the kubeflow/notebooks repository.

Motivation

We're running internal PyPI repositories and we need to have the ability to inject these repo information (URL, username, credentials) automatically in the workbench environment so that pip installing from them works out of the box.

This was previously filled as

• Some notebooks need configuration files to access private servers. kubeflow#5210

It appears to largely duplicate

• Allow mounting Secrets on Workspaces #239

Also somewhat related issues

• Support for installing libraries from GCP artifact registry or enterprise artifactory! kubeflow#7332
• Consider including a scanned PyPI package mirror component kubeflow#313

Implementation

This opens up the auth inside a workspace can of worms.

Are you willing & able to help?

• I am able to submit a PR!
• I can help test the feature!

[thesuperzapper]

This seems like it will be related to mounting secrets as proposed in #239

I am not sure about having a tight integration with PyPi specifically, it's probably easier to simply have the user mount their token from a Secret, and read it using their code in whatever way is necessary for their environment.

---

Alternatively, we discussed the idea of having "required" secrets for a specific imageConfig option (specified in the WorkspaceKind under an imageConfig's option spec).

These "required secrets" would prompt the user to choose a secret name/key when creating/updating a Workspace with that imageConfig option. The image builder can then assume its there is a secret file mounted at a specified location, and use it accordingly. This would be a generic solution for admins, and also solve other "out of the box" auth situations, rather than just PyPi.

PS: we could also do "required volumes", if we can think of a use-case for that.

[jiridanek]

For a PyPI repo url, I can use PIP_INDEX_URL env variable.

Being able to mount a configmap (not only secrets) may come useful when the config I'm mounting does not contain anything secret. But secrets should be usable too, ofc.

Adding what needs to be mounted into WorkspaceKind should take care of the out-of-the-box part of the requirement.