Refactor existing_arrikto into a community supported config

[yanniszark]

There seems to be a community agreement that Kubeflow needs a config in the lines of
existing_arrikto.
Since our intention was for it to be a community config from the start, we refactor it into a community-supported config.

The proposed new name is kfctl_istio_dex.yaml.

What is included:

• Kubeflow components: the intention is for this config to be a batteries-included config, with all the Kubeflow components (Notebooks, Pipelines, TFJobs, etc.).
• Service Mesh: Istio is used and installed by default.
• Auth: OIDC with Dex as the OIDC Provider and the oidc-authservice as the OIDC Client.
• Exposing Kubeflow: Kubeflow is exposed on HTTPS, with a LoadBalancer with a self-signed cert by default.
• Restrictions: Must not depend on any cloud-specific feature. This is targeted at existing, on-prem clusters with no access to cloud-specific services.

/cc @krishnadurai @ddutta @jlewi @johnugeorge @vkoukis @cvenets

[krishnadurai]

Thanks, @yanniszark for filing this issue. Let me link @jlewi's comment which summarizes this:

Currently kfctl_k8s_istio.yaml is a configuration that works for everyone but is not identity aware and doesn't have a good story for how you access it from outside the cluster (kubectl port-forward works but isn't great).

Is the proposal to change that to

• Exposing the istio ingress gateway on an IP address (which could be either internal or public depending on the "Cloud" Provider's implementation of service type loadbalancer and possibly additional service annotations)
• Always including DEX for auth
  • Defaulting to basic auth with username and password set by environment variables
• Using a self sign certificate to do https termination

Adding advanced options to

• Use LetsEncrypt to generate a signed certificate and/or upload a signed certificate
  • I don't think we want to use LetsEncrypt by default because it requires an external DNS name and public ip
  • LetsEncrypt ACME challenge is also slow
• Specify a DOMAIN that would be registered in some external registrar
  • I don't think we want to require this by default because that requires users to perform another manual step in a cloud/infra dependent way

Originally posted by @jlewi in #560 (comment)

[krishnadurai]

It follows from here that we would stop supporting kfctl_k8s_istio.yaml and replace it with kfctl_k8s_istio_dex.yaml

[jlewi]

Thanks @yanniszark; I'd like to propose one additional

• Basic Auth use Dex basic auth by default

  • kfctl should look for environment variables KUBEFLOW_USERNAME and KUBEFLOW_PASSWORD and use them to configure basic auth

  • This is what GCP basic auth does today so we could refactor that code into a generic kfctl plugin

The motivation is as follows

• On cloud providers a Service of type load balancer creates a public IP address so we need some form of Auth
• Using an OIDC provider (e.g. GitHub, Google, Microsoft etc...) requires the user to create an OAuthClient with the identity provider
• I think its better if to get started we don't have dependencies on external systems

[krishnadurai]

One important assumption we need to note is that with 0.7 release the identity at Kubeflow gateway and Kubernetes' API are assumed to be the same for a user.
The reference architecture needs to support the same Identity at both ends.

[krishnadurai]

We've thought of introducing Istio 1.3.1 with this config.
Add E2E tests for Istio 1.3.1: kubeflow/kubeflow#4480
Upgrade kfctl_istio_dex kfdef to use Istio 1.3.1 and update tests: kubeflow/kubeflow#4501

[jlewi]

@krishnadurai and @yanniszark any update on this ? What is the remaining work that needs to be done for 1.0?

[krishnadurai]

@jlewi Here's the status:

• Dex Plugin implementation needs to be updated with Unit Tests. [WIP] Dex Plugin kfctl#130

• Manifests for issuing certificates for 'DOMAIN' or LoadBalancer IP needs to be implemented.

• I've raised a PR in Dex to accept environment variables for StaticPasswords. This allows our Dex Plugin to programmatically set secrets for Dex deployment to consume. This is already approved by the Dex community.
  Option to add staticPasswords from environment variables dexidp/dex#1601

Overall, I'm looking to complete the implementation by sometime next week.