Skip to content

README.md

Latest commit

 

History

History
168 lines (134 loc) · 13 KB

README.md

File metadata and controls

168 lines (134 loc) · 13 KB

Kubeflow Authentication using Oauth2 Proxy

Istio envoyExtAuthzHttp

This is Istio's recommended approach for External Authorization1. It is not limited to the use of oauth2-proxy2 alone. This method is an industry standard, meeting all of Kubeflow's current and foreseeable authentication needs.

Kubeflow Pipelines User and M2M Authentication and Authorization

The Kubeflow Pipelines component relies on the built-in kubernetes functionalities to authenticate and authorize user requests, specifically the TokenReviews3 and SubjectAccessReview4.

The best way to describe how it works is to explain with an example. Lets analyze the flow when a client calls the API to list the KF Pipeline runs:

  1. api-server starts endpoints in:

    https://github.com/kubeflow/pipelines/blob/2.0.5/backend/src/apiserver/main.go#L95

    Focusing on the pipelines run service:

    1. Register Run Service:
    2. proto RPC definition of ListRunsV1
    3. code definition of ListRunsV1
    4. ListRunsV1 calls internal method listRuns
    5. listRuns calls internal method canAccessRun which itself calls s.resourceManager.IsAuthorized
    6. ResourceManager.IsAuthorized first tries to authenticate over every available authenticator, which are the TokenReviewAuthenticator and HTTPHeaderAuthenticator
    7. TokenReviewAuthenticator.GetUserIdentity gets the token from Authorization header and calls the K8s Auth authv1.TokenReview with given token which in return provides userInfo := review.Status.User. GetUserIdentity return userInfo.Username which at this point is the system:serviceaccount:default:default.
    8. Next in ResourceManager.IsAuthorized a SubjectAccessReview is created with r.subjectAccessReviewClient.Create with arguments specifying RBAC verbs provided in code definition of RunServer.listRuns. If the user (sa) is not authorized, an error is thrown

  2. User calls api to list pipeline runs as unauthorized service account.

    • This can be done by running Pod with curl in default namespace: 
      $ kubectl -n default run -ti --rm curl --image curlimages/curl --command -- sh
# v1beta1
~ $ curl "istio-ingressgateway.istio-system/pipeline/apis/v1beta1/runs?resource_reference_key.type=NAMESPACE&resource_reference_key.id=kubeflow-user-example-com" -H "Authorization: Bearer $(cat /run/secrets/kubernetes.io/serviceaccount/token)"
{"error":"Failed to list v1beta1 runs: Failed to list runs due to authorization error. Check if you have permission to access namespace kubeflow-user-example-com: Failed to access run . Check if you have access to namespace kubeflow-user-example-com: PermissionDenied: User 'system:serviceaccount:default:default' is not authorized with reason:  (request: \u0026ResourceAttributes{Namespace:kubeflow-user-example-com,Verb:list,Group:pipelines.kubeflow.org,Version:v1beta1,Resource:runs,Subresource:,Name:,}): Unauthorized access","code":7,"message":"Failed to list v1beta1 runs: Failed to list runs due to authorization error. Check if you have permission to access namespace kubeflow-user-example-com: Failed to access run . Check if you have access to namespace kubeflow-user-example-com: PermissionDenied: User 'system:serviceaccount:default:default' is not authorized with reason:  (request: \u0026ResourceAttributes{Namespace:kubeflow-user-example-com,Verb:list,Group:pipelines.kubeflow.org,Version:v1beta1,Resource:runs,Subresource:,Name:,}): Unauthorized access","details":[{"@type":"type.googleapis.com/google.rpc.Status","code":7,"message":"User 'system:serviceaccount:default:default' is not authorized with reason:  (request: \u0026ResourceAttributes{Namespace:kubeflow-user-example-com,Verb:list,Group:pipelines.kubeflow.org,Version:v1beta1,Resource:runs,Subresource:,Name:,})"}]}
# v2beta1
~ $ curl istio-ingressgateway.istio-system/pipeline/apis/v2beta1/runs?namespace=kubeflow-user-example-com -H "Authorization: Bearer $(cat /run/secrets/kubernetes.io/serviceaccount/token)"
{"error":"Failed to list runs: Failed to list runs due to authorization error. Check if you have permission to access namespace kubeflow-user-example-com: Failed to access run . Check if you have access to namespace kubeflow-user-example-com: PermissionDenied: User 'system:serviceaccount:default:default' is not authorized with reason:  (request: \u0026ResourceAttributes{Namespace:kubeflow-user-example-com,Verb:list,Group:pipelines.kubeflow.org,Version:v1beta1,Resource:runs,Subresource:,Name:,}): Unauthorized access","code":7,"message":"Failed to list runs: Failed to list runs due to authorization error. Check if you have permission to access namespace kubeflow-user-example-com: Failed to access run . Check if you have access to namespace kubeflow-user-example-com: PermissionDenied: User 'system:serviceaccount:default:default' is not authorized with reason:  (request: \u0026ResourceAttributes{Namespace:kubeflow-user-example-com,Verb:list,Group:pipelines.kubeflow.org,Version:v1beta1,Resource:runs,Subresource:,Name:,}): Unauthorized access","details":[{"@type":"type.googleapis.com/google.rpc.Status","code":7,"message":"User 'system:serviceaccount:default:default' is not authorized with reason:  (request: \u0026ResourceAttributes{Namespace:kubeflow-user-example-com,Verb:list,Group:pipelines.kubeflow.org,Version:v1beta1,Resource:runs,Subresource:,Name:,})"}]}

  3. User calls api to list pipeline runs as authorized service account.

    • This can be done by running Pod with curl in kubeflow-user-example-com namespace specifying correct service account: 
      $ kubectl -n kubeflow-user-example-com run -ti --rm curl --image curlimages/curl --command --overrides='{"spec": {"serviceAccountName": "default-editor"}}' -- sh
# v1beta1
~ $ curl "istio-ingressgateway.istio-system/pipeline/apis/v1beta1/runs?resource_reference_key.type=NAMESPACE&resource_reference_key.id=kubeflow-user-example-com" -H "Authorization: Bearer $(cat /run/secrets/kubernetes.io/serviceaccount/token)"
{}  # empty response which is fine because no pipeline runs exist
# v2beta1
~ $ curl istio-ingressgateway.istio-system/pipeline/apis/v2beta1/runs?namespace=kubeflow-user-example-com -H "Authorization: Bearer $(cat /run/secrets/kubernetes.io/serviceaccount/token)"
{}  # empty response which is fine because no pipeline runs exist

Authentication and Authorization analysis diagram for Kubeflow Pipelines

Kubeflow Auth Diagram

Change the default authentication from "Dex + Oauth2-proxy" to "Oauth2-proxy" only

The authentication in Kubeflow evolved over time and we dropped envoyfilters and oidc-authservice in favor of RequestAuthentication and Oauth2-proxy in Kubeflow 1.9. auth-flow

You can adjust OAuth2 Proxy to directly connect to your own IDP(Identity Provider) suchg as GCP, AWS, Azure etc:

  1. Create an application on your IdP (purple line)
  2. Change your OAuth2 Proxy issuer to your IdP. Of course never ever directly, but with kustomize overlays and components.
  3. In the istio-system namespace is a RequestAuthentication resource. You need to change its issuer to your own IdP, or even better create an additional one.
  4. You can now directly issue a token from your IdP and use this token to access your Kubeflow platform.

This feature is useful when you need to integrate kubeflow with you current CI/CD platform (GitHub Actions, Jenkins) via machine-to-machine authentication.

Example for obtaining and using a JWT token From your IDP:

import requests
token_url = "https://your-idp.com/oauth/token"
client_id = "YOUR_CLIENT_ID"
client_secret = "YOUR_CLIENT_SECRET"
username = "YOUR_USERNAME"
password = "YOUR_PASSWORD"
# request header
headers = {
    "Content-Type": "application/x-www-form-urlencoded"
}
data = {
    "grant_type": "password",
    "client_id": client_id,
    "client_secret": client_secret,
    "username": username,
    "password": password,
    "scope": "openid profile email"  #change your scope
}
response = requests.post(token_url, headers=headers, data=data)
TOKEN = response.json()['access_token']

import kfp
kubeflow_host="https://your_host"
pipeline_host = kubeflow_host + "/pipeline" 
client = kfp.Client(host=pipeline_host, existing_token=TOKEN)
print(client.list_runs(namespace="your-profile-name"))

Kubeflow Notebooks User and M2M Authentication and Authorization

The underlying mechanism is the same as in Kubeflow Pipelines.

Similarly, to explain how it works, let's analyze the code step by step, starting from the api route definition for listing notebooks:

KServe Authentication

The analysis of KServe auth capabilities suggests that while it's possible to limit access to only authenticated agents, there might be some improvements required to enable access only to authorized agents.

This is based on the following:

  1. KServe Controller Manager patch integrating kube-rbac-proxy5.

    This suggests the kserve might use the same mechanism based on SubjectAccessReviews. Having a look at the kubeflow/manifests I see it's not enabled.

  2. Search through the docs and code:

    The docs above mention that while it's possible to enable authentication, authorization is more complicated and probably we need to add AuthorizationPolicy...

    create an Istio AuthorizationPolicy to grant access to the pods or disable it

    Most probably some work is needed to enable authorized access to kserve models.

Links

Footnotes

  1. External Authorization ↩

  2. oauth2-proxy ↩

  3. Kubernetes TokenReview ↩

  4. Kubernetes SubjectAccessReview ↩

  5. Kube RBAC Proxy ↩