backend: oidc: impersonate instead of forwarding oidc token to k8s api #2814

iSE-stefan-may · 2025-01-30T16:38:55Z

By default, the OIDC JWT is forwarded to the Kubernetes API server. In some cases it might not be possible to have the Kubernetes API server configured with OIDC, in these cases the OIDC JWT should be evaluated by Headlamp and the request should use the service account of the pod together with impersonating the user in the token.

Related issue:

Support OIDC without the Kuberentes API Server configuration (something similar to Argo CD configuration) #2614

Testing

To test this PR, do the following:

build the container image
follow the documentation at https://headlamp.dev/docs/latest/installation/in-cluster/keycloak/ for setting up a keycloak client and configuring a clusterrole binding (do NOT follow the steps for setting up minikube)
deploy the helm chart with the oidc values and oidc.impersonate=true and oidc.impersonateClaim=preferred_username using the container image from step 1
go to headlamp and login with oidc

zivcex

lg2m

illume

Hello. Thanks for this!

I left a few questions/notes.

Can you please add some steps in the PR description on how we can test this?

Would it be possible for you to add some test cases?

illume · 2025-02-05T10:31:19Z

backend/cmd/headlamp.go

+		}
+
+		idToken, err := c.getValidToken(r.Context(), token)
+		if err != nil {


Can you please add some logging in the error conditions?

Logging added, but I'm not really sure, whether it is really the best case to log always when the token validation fails. It may also be, that OIDC impersonation is active, but the user still used a serviceaccount token, in those cases a warning in the log would be irritating.

Ok. That sounds like it makes sense to remove.

Is something like this possible? Otherwise remove that logging?

if ! OIDCImpersonationEnaled and serviceAccount { log() }

Wether the user provided token is a kubernetes serviceaccount token can only be evaluated from the Kubernetes API, therefore this check won't be possible. Only option where this would be possible would be, if we disable the option to login in the ui via a serviceaccount token altogether. In this case we could give a log warning here, but as far as I can see, that's currently not possible.
I'll cleanup the logging for the cases, where it may not be an error case.

illume · 2025-02-05T10:33:25Z

backend/cmd/headlamp.go

 func StartHeadlampServer(config *HeadlampConfig) {
 	handler := createHeadlampHandler(config)

 	handler = config.OIDCTokenRefreshMiddleware(handler)
+	if config.oidcImpersonate {
+		handler = config.OIDCImpersonateMiddleware(handler)


Can you please explain why this middleware is after the OIDCTokenRefreshMiddleware?

It was the most fitting position I thought of, but not a particular reason. Do you have better recommendations?

I was wondering how it would interact with the other middleware? Wondering if it would make sense to go before OIDCTokenRefreshMiddleware or not?

Because would the token refresh need to use the impersonated stuff? If that's the case I was wondering if the OIDCImpersonateMiddleware should go first?

You are right, the OIDCTokenRefreshMiddleware needs the original token, therefore it must be evaluated before OIDCImpersonateMiddleware. I'll change the order of those two and add a comment.

illume · 2025-02-05T10:35:19Z

backend/pkg/config/config.go

@@ -166,6 +168,10 @@ func flagset() *flag.FlagSet {
 	f.String("oidc-idp-issuer-url", "", "Identity provider issuer URL for OIDC")
 	f.String("oidc-scopes", "profile,email",
 		"A comma separated list of scopes needed from the OIDC provider")
+	f.Bool("oidc-impersonate", false,


@knrt10 would this need to be added to the helm chart as well?

Added to the helm chart, however I'm not convinced that I did it in a good way, since the chart is a little bit confusing for me. So please recheck.

Our main helm chart expert is on holidays at the moment. So this will take a bit longer to review. Bear with me :)

backend/cmd/headlamp.go

illume · 2025-02-05T11:26:42Z

Some lint issues in CI job: https://github.com/headlamp-k8s/headlamp/actions/runs/13056969725/job/36711540140?pr=2814#step:7:27

Signed-off-by: Stefan May <[email protected]>

iSE-stefan-may · 2025-02-06T16:44:32Z

Have updated some code regarding the comments, but not fully tested yet and not added a test description. Will work on that tomorrow, but perhaps you can already review the answers to the comments.

audunsolemdal · 2025-02-06T18:17:55Z

Hi, not an expert here. I am wondering if this implementation will support impersonation based on groups listed in the groups claim found in the token returned from the OIDC provider?

illume · 2025-02-07T07:28:23Z

Have updated some code regarding the comments, but not fully tested yet and not added a test description. Will work on that tomorrow, but perhaps you can already review the answers to the comments.

Thanks. Reading through it now :)

Signed-off-by: Stefan May <[email protected]>

zivcex · 2025-02-07T11:08:14Z

charts/headlamp/templates/deployment.yaml

@@ -147,11 +177,21 @@ spec:
            # Check if scopes are non empty either from env or oidc.config
            - "-oidc-scopes=$(OIDC_SCOPES)"
            {{- end }}
+            {{- if or ($oidc.impersonate) ($impersonate) }}


Hey great PR!!

I'm wondering would it be too heavy if you can provide the groups as well here within the chart and deployment?

more details are here:
https://kubernetes.io/docs/reference/access-authn-authz/authentication/#user-impersonation:~:text=Impersonate%2DGroup%3A%20A%20group%20name%20to%20act%20as.%20Can%20be%20provided%20multiple%20times%20to%20set%20multiple%20groups.%20Optional.%20Requires%20%22Impersonate%2DUser%22.

Convention here is that groups claim in JWT hold the user group coming in from IDP. It would be good if it was configurable as some IDPs send the groups membership under other claim name.

I have added functionality to use groups too and changed the parameters therefore to -oidc-user-claim and -oidc-groups-claim. Also added a filter for additional impersonate-* headers to prevent any authentication attacks, since the service account token is now used.

illume · 2025-02-08T04:37:50Z

It looks like the backend/lint CI job is failing.

You can run make backend-lint locally.

illume · 2025-02-08T04:38:19Z

The helm template job is having an issue: https://github.com/headlamp-k8s/headlamp/actions/runs/13197046279/job/36885634312?pr=2814

Template test failed for ./charts/headlamp/tests/test_cases/oidc-create-secret.yaml against ./charts/headlamp/tests/expected_templates/oidc-create-secret.yaml:
--- ./charts/headlamp/tests/oidc-create-secret.yaml_output/rendered_templates.yaml	2025-02-08 02:48:20.528327104 +0000
+++ ./charts/headlamp/tests/expected_templates/oidc-create-secret.yaml	2025-02-08 02:48:20.536327[13](https://github.com/headlamp-k8s/headlamp/actions/runs/13197046279/job/36885634312?pr=2814#step:5:14)6 +0000
@@ -126,11 +126,6 @@
                 secretKeyRef:
                   name: oidc
                   key: scopes
-            - name: OIDC_IMPERSONATE_CLAIM
-              valueFrom:
-                secretKeyRef:
-                  name: oidc
-                  key: impersonateClaim
make: *** [Makefile:[18](https://github.com/headlamp-k8s/headlamp/actions/runs/13197046279/job/36885634312?pr=2814#step:5:19)5: helm-template-test] Error 1
           args:
             - "-in-cluster"
             - "-plugins-dir=/headlamp/plugins"
Error: Process completed with exit code 2.

To test the charts locally

make helm-template-test

Signed-off-by: Stefan May <[email protected]>

mlbiam · 2025-02-11T00:55:22Z

By default, the OIDC JWT is forwarded to the Kubernetes API server. In some cases it might not be possible to have the Kubernetes API server configured with OIDC, in these cases the OIDC JWT should be evaluated by Headlamp and the request should use the service account of the pod together with impersonating the user in the token.

This is not a good idea. Dashboards should never be configured with privileged token. it's too easy to expose a dashbaord by accident. Even when impersonating the user, headlamp should rely on the token from the proxy, not its own. This way someone accidentally exposes the dashbaord outside of the proxy, they haven't created a vector to access the API server with a privleged account.

illume · 2025-02-11T12:18:24Z

This is not a good idea. Dashboards should never be configured with privileged token. it's too easy to expose a dashbaord by accident. Even when impersonating the user, headlamp should rely on the token from the proxy, not its own. This way someone accidentally exposes the dashbaord outside of the proxy, they haven't created a vector to access the API server with a privleged account.

It is a good idea to try and help people prevent making mistakes with configuration accidentally opening things up.

Also, when hearing about other peoples use cases it is often the situation that they are best placed to evaluate and decide why they want to do things in a particular way.

@iSE-stefan-may maybe you could give us some ideas about this? Could you elaborate on your use case a bit? Also, what do you think about this possibility of opening things up accidentally and how we could address this?

joaquimrocha · 2025-02-14T10:50:08Z

By default, the OIDC JWT is forwarded to the Kubernetes API server. In some cases it might not be possible to have the Kubernetes API server configured with OIDC, in these cases the OIDC JWT should be evaluated by Headlamp and the request should use the service account of the pod together with impersonating the user in the token.

This is not a good idea. Dashboards should never be configured with privileged token. it's too easy to expose a dashbaord by accident. Even when impersonating the user, headlamp should rely on the token from the proxy, not its own. This way someone accidentally exposes the dashbaord outside of the proxy, they haven't created a vector to access the API server with a privleged account.

So far we have avoided requiring or encouraging running Headlamp with its own token/role. We did have some users asking for this and we resorted to advising to use a reverse proxy. We even have thought of maybe having a side project to facilitate this for Headlamp; would this bring any advantage here?

@mlbiam I'd like to ask your opinion whether this is still a security concern if we add the feature as optional and somehow make it clear, via CLI options' names, extra options, and documentation what the implications of this are. What would that look like?

mlbiam · 2025-02-14T12:25:47Z

We did have some users asking for this and we resorted to advising to use a reverse proxy.

Just because users want it doesn't mean its a good idea. Users in general tend to rely on what they can easily control, and a port-forward to a dashboard that is privileged is something they control. Unfortunately, it's also something anyone else can control too! This was how Tesla was breached (https://www.trendmicro.com/vinfo/us/security/news/cybercrime-and-digital-threats/tesla-and-jenkins-servers-fall-victim-to-cryptominers) and has led to the assumption by security teams that the Kubernetes dashboard is "insecure". It's also the reason why the Kubernetes Dashboard moved away from being able to use its own ServiceAccount token and requiring a reverse proxy to inject a token in the 7.x version.

Headlamp already has a "local" mode. If users don't want to do it "correctly" in a centralized way, they should use the local version.

We even have thought of maybe having a side project to facilitate this for Headlamp; would this bring any advantage here?

easier with good security is always better. That said, i don't know you need a new project. OpenUnison does this today. You can also use oauth2proxy+kube-oidc-proxy too if you don't want to use openunison: https://www.tremolo.io/post/kubernetes-authentication-comparing-solutions

That said, you're essentially talking about rolling your own authentication code, which i'd recommend against. Best to invest time in good docs, better helm charts, etc, for existing projects to make this easier. The hard part isn't the proxies, it's setting up DNS, TLS, integration with a authentication team that is in a different silo in your org, etc. Creating another project won't fix those issues, just create one more thing to support.

I'd like to ask your opinion whether this is still a security concern if we add the feature as optional and somehow make it clear, via CLI options' names, extra options, and documentation what the implications of this are

I'd still say this is an anti-pattern. All the red flags and warnings won't stop users from doing the easy thing. ie: clicking through a security warning for an invalid certificate. It will only take one major breach for Headlamp to earn an "insecure" badge.

The Kubernetes Dashbaord moved awat from this pattern for 7.x and has received considerable pushback. Their answer (which I think is correct), is setup a reverse proxy that hard codes an SA in it. It's terribly insecure, but so's running the dashboard with a privileged sa. It at least makes it harder to do the insecure thing and protects the project's reputation.

zivcex · 2025-02-19T10:57:20Z

Hi @iSE-stefan-may will you close this one given the new circumstances? Or leave this setting optional?

aslafy-z · 2025-02-24T04:34:25Z

backend/cmd/headlamp.go

+		}
+		r.Header.Set("Impersonate-User", user)
+		for _, group := range groups {
+			r.Header.Set("Impersonate-Group", group)


I believe this should use Add instead of Set.

Suggested change

r.Header.Set("Impersonate-Group", group)

r.Header.Add("Impersonate-Group", group)

k8s-triage-robot · 2025-04-01T11:26:20Z

Unknown CLA label state. Rechecking for CLA labels.

Send feedback to sig-contributor-experience at kubernetes/community.

/check-cla
/easycla

linux-foundation-easycla · 2025-04-01T11:26:26Z

❌ - login: @iSE-stefan-may . The commit (039dde1, bd747b2, 53fadbe, a1bbd62) is not authorized under a signed CLA. Please click here to be authorized. For further assistance with EasyCLA, please submit a support request ticket.

engnatha · 2025-04-11T13:22:40Z

The security arguments above are valid. I will offer that I think it's quite simple to restrict impersonation side-effects. The service account mounted can be limited to which groups it's allowed to impersonate with simple Roles/ClusterRoles. If something like this isn't supported, it leaves users in a catch 22. None of the documentation or issues want to describe how to utilize this on managed platforms like GKE of which they are strongly moving away basic OIDC in favor of google-native solutions (ref).

Impersonation offers a nice middle ground in these scenarios. To offer some motivation, this is how the weaveworks UI works for fluxcd. https://docs.gitops.weaveworks.org/docs/0.24.0/configuration/recommended-rbac-configuration/#summary. By tightly controlling the groups that can be impersonated, I can very confidently restrict what my app's service account is capable of doing.

joaquimrocha · 2025-04-11T14:11:19Z

What if we do something similar to https://github.com/aslafy-z/k8s-dashboard-impersonation-proxy and provide a way to easily use this with Headlamp's Helm chart?

mlbiam · 2025-04-11T14:52:03Z

To @engnatha 's point about using RBAC to limit impersonation - There are two threats:

if someone gets a hold of the container's token. Either via an RCE against the headlamp container or sideways from inside the cluster (as in the Tesla breach on the Kubernetes Dashboard). Forcing headlamp users to use a purpose built proxy with the privileged account cuts down on the attach surface (if the frontend only exposes APIs, there's less to attack) and making it easier to upgrade, keep up to date, test against, etc.
RBAC impersonation is very limited for impersonation. You can limit which groups are impersonated, and you can limit which users are impersonated, but you can't limit which users can impersonate which groups. These leads to an over permissioned token that if stolen, allows an attacker incredible flexibility.

Using the dashbaord's service account as privileged was pretty big security issue with the kubernetes dashboard and they removed that capability correctly (hell, i wrote an entire chapter of the Kube: an enterprise guide around how to attack the dashboard running a privileged account until they removed the capability in 7.x)

While I think the lessons learned from the kubernetes dashboard project advise against making the headlamp service account privileged, if you really want to go this route it shouldn't be the default. The headlamp account shouldn't be used as part of impersonation, but if you want it to be there should be a switch that has to be turned on.

@joaquimrocha including a reverse proxy in the helm charts to do this would probably be not as straight forward as you'd hope. This biggest issues we run into helping customers and users run OpenUnison is certificate management and ingress integration. Assuming you're encrypting e2e (and given these tokens are privileged, i certainly would hope that would be the requirement) you're talking about either requiring cert-manager or building your own cert-manager like system. Then there's testing with each ingress controller/gateway. You're going to get much better results ivesting efforts into documenting how to deploy headlamp with impersonation then trying to engineer the issue.

Also, who "owns" that proxy you're including? Who's responsible for continuous testing? Not just happy-path testing, but failure mode testing to protect against mallicious use? If that component fails, they're going to blame the headlamp project the same way that the kubernetes dashboard still has the assumption of being "insecure" post tesla breach.

engnatha · 2025-04-11T15:42:25Z

I think impersonation works better for the weaveworks UI because the operations exposed are so much more limited. You're really only interested in viewing resources and patching existing resources of a certain kind. While you could do some harm with that, I think the blast radius minimized. In the headlamp system, there would be a much richer ecosystem of permissions to give more access to the cluster for admins of different components.

I agree that it will never not be a threat. I'm mostly just trying to advocate as a potential user that the current system is forcing users away from the product. With no guides on how to set up the system on managed clusters (which is becoming the norm rather than the exception) and no willingness to provide alternate mechanisms like proxies, impersonation, or otherwise, we aren't left with many options. I've tried hard to search through existing issues and documentation on how to do this with modern systems and it just isn't there. Even looking at the platforms page, the big cloud provider mentioned for support is AWS and the rationale is someone in 2021 saying it worked with no mention of how. I do really respect the push for security here and if this PR does get blocked, I hope it provides motivation to teach users how to utilize this system properly. The alternative is everyone reinventing the wheel and likely introducing their own vulnerabilities with proxies and impersonations of their own or more widely sharing long lived service account tokens.

engnatha · 2025-04-11T18:10:58Z

Looking into this more, I actually think the current authentication strategy imposed by headlamp is impossible to accommodate in modern GKE. Based on the guidance in https://cloud.google.com/kubernetes-engine/docs/how-to/oidc#external-idp-authentication-methods, we see the following

In GKE Standard clusters only, GKE also supports Identity Service for GKE. Identity Service for GKE is limited to OIDC IdPs and installs additional components in your cluster. GKE strongly recommends using Workforce Identity Federation instead of Identity Service for GKE.

They are migrating everyone to workforce identity federation which would require headlamp to route through that system to get the proper short-lived google service account tokens. I definitely don't know as much about this system, so I'd really like to be wrong about this if anyone has more experience 😁

dosubot bot added the size:L This PR changes 100-499 lines, ignoring generated files. label Jan 30, 2025

iSE-stefan-may force-pushed the feat/oidc-impersonate branch from 8661611 to cdb40c2 Compare January 30, 2025 16:41

iSE-stefan-may mentioned this pull request Jan 31, 2025

Support OIDC without the Kuberentes API Server configuration (something similar to Argo CD configuration) #2614

Open

zivcex approved these changes Feb 4, 2025

View reviewed changes

illume requested changes Feb 5, 2025

View reviewed changes

umarhussain15 mentioned this pull request Feb 5, 2025

Secure access to in-cluster headlamp with a reverse proxy #2207

Open

backend: oidc: impersonate instead of forwarding oidc token to k8s api

a1bbd62

Signed-off-by: Stefan May <[email protected]>

iSE-stefan-may force-pushed the feat/oidc-impersonate branch from cdb40c2 to a1bbd62 Compare February 6, 2025 16:36

iSE-stefan-may added 2 commits February 7, 2025 10:02

Merge branch 'main' into feat/oidc-impersonate

bd747b2

backend: oidc: adaptions

53fadbe

Signed-off-by: Stefan May <[email protected]>

zivcex reviewed Feb 7, 2025

View reviewed changes

backend: oidc: cleanup, oidc groups added, filter for other headers

039dde1

Signed-off-by: Stefan May <[email protected]>

aslafy-z reviewed Feb 24, 2025

View reviewed changes

k8s-ci-robot added the cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. label Apr 1, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

backend: oidc: impersonate instead of forwarding oidc token to k8s api #2814

backend: oidc: impersonate instead of forwarding oidc token to k8s api #2814

iSE-stefan-may commented Jan 30, 2025 •

edited by illume

Loading

zivcex left a comment

illume left a comment

illume Feb 5, 2025

iSE-stefan-may Feb 6, 2025

illume Feb 8, 2025

iSE-stefan-may Feb 10, 2025

illume Feb 5, 2025

iSE-stefan-may Feb 6, 2025

illume Feb 8, 2025

iSE-stefan-may Feb 10, 2025

illume Feb 5, 2025

iSE-stefan-may Feb 6, 2025

illume Feb 8, 2025

illume commented Feb 5, 2025

iSE-stefan-may commented Feb 6, 2025

audunsolemdal commented Feb 6, 2025

illume commented Feb 7, 2025

zivcex Feb 7, 2025

iSE-stefan-may Feb 10, 2025

illume commented Feb 8, 2025 •

edited

Loading

illume commented Feb 8, 2025 •

edited

Loading

mlbiam commented Feb 11, 2025

illume commented Feb 11, 2025

joaquimrocha commented Feb 14, 2025

mlbiam commented Feb 14, 2025

zivcex commented Feb 19, 2025

aslafy-z Feb 24, 2025

k8s-triage-robot commented Apr 1, 2025

linux-foundation-easycla bot commented Apr 1, 2025

engnatha commented Apr 11, 2025

joaquimrocha commented Apr 11, 2025

mlbiam commented Apr 11, 2025

engnatha commented Apr 11, 2025

engnatha commented Apr 11, 2025

	r.Header.Set("Impersonate-Group", group)
	r.Header.Add("Impersonate-Group", group)

backend: oidc: impersonate instead of forwarding oidc token to k8s api #2814

Are you sure you want to change the base?

backend: oidc: impersonate instead of forwarding oidc token to k8s api #2814

Conversation

iSE-stefan-may commented Jan 30, 2025 • edited by illume Loading

Testing

zivcex left a comment

Choose a reason for hiding this comment

illume left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

illume commented Feb 5, 2025

iSE-stefan-may commented Feb 6, 2025

audunsolemdal commented Feb 6, 2025

illume commented Feb 7, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

illume commented Feb 8, 2025 • edited Loading

illume commented Feb 8, 2025 • edited Loading

To test the charts locally

mlbiam commented Feb 11, 2025

illume commented Feb 11, 2025

joaquimrocha commented Feb 14, 2025

mlbiam commented Feb 14, 2025

zivcex commented Feb 19, 2025

Choose a reason for hiding this comment

k8s-triage-robot commented Apr 1, 2025

linux-foundation-easycla bot commented Apr 1, 2025

engnatha commented Apr 11, 2025

joaquimrocha commented Apr 11, 2025

mlbiam commented Apr 11, 2025

engnatha commented Apr 11, 2025

engnatha commented Apr 11, 2025

iSE-stefan-may commented Jan 30, 2025 •

edited by illume

Loading

illume commented Feb 8, 2025 •

edited

Loading

illume commented Feb 8, 2025 •

edited

Loading