Keep being redirected to login page with Keycloak OIDC provider and pinniped

[rbuffi]

My goal is to authenticate to kubeapps with keycloak and pinniped. I have configured everything but i keep bein redirected to the login page...

Here is my values.yaml:

authProxy:

  enabled: true

  skipKubeappsLoginPage: false

  provider: oidc

  clientID: kubeapps

  clientSecret: xxxx

  cookieSecret: xxx

  emailDomain: "*"

  extraFlags:

    - --cookie-refresh=0

    - --ssl-insecure-skip-verify

    - --cookie-secure=false

    - --scope=openid groups email

    - --oidc-issuer-url=https://kc.testlab.xxxx.local/realms/kubeapps

    - --pass-authorization-header=true
 

pinnipedProxy:

  enabled: true

  clusters:

     - name: default

       apiServiceURL: https://x.x.x.x/

       certificateAuthorityData: xxxx

       isKubeappsCluster: true

       pinnipedConfig:

         enabled: true

I now have set up the impersonation proxy:

apiVersion: v1

items:

- apiVersion: [config.concierge.pinniped.dev/v1alpha1](http://config.concierge.pinniped.dev/v1alpha1)

  kind: CredentialIssuer

  metadata:

    creationTimestamp: "2024-06-25T14:36:04Z"

    generation: 2

    labels:

      app: pinniped-concierge

    name: pinniped-concierge-config

    resourceVersion: "16012020"

    uid: a6b6b570-311b-4b00-9706-71f44671cfa7

  spec:

    impersonationProxy:

      mode: enabled

      service:

        annotations:

          [service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout](http://service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout): "4000"

        type: LoadBalancer

  status:

    strategies:

    - lastUpdateTime: "2024-06-25T14:36:13Z"

      message: could not find a healthy kube-controller-manager pod (0 candidates)

      reason: CouldNotFetchKey

      status: Error

      type: KubeClusterSigningCertificate

    - frontend:

        impersonationProxyInfo:

          certificateAuthorityData: xxxx

          endpoint: https://x.x.x.x/

        type: ImpersonationProxy

      lastUpdateTime: "2024-06-25T22:41:48Z"

      message: impersonation proxy is ready to accept client connections

      reason: Listening`

And jwtauthenticator:

apiVersion: v1

items:

- apiVersion: [authentication.concierge.pinniped.dev/v1alpha1](http://authentication.concierge.pinniped.dev/v1alpha1)

  kind: JWTAuthenticator

  metadata:

    creationTimestamp: "2024-06-26T00:20:50Z"

    generation: 1

    name: jwt-authenticator

    resourceVersion: "16033939"

    uid: ac12cf5c-228d-494c-9f1f-80044a75f01c

  spec:

    audience: kubeapps

    claims:

      groups: groups

      username: email

    issuer: https://kc.testlab.x.x/realms/kubeapps

    tls:

      certificateAuthorityData: xxxx
kind: List

metadata:

  resourceVersion: ""

With this config i'm able to authenticate to kubeapps with keycloak but after authentication i'm being redirected to the login page. In kubeapps auth-proxy pod logging I see nothing strange and nothing being logged in pinniped-proxy pod!

[10.244.1.1:45115](http://10.244.1.1:45115/) - 372269b1-2a3d-4de1-88b6-31843b95e5e5 - [user@anonymous.nl](mailto:user@anonymous.nl) [2024/06/26 08:31:24] [AuthSuccess] Authenticated via OAuth2: Session{email:user@anonymous user:93424824-a080-4690-ae1d-8346c40efc0e [PreferredUsername:user@anonymous.nl](mailto:PreferredUsername%3Auser@anonymous.nl) token:true id_token:true created:2024-06-26 08:31:24.585448393 +0000 UTC m=+2920.873606365 expires:2024-06-26 08:36:24.500799825 +0000 UTC m=+3220.788957799 refresh_token:true groups:[kubeapps-admin]}

[10.244.1.1:45115](http://10.244.1.1:45115/) - 372269b1-2a3d-4de1-88b6-31843b95e5e5 - - [2024/06/26 08:31:24] 192.168.210.116 GET - "/oauth2/callback?state=7mtfXVKtt4-AbTYHzCvZIlvAizmJ1CdwH-LIu2rPo_s%3A%2F&session_state=96c2dfdb-3722-4d3d-bb52-e54c3d501829&iss=https%3A%2F%2Fkc.testlab.xxx.local%2Frealms%2Fkubeapps&code=4fa74f08-ca24-4193-8bb1-d0db9b293f4f.96c2dfdb-3722-4d3d-bb52-e54c3d501829.cb382bec-bc96-4750-a889-7e34456c8a8d" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/[126.0.0.0](http://126.0.0.0/) Safari/53

But in the apiserver logging I see the following:

I0626 08:56:41.131411       1 handler.go:232] Adding GroupVersion [identity.concierge.pinniped.dev](http://identity.concierge.pinniped.dev/) v1alpha1 to ResourceManager

I0626 08:56:41.144661       1 handler.go:232] Adding GroupVersion [login.concierge.pinniped.dev](http://login.concierge.pinniped.dev/) v1alpha1 to ResourceManager

E0626 08:57:06.728431       1 controller.go:102] loading OpenAPI spec for "[v1alpha1.identity.concierge.pinniped.dev](http://v1alpha1.identity.concierge.pinniped.dev/)" failed with: failed to download [v1alpha1.identity.concierge.pinniped.dev](http://v1alpha1.identity.concierge.pinniped.dev/): resource not found

I0626 08:57:06.728494       1 controller.go:109] OpenAPI AggregationController: action for item [v1alpha1.identity.concierge.pinniped.dev](http://v1alpha1.identity.concierge.pinniped.dev/): Rate Limited Requeue.

E0626 08:57:06.828889       1 controller.go:102] loading OpenAPI spec for "[v1alpha1.login.concierge.pinniped.dev](http://v1alpha1.login.concierge.pinniped.dev/)" failed with: failed to download [v1alpha1.login.concierge.pinniped.dev](http://v1alpha1.login.concierge.pinniped.dev/): resource not found

 1 authentication.go:73] "Unable to authenticate the request" err="invalid bearer token"
When I try to decode the token as described (https://kubeapps.dev/docs/latest/howto/oidc/oauth2oidc-debugging/)
I get the following error:

{"alg":"RS256","typ" : "JWT","kid" : "wkF65vug7ZdfpsKzc5Fpt_qCUHNZo_37uwxhDzoU5v8"}base64: invalid input

In the concierge pod logging I do not see any token requests.

I able to get token with pinniped-cli and keycloak/pinniped impersonating proxy:

pinniped-cli-windows-amd64.exe login oidc --issuer https://kc.testlab.xxx.local/realms/kubeapps --ca-bundle-data XXXX  --client-id kubeapps --enable-concierge --concierge-endpoint https://192.168.x.x --concierge-authenticator-name jwt-authenticator --concierge-authenticator-type jwt --scopes openid,groups,email --concierge-ca-bundle-data xxxx

`Wed, 26 Jun 2024 14:30:07 CEST rest/warnings.go:70 Use tokens from the TokenRequest API or manually created secret-based tokens instead of auto-generated secret-based tokens.

Result:
 {"kind":"ExecCredential","apiVersion":"[client.authentication.k8s.io/v1beta1](http://client.authentication.k8s.io/v1beta1)","spec":{"interactive":false},"status":{"expirationTimestamp":"2024-06-26T12:35:07Z","clientCertificateData":"-----BEGIN CERTIFICATE-----\nCERTIFICATE\n-----END CERTIFICATE-----\n","clientKeyData":"-----BEGIN PRIVATE KEY-----\nKEY\n-----END PRIVATE KEY-----\n"}}`

• Version: latest
• Talos version 1.7
• K8s version 1.28

[saashqdev]

Did you ever resolve this? I have the same issue. I think it has to do with ID token vs Access token. I think Kubernetes is expecting an access token and an id token is being sent. I'm not sure.

[rbuffi]

My issue was kubeapps related, there was a indentation error in the pinniped-proxy config .

[saashqdev]

Ah, OK thanks. I'm running mine on a K3s cluster at Hetzner with a Hetzner loadbalancer. Pinniped doesn;t seem to like the Hetnzer loadbalancer or something It just refuses to start. (I'm using the latest values.yaml file) from the kubeapps project.

[saashqdev]

Just a shot in the dark here but would you be interested in a small 1 hour consult to help us get this going on our cluster? We've got everything up to Keycloak running fine - it's just this last piece we can't seem to figure out. Cheers, Dave

[rbuffi]

What refuses to start? Pinniped or pinniped-proxy. Does it throw an error? How could we do this consult?

[saashqdev]

Thanks for getting back. We have a k3s cluster at Hetzner with a Hetzner loadbalancer. The plan is to install:

1) Keycloak (oidc provider)
2) Kubeapps

and offer a service to small businesses utilizing Kubeapps.

Keycloak is installed and is setup with a proper realm and client/client scopes/mappers/users and it all works and has been tested. The next step was installing Kubeapps and use Keycloak as the oidc provider to Kubeapps.

We installed Kubeapps with:

helm upgrade --install kubeapps -n kubeapps oci://registry-1.docker.io/bitnamicharts/kubeapps -f values.yaml

We enabled pinniped in the values file and used a separate ingress.yaml (all attached). Kubeapps comes up when we hit https://kubeapps.saashq.org but bypasses Pinniped and offers login via Token. We can't seem to get it to launch Keycloak and then log us in and redirect to Kubeapps.

ingress.txt
values.txt

osboxes@osboxes:~/Downloads/hetzner/kubeapps$ kubectl get all -n kubeapps
NAME                                                             READY   STATUS      RESTARTS        AGE
pod/apprepo-kubeapps-sync-bitnami-28910640-4brt4                 0/1     Completed   0               20m
pod/apprepo-kubeapps-sync-bitnami-28910650-xs5sn                 0/1     Completed   0               10m
pod/apprepo-kubeapps-sync-bitnami-28910660-xcxt2                 0/1     Completed   0               9s
pod/kubeapps-59bfcc5c84-52shc                                    1/1     Running     0               16h
pod/kubeapps-59bfcc5c84-qkn79                                    1/1     Running     0               16h
pod/kubeapps-6c589fdc7d-rkg4f                                    0/2     Pending     0               58m
pod/kubeapps-internal-apprepository-controller-79c6d7957-rsbck   1/1     Running     0               3h38m
pod/kubeapps-internal-dashboard-5c95f97cdd-pnrkw                 1/1     Running     0               15h
pod/kubeapps-internal-dashboard-5d885c754b-dhn7l                 0/1     Pending     0               58m
pod/kubeapps-internal-dashboard-5d885c754b-nlt82                 1/1     Running     0               3h38m
pod/kubeapps-internal-kubeappsapis-5f44c54ffb-cjmj6              1/1     Running     2 (16h ago)     16h
pod/kubeapps-internal-kubeappsapis-5f44c54ffb-s9jtp              1/1     Running     2 (16h ago)     16h
pod/kubeapps-internal-kubeappsapis-8464cfd7f4-xpc5f              0/1     Pending     0               58m
pod/kubeapps-postgresql-0                                        1/1     Running     26 (157m ago)   15h

NAME                                       TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/kubeapps                           ClusterIP   10.43.109.179   <none>        80/TCP     16h
service/kubeapps-internal-dashboard        ClusterIP   10.43.233.140   <none>        8080/TCP   16h
service/kubeapps-internal-kubeappsapis     ClusterIP   10.43.105.112   <none>        8080/TCP   16h
service/kubeapps-internal-pinniped-proxy   ClusterIP   10.43.216.191   <none>        3333/TCP   58m
service/kubeapps-postgresql                ClusterIP   10.43.47.137    <none>        5432/TCP   16h
service/kubeapps-postgresql-hl             ClusterIP   None            <none>        5432/TCP   16h

NAME                                                         READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/kubeapps                                     2/2     1            2           16h
deployment.apps/kubeapps-internal-apprepository-controller   1/1     1            1           16h
deployment.apps/kubeapps-internal-dashboard                  2/2     2            2           16h
deployment.apps/kubeapps-internal-kubeappsapis               2/2     1            2           16h

NAME                                                                    DESIRED   CURRENT   READY   AGE
replicaset.apps/kubeapps-564cb48bc5                                     0         0         0       15h
replicaset.apps/kubeapps-56bd844864                                     0         0         0       114m
replicaset.apps/kubeapps-59bfcc5c84                                     2         2         2       16h
replicaset.apps/kubeapps-6c589fdc7d                                     1         1         0       3h38m
replicaset.apps/kubeapps-6d797f55b                                      0         0         0       112m
replicaset.apps/kubeapps-6f8c48cd77                                     0         0         0       3h24m
replicaset.apps/kubeapps-7b9c64cf6c                                     0         0         0       15h
replicaset.apps/kubeapps-7fcf5879c4                                     0         0         0       91m
replicaset.apps/kubeapps-internal-apprepository-controller-5db87695f7   0         0         0       16h
replicaset.apps/kubeapps-internal-apprepository-controller-79c6d7957    1         1         1       3h38m
replicaset.apps/kubeapps-internal-dashboard-5c95f97cdd                  1         1         1       15h
replicaset.apps/kubeapps-internal-dashboard-5d885c754b                  2         2         1       3h38m
replicaset.apps/kubeapps-internal-dashboard-7479bdf494                  0         0         0       114m
replicaset.apps/kubeapps-internal-dashboard-7f669b85b8                  0         0         0       16h
replicaset.apps/kubeapps-internal-kubeappsapis-5f44c54ffb               2         2         2       16h
replicaset.apps/kubeapps-internal-kubeappsapis-5f4bd9bdd4               0         0         0       112m
replicaset.apps/kubeapps-internal-kubeappsapis-8464cfd7f4               1         1         0       3h38m

NAME                                   READY   AGE
statefulset.apps/kubeapps-postgresql   1/1     16h

NAME                                          SCHEDULE       TIMEZONE   SUSPEND   ACTIVE   LAST SCHEDULE   AGE
cronjob.batch/apprepo-kubeapps-sync-bitnami   */10 * * * *   <none>     False     0        10s             16h

NAME                                               STATUS     COMPLETIONS   DURATION   AGE
job.batch/apprepo-kubeapps-sync-bitnami-28910640   Complete   1/1           3s         20m
job.batch/apprepo-kubeapps-sync-bitnami-28910650   Complete   1/1           4s         10m
job.batch/apprepo-kubeapps-sync-bitnami-28910660   Complete   1/1           3s         10s

OK, UPDATE - I just cleared out ~/.kube/cache/ and now the login is asking for the proper oidc provider. Now it goes to https://kubeapps.saashq.org/oauth2/start/ and is blank. I've go the re-direct urls set to: https://kubeapps.saashq.org/* in Keycloak

For the consult we can give you access to the test cluster and maybe you could try it from your end and provide some insight. Whatever your hourly rate is. Cheers, Dave

[rbuffi]

I can take a look next week if that’s soon enough for you. How can I get back to you?

[saashqdev]

No worries, I finally got it going.I had to create an audience in Keycloak and assign to Client. Back in the day aud was automatically included in the id_tokens. Need to add them now.

[rbuffi]

Good to hear you got it working! Cheers, Ronald