OIDC error with Azure

[cdenneen]

When trying to setup Flux Operator web with Azure (like I've done with other web ui's). I'm getting an error:

Flux Operator log show:
failed to verify OIDC ID token: failed to verify signature: failed to verify id token signature

Web UI shows: "Authentication failed. Please try again."

Not sure how to debug.

[matheuscscp]

Not sure how to debug.

We can start with:

• Seeing your configuration (with secrets redacted). I need to be able to reproduce it.
• Seeing which other apps you have configured with Azure, and how the configuration looks like.

[stefanprodan]

@cdenneen have you've setup a Dex connector for Azure AD like so https://dexidp.io/docs/connectors/oidc/

Please note that the Flux Operator does not work directly with Azure or any other cloud IDP, you need Dex as an intermediary IDP.

[matheuscscp]

@cdenneen It worked for me with this configuration (redacted values):

baseURL: https://flux-status.example.com
authentication:
  type: OAuth2
  oauth2:
    provider: OIDC
    clientID: 2d01bd48-7914-4b50-9667-068be6afd2f2
    clientSecret: "O.y8Q~MGIm9B.ahUlOx376EP7l5mu9xgIet6hdBD"
    issuerURL: https://login.microsoftonline.com/4bd94393-a3a0-ab26-4c05-bfc69377f6c0/v2.0
    scopes: [openid, profile, email, offline_access]

[matheuscscp]

@cdenneen While the configuration above works, it is limited to username only for RBAC impersonation. To have also groups you will need what Stefan mentioned, a third-party tool like Dex that has native integration with Azure for retrieving groups through their APIs. The correct docs for Azure in Dex are https://dexidp.io/docs/connectors/microsoft/ (and not https://dexidp.io/docs/connectors/oidc/).

[cdenneen]

Does Flux support anything other than Dex? Amazon Cognito? Having Dex as middleman for Azure is something I'd like to get away from managing(we have Cognito already setup for Azure so would definitely consider this if I can't go directly to Entra). I'll try to update my setup with the scopes (thought those were the defaults if not set) to see if signature issue goes away. I do really want the groups but if it's working as username I should see different error due to my user CDENNEEN not having proper permissions vs the signature error.

[cdenneen]

@cdenneen While the configuration above works, it is limited to username only for RBAC impersonation. To have also groups you will need what Stefan mentioned, a third-party tool like Dex that has native integration with Azure for retrieving groups through their APIs. The correct docs for Azure in Dex are https://dexidp.io/docs/connectors/microsoft/ (and not https://dexidp.io/docs/connectors/oidc/).

I've used Entra with Grafana and Headlamp and it does return the groups claim so not 100% sure why one would need Dex here but I will provide redacted auth config for either of them if it helps.

[matheuscscp]

I've used Entra with Grafana and Headlamp and it does return the groups claim so not 100% sure why one would need Dex here but I will provide redacted auth config for either of them if it helps.

That's because Grafana and Headlamp have Azure-specific code. Our only implementation at the moment relies on pure/generic OIDC. This means getting an OIDC ID token from the provider and looking at what claims are inside. Azure OIDC does not return group claims in the ID token. These apps work directly with Azure because they have Azure-specific code that calls the Azure APIs to list groups.

Dex is an SSO broker that has this kind of provider-specific integration for many providers, hence why we recommend it.

We will implement provider-specific integrations when ControlPlane customers ask for it.

Does Flux support anything other than Dex? Amazon Cognito?

The Flux Operator Web UI supports any OIDC provider. It will look at the claims the OIDC provider adds to the ID token. If the provider adds group claims to the ID token (like Dex), then groups will work. Azure OIDC does not add group claims to the ID token. Does Amazon Cognito support OIDC? If yes, does Amazon Cognito OIDC support group claims? If yes, then groups will work.

thought those were the defaults if not set

Those are not. The default scopes are documented here: https://fluxoperator.dev/docs/web-ui/web-config-api/#oidc-provider

But the error you would see for wrong scopes is not the one you saw. (I really can't explain it, btw)

[matheuscscp]

Also, not sure you're following the PR I linked to resolve this issue, but I tested the integration via Dex and the groups do work:

#599 (comment)

[cdenneen]

@cdenneen It worked for me with this configuration (redacted values):

baseURL: https://flux-status.example.com
authentication:
type: OAuth2
oauth2:
provider: OIDC
clientID: 2d01bd48-7914-4b50-9667-068be6afd2f2
clientSecret: "O.y8Q~MGIm9B.ahUlOx376EP7l5mu9xgIet6hdBD"
issuerURL: https://login.microsoftonline.com/4bd94393-a3a0-ab26-4c05-bfc69377f6c0/v2.0
scopes: [openid, profile, email, offline_access]

Interesting so I changed my issuerURL from the sts.windows.net that I had used for other apps to this and got the error about groups scope in operator logs not the web UI. So I explicitly added the scopes (minus the groups) and it now doesn't error at all on web. No error in flux-operator logs but just keeps looping back to SSO screen. At this point I'd expect the RBAC error for CDENNEEN has no perm??

apiVersion: web.fluxcd.controlplane.io/v1
kind: Config
spec:
  authentication:
    oauth2:
      clientID: REDACT
      clientSecret: REDACT
      issuerURL: https://login.microsoftonline.com/REDACT/v2.0
      provider: OIDC
      scopes: [openid, profile, email, offline_access]
    type: OAuth2
  baseURL: https://flux.example.com

[matheuscscp]

At this point I'd expect the RBAC error for CDENNEEN has no perm??

RBAC comes from user and/or groups. The CEL expressions we provide for extracting these two things from the ID token claims default to empty. So if e.g. groups is not present, you should not see RBAC errors, you just don't have any groups but you still have user. This is what you should see with direct integration and zero RBAC:

[matheuscscp]

To further debug:

• Open the network debugging panel in the browser and see the HTTP requests that are happening
• Set --log-level=debug in the Flux Operator/Standalone Web UI deployment

[matheuscscp]

@cdenneen Sorry, the yellow panel above is not released yet. This is from our preview environment, which mirrors the main branch from this repo... But you should see everything else besides this yellow panel.