[ISSUE] Unable to disable identity component independently from identityKeycloak in Camunda Platform Helm chart

[artmimois]

[ISSUE] Keycloak OTP prevents Identity pod connection and Identity pod cannot be disabled independently

Describe the issue:

This report covers two related issues encountered with the Camunda Platform Helm chart regarding the Identity component and its interaction with Keycloak:

1. Identity Connection Failure with Keycloak Admin OTP: When the primary administrative user in the bundled Keycloak instance has OTP (One-Time Password) enabled for enhanced security, the Camunda Identity pod fails to establish a connection to Keycloak.
2. Inability to Disable Identity Pod Independently: The Helm chart includes a constraint that prevents disabling the Identity pod (identity.enabled: false) if the identityKeycloak component (which manages the embedded Keycloak) is enabled (identityKeycloak.enabled: true). This makes it difficult to run the platform with the embedded Keycloak but without the Camunda Identity component itself.

Actual behavior:

1. OTP Issue: When OTP is configured and enforced for the Keycloak admin user (or the user Identity attempts to connect with), the Identity pod enters a crash loop or continually logs connection errors similar to the following:

   2025-05-02 20:42:21.431 ERROR 1 --- [ main] i.c.i.i.k.c.KeycloakConfiguration : Failure #3. Unable to connect to Keycloak.
   

   (Note: The timestamp is illustrative as provided in the original report). The pod fails to become ready because it cannot authenticate with Keycloak using the standard credentials when OTP is required for that user.

2. Disabling Constraint Issue: When attempting to deploy or upgrade the Helm chart with the following values:

   identity:
     enabled: false
   identityKeycloak:
     enabled: true # Assuming embedded Keycloak is desired
   # ... other values

   The Helm operation fails during the template rendering phase with a constraint violation error:

   Unable to save changes: application spec for camunda is invalid: InvalidSpecError: Unable to generate manifests in : rpc error: code = Unknown desc = helm template . --name-template camunda --namespace camunda --kube-version 1.31 --values /tmp/f21185a2-2aad-480e-9619-cb1d1e20c6ef <api versions removed> --include-crds failed exit status 1: Error: execution error at (camunda-platform/templates/camunda/constraints.tpl:68:52): [camunda][error] Identity is disabled but identityKeycloak is enabled. Please ensure that if identityKeycloak is enabled, Identity must also be enabled.
   

Expected behavior:

1. OTP Issue: The Identity pod should successfully connect to Keycloak even if the administrative user it's configured to use has OTP enabled. This might involve:
   • Using a dedicated service account or client credentials flow that doesn't require OTP.
   • Providing configuration options within the Identity component to handle OTP, although this seems less likely for a backend service connection.
   • Clear documentation on how to configure Keycloak and Identity for secure connection (e.g., recommending a separate service user without OTP but with restricted permissions).
2. Disabling Constraint Issue: It should be possible to disable the Camunda Identity component (identity.enabled: false) while still using the bundled Keycloak (identityKeycloak.enabled: true), perhaps for managing users for other applications or custom integrations, without triggering a Helm chart constraint error. Alternatively, the relationship and dependencies should be clearly documented, possibly suggesting disabling both if Identity is not needed.

How to reproduce:

For Issue 1 (OTP Connection Failure):

1. Deploy Camunda Platform using the Helm chart (version 11.3.0 or similar) with default settings for Identity and Keycloak (identity.enabled: true, identityKeycloak.enabled: true).
2. Access the bundled Keycloak instance UI.
3. Log in as the admin user.
4. Navigate to the user's account settings or security settings.
5. Enable OTP (e.g., using Google Authenticator or FreeOTP).
6. Log out and log back in to ensure OTP is enforced for the admin user.
7. Restart or observe the Camunda Identity pod (kubectl get pods -n <namespace>, kubectl logs <identity-pod-name> -n <namespace>).
8. Note the recurring connection errors in the Identity pod logs.

For Issue 2 (Disabling Constraint):

1. Prepare a values.yaml file for the Camunda Platform Helm chart (version 11.3.0 or similar).
2. Set the following values:

   identity:
     enabled: false
   identityKeycloak:
     enabled: true
   # Ensure other necessary values are set

3. Attempt to deploy or upgrade the chart using Helm:

   helm template camunda camunda/camunda-platform --version 11.3.0 --namespace <namespace> -f values.yaml
   # or
   helm upgrade --install camunda camunda/camunda-platform --version 11.3.0 --namespace <namespace> -f values.yaml

4. Observe the Helm error message related to the constraint violation.

Logs:

Issue 1 Log Snippet (Identity Pod):

2025-05-02 20:42:21.431 ERROR 1 --- [ main] i.c.i.i.k.c.KeycloakConfiguration : Failure #3. Unable to connect to Keycloak.

Helm values:

        global:
          secrets:
            autoGenerated: false
          elasticsearch:
            enabled: false
            external: false
          opensearch:
            enabled: true
            distribution: opensearch
            version: 2.17
            auth:
              username: "admin"
              existingSecret: "camunda-camunda-platform-opensearch"
              existingSecretPasswordKey: "password"
            url:
              protocol: https
              host: vpc-prod-camunda.us-west-2.es.amazonaws.com
              port: 443
            aws:
              enabled: false
          identity:
            auth:
              publicIssuerUrl: "https://camunda.prod-001.mycompany.io/auth/realms/camunda-platform"
              operate:
                redirectUrl: "https://camunda.prod-001.mycompany.io/operate"
              tasklist:
                redirectUrl: "https://camunda.prod-001.mycompany.io/tasklist"
              optimize:
                redirectUrl: "https://camunda.prod-001.mycompany.io/optimize"
              webModeler:
                redirectUrl: "https://camunda.prod-001.mycompany.io/modeler"
              console:
                redirectUrl: "https://camunda.prod-001.mycompany.io/console"
        elasticsearch:
          enabled: false

        console:
          image:
            repository: camunda/console
            tag: 8.6.55
          contextPath: "/console"
          enabled: true
          configuration: |
            camunda:
              console:
                oAuth:
                  audience: "console-api"
                  clientId: "console"
                  issuer: "https://camunda.prod-001.mycompany.io/auth/realms/camunda-platform"
                  jwksUri: "http://camunda-keycloak:80/auth/realms/camunda-platform/protocol/openid-connect/certs"
                  type: "KEYCLOAK"
                  wellKnown: "https://well-known-uri"
                managed:
                  method: plain
                  releases:
                  - name: camunda
                    namespace: camunda
                    version: 11.3.0
                    components:
                    - name: Console
                      id: console
                      version: 8.6.55
                      url: https://camunda.prod-001.mycompany.io/console
                      readiness: http://camunda-console.camunda:9100/health/readiness
                      metrics: http://camunda-console.camunda:9100/prometheus
                    - name: Keycloak
                      id: keycloak
                      version: 25.0.6
                      url: https://camunda.prod-001.mycompany.io/auth
                    - name: Identity
                      id: identity
                      version: 8.6.8
                      url: https://camunda.prod-001.mycompany.io/identity
                      readiness: http://camunda-identity.camunda:82/actuator/health
                      metrics: http://camunda-identity.camunda:82/actuator/prometheus
                    - name: Operate
                      id: operate
                      version: 8.6.9
                      url: https://camunda.prod-001.mycompany.io/operate
                      readiness: http://camunda-operate.camunda:9600/operate/actuator/health/readiness
                      metrics: http://camunda-operate.camunda:9600/operate/actuator/prometheus
                    - name: Optimize
                      id: optimize
                      version: 8.6.5
                      url: https://camunda.prod-001.mycompany.io/optimize
                      readiness: http://camunda-optimize.camunda:80/optimize/api/readyz
                      metrics: http://camunda-optimize.camunda:8092/actuator/prometheus
                    - name: Tasklist
                      id: tasklist
                      version: 8.6.9
                      url: https://camunda.prod-001.mycompany.io/tasklist
                      readiness: http://camunda-tasklist.camunda:9600/tasklist/actuator/health/readiness
                      metrics: http://camunda-tasklist.camunda:9600/tasklist/actuator/prometheus
                    - name: WebModeler WebApp
                      id: webModelerWebApp
                      version: 8.6.7
                      url: https://camunda.prod-001.mycompany.io/modeler
                      readiness: http://camunda-web-modeler-webapp.camunda:8071/health/readiness
                      metrics: http://camunda-web-modeler-webapp.camunda:8071/metrics
                    - name: Zeebe Gateway
                      id: zeebeGateway
                      version: 8.6.9
                      urls:
                        grpc: http://camunda-zeebe-gateway.camunda:26500
                        http: https://camunda.prod-001.mycompany.io/zeebe
                      readiness: http://camunda-zeebe-gateway.camunda:9600/zeebe/actuator/health/readiness
                      metrics: http://camunda-zeebe-gateway.camunda:9600/zeebe/actuator/prometheus
                    - name: Zeebe
                      id: zeebe
                      version: 8.6.9
                      readiness: http://camunda-zeebe.camunda:9600/actuator/health/readiness
                      metrics: http://camunda-zeebe.camunda:9600/actuator/prometheus

        connectors:
          image:
            repository: camunda/connectors-bundle
            tag: 8.6.8
          contextPath: "/connectors"

        operate:
          image:
            repository: camunda/operate
            tag: 8.6.9
          contextPath: "/operate"

        optimize:
          image:
            repository: camunda/optimize
            tag: 8.6.5
          contextPath: "/optimize"

        tasklist:
          image:
            repository: camunda/tasklist
            tag: 8.6.9
          contextPath: "/tasklist"

        webModeler:
          image:
            tag: 8.6.7
          enabled: true
          restapi:
            mail:
              fromAddress: noreply@camunda.prod-001.mycompany.io
              fromName: Camunda 8 WebModeler
            externalDatabase:
              url: "jdbc:postgresql://prod-camunda-aurpg.amazonaws.com:5432/modeler"
              user: "postgres"
              existingSecret: "camunda-web-modeler-restapi"
              existingSecretKey: "password"

        zeebe:
          image:
            repository: camunda/zeebe
            tag: 8.6.9

        zeebeGateway:
          image:
            repository: camunda/zeebe
            tag: 8.6.9
          contextPath: "/zeebe"

        identityKeycloak:
          auth:
            adminUser: admin
            existingSecret: "camunda-keycloak"
            existingSecretPasswordKey: "admin-password"
          image:
            repository: bitnami/keycloak
            tag: 25.0.6
          postgresql:
            enabled: false
            image:
              repository: bitnami/postgresql
              tag: 15.10.0-debian-12-r2
          externalDatabase:
            host: "prod-camunda-aurpg.amazonaws.com" 
            port: 5432
            user: "postgres"
            existingSecret: "camunda-externaldb"
            existingSecretKey: "password"
            database: "keycloak"
          extraEnvVars:
            - name: KEYCLOAK_PROXY_ADDRESS_FORWARDING
              value: "true"

@hamza-m-masood

[hamza-m-masood]

I will take care of this one. I would like this assigned to the ready column.
I haven't taken this out of the inbox yet because I want everyone in the grooming to see this ticket.

[hisImminence]

The Helm chart includes a constraint that prevents disabling the Identity pod (identity.enabled: false) if the identityKeycloak component (which manages the embedded Keycloak) is enabled (identityKeycloak.enabled: true).

What is the usecase here? We are wondering why you want to run Keycloak without Identity? (You can still install keycloak from the bitnami charts yourself if you want to run it as independently from Camunda)

[artmimois]

The Helm chart includes a constraint that prevents disabling the Identity pod (identity.enabled: false) if the identityKeycloak component (which manages the embedded Keycloak) is enabled (identityKeycloak.enabled: true).

What is the usecase here? We are wondering why you want to run Keycloak without Identity? (You can still install keycloak from the bitnami charts yourself if you want to run it as independently from Camunda)

Correct me if I'm wrong. From what I understand, the /identity endpoint is basically a wrapper/abstraction around Keycloak, because you can do the same thing as on Keycloak, such as creating clients, retrieving the client secrets, which can be used later, for example, in WebModeler. You can also assign roles and permissions, etc. and you use same username/password as for keycloak (/auth)

So taking that into account, if I disable just that identity endpoint and I'm still able to manage clients and users in Keycloak, that means that identity as a whole is not being disabled at all. It still functions. It's just we don't have that abstraction in face of /identity

The root of the problem is that the identity pod expects the Keycloak admin user to authenticate in a simple way, without additional security layers like OTP or SSO. When I enable these security measures for the Keycloak admin user like OTP or SSO via Google SAML 2.0 the identity pod can’t handle the more complex authentication flow, causing it to fail.

In other words, the identity pod doesn’t support these advanced authentication methods, and that’s why it breaks when you add that extra security to the Keycloak admin user. This is a key reason I want to disable the identity component and manage everything directly in Keycloak, so I can maintain stronger security for my admin user in master realm.

I use argocd for all this and I'm basically left with two choices which are either let identity pod in degraded state because it can't authenticate to Keycloak anymore or simply delete the identity deployment after initial installation using helm with consequential need to disable auto sync so that argocd doesn't heal it bringing identity pods back, I see both these as workarounds and sub-optimal measures

Would be great if this can be addressed!

[hamza-m-masood]

@theburi do we support the one time password feature for the embedded bitnami Keycloak?