Creating KeycloakClient while Keycloak server is down #17
Description
We are creating Keycloak clients with keycloak-controller for existing Keycloak
server (7.0.0). For that to happen, first we're creating Keycloak object
pointing to the Keycloak server(url points to internal kubernetes service here,
which works great by the way):
apiVersion: k8s.kiwigrid.com/v1beta1
kind: Keycloak
metadata:
name: fully-configured-keycloak
namespace: keycloak
spec:
url: http://keycloak-http.keycloak.svc.cluster.local/auth
realm: master
clientId: admin-cli
username: keycloak
passwordSecretNamespace: keycloak
passwordSecretName: keycloak-http
passwordSecretKey: password
Output shows it is connected:
INFO com.kiwigrid.keycloak.controller.keycloak.KeycloakController Connected to fully-configured-keycloak in version 7.0.0.
While keycloak is availabe/ready, clients are created succefully and
available via keycloak UI. Example of KeycloakClient object:
apiVersion: k8s.kiwigrid.com/v1beta1
kind: KeycloakClient
metadata:
name: client-example
namespace: keycloak
spec:
keycloak: fully-configured-keycloak
realm: myrealm
clientId: client-example
clientType: confidential
directAccessGrantsEnabled: false
standardFlowEnabled: true
implicitFlowEnabled: false
secretNamespace: keycloak
secretName: clinet-example-client-secret
secretKey: secret
mapper:
- name: example-service-audience
protocolMapper: oidc-audience-mapper
config:
claim.name: audience
access.token.claim: "true"
id.token.claim: "true"
included.custom.audience: my-service
Error occurs when we are creating KeycloakClient, while keycloak is not
available. There may be several reasons for that e.g. keycloak pod restart
due to upgrade procedures either of keycloak itself or other pieces of
infrastructure.
The error itself is obvious - keycloak is not avilable during KeycloakClient
object creation:
09:00:35.689 16.0.1/... ERROR com.kiwigrid.keycloak.controller.client.ClientController Failed to ADDED resource keycloak/client-example.
javax.ws.rs.ProcessingException: javax.ws.rs.ServiceUnavailableException: HTTP 503 Service Unavailable
at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.filterRequest(ClientInvocation.java:599)
at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.invoke(ClientInvocation.java:436)
at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientInvoker.invokeSync(ClientInvoker.java:148)
at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientInvoker.invoke(ClientInvoker.java:112)
at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientProxy.invoke(ClientProxy.java:76)
at com.sun.proxy.$Proxy59.toRepresentation(Unknown Source)
at com.kiwigrid.keycloak.controller.client.ClientController.lambda$realm$2(ClientController.java:159)
at java.base/java.util.Optional.filter(Optional.java:223)
at com.kiwigrid.keycloak.controller.client.ClientController.realm(ClientController.java:157)
at com.kiwigrid.keycloak.controller.client.ClientController.apply(ClientController.java:46)
at com.kiwigrid.keycloak.controller.client.ClientController.apply(ClientController.java:26)
at com.kiwigrid.keycloak.controller.KubernetesController.eventReceived(KubernetesController.java:67)
at com.kiwigrid.keycloak.controller.KubernetesController.eventReceived(KubernetesController.java:14)
at io.fabric8.kubernetes.client.utils.WatcherToggle.eventReceived(WatcherToggle.java:49)
at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1.onMessage(WatchConnectionManager.java:232)
at okhttp3.internal.ws.RealWebSocket.onReadMessage(RealWebSocket.java:323)
at okhttp3.internal.ws.WebSocketReader.readMessageFrame(WebSocketReader.java:219)
at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:105)
at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:274)
at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:214)
at okhttp3.RealCall$AsyncCall.execute(RealCall.java:206)
at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: javax.ws.rs.ServiceUnavailableException: HTTP 503 Service Unavailable
at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.handleErrorStatus(ClientInvocation.java:231)
at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.extractResult(ClientInvocation.java:191)
at org.jboss.resteasy.client.jaxrs.internal.proxy.extractors.BodyEntityExtractor.extractEntity(BodyEntityExtractor.java:60)
at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientInvoker.invokeSync(ClientInvoker.java:150)
at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientInvoker.invoke(ClientInvoker.java:112)
at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientProxy.invoke(ClientProxy.java:76)
at com.sun.proxy.$Proxy47.refreshToken(Unknown Source)
at org.keycloak.admin.client.token.TokenManager.refreshToken(TokenManager.java:106)
at org.keycloak.admin.client.token.TokenManager.getAccessToken(TokenManager.java:71)
at org.keycloak.admin.client.token.TokenManager.getAccessTokenString(TokenManager.java:64)
at org.keycloak.admin.client.resource.BearerAuthFilter.filter(BearerAuthFilter.java:52)
at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.filterRequest(ClientInvocation.java:586)
... 24 common frames omitted
What we are missing here is a sort of resilience - while Keycloak is down,
KeycloakClient object is created by controller, but no actual client is created
in the Keycloak even when it is up again. The only fix we've found is either
to restart keycloak-controller or to delete/apply KeycloakClient again once
Keycloak server is ready.
Would be nice to have a feature when keycloak-controller tries to recover
connection to keycloak on attempt to create KeycloakClient, if keycloak is not
available at that moment. Same approach is used when controller tries to connect
to keycloak on start and logging WARN each 60s:
WARN com.kiwigrid.keycloak.controller.keycloak.KeycloakController Connecting to fully-configured-keycloak failed: Keycloak returned 503 with: no healthy upstream
What do you think about this / how you're solving this?