ci.jenkins.io is not allocating Kubernetes agents

[MarkEWaite]

Service(s)

ci.jenkins.io

Summary

The Kubernetes agents on ci.jenkins.io are not being allocated

Reproduction steps

1. Check the pending jobs in the queue and note that many are requesting maven-21 or maven-21-bom]
2. The system log includes messages like:

Error in provisioning; agent=org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave[jnlp-maven-21-ps1jl], template=bb4d3e6b-4e0c-477a-a9c2-c8c9afbeed9a
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://EFD41748F9C38AAB83FA81F34C59B313.gr7.us-east-2.eks.amazonaws.com/api/v1/namespaces/jenkins-agents/pods/jnlp-maven-21-ps1jl. Message: Unauthorized. Received status: Status(apiVersion=v1, code=401, details=null, kind=Status, message=Unauthorized, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Unauthorized, status=Failure, additionalProperties={}).
	at PluginClassLoader for kubernetes-client-api//io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:642)
	at PluginClassLoader for kubernetes-client-api//io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:622)
	at PluginClassLoader for kubernetes-client-api//io.fabric8.kubernetes.client.dsl.internal.OperationSupport.assertResponseCode(OperationSupport.java:582)
	at PluginClassLoader for kubernetes-client-api//io.fabric8.kubernetes.client.dsl.internal.OperationSupport.lambda$handleResponse$0(OperationSupport.java:549)
	at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(Unknown Source)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
	at java.base/java.util.concurrent.CompletableFuture.complete(Unknown Source)
	at PluginClassLoader for kubernetes-client-api//io.fabric8.kubernetes.client.http.StandardHttpClient.lambda$completeOrCancel$10(StandardHttpClient.java:141)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
	at java.base/java.util.concurrent.CompletableFuture.complete(Unknown Source)
	at PluginClassLoader for kubernetes-client-api//io.fabric8.kubernetes.client.http.ByteArrayBodyHandler.onBodyDone(ByteArrayBodyHandler.java:51)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
	at java.base/java.util.concurrent.CompletableFuture.complete(Unknown Source)
	at PluginClassLoader for kubernetes-client-api//io.fabric8.kubernetes.client.okhttp.OkHttpClientImpl$OkHttpAsyncBody.doConsume(OkHttpClientImpl.java:134)
Caused: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://EFD41748F9C38AAB83FA81F34C59B313.gr7.us-east-2.eks.amazonaws.com/api/v1/namespaces/jenkins-agents/pods/jnlp-maven-21-ps1jl. Message: Unauthorized. Received status: Status(apiVersion=v1, code=401, details=null, kind=Status, message=Unauthorized, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Unauthorized, status=Failure, additionalProperties={}).
	at PluginClassLoader for kubernetes-client-api//io.fabric8.kubernetes.client.KubernetesClientException.copyAsCause(KubernetesClientException.java:205)
	at PluginClassLoader for kubernetes-client-api//io.fabric8.kubernetes.client.dsl.internal.OperationSupport.waitForResult(OperationSupport.java:507)
	at PluginClassLoader for kubernetes-client-api//io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:524)
	at PluginClassLoader for kubernetes-client-api//io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleGet(OperationSupport.java:467)
	at PluginClassLoader for kubernetes-client-api//io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handleGet(BaseOperation.java:792)
	at PluginClassLoader for kubernetes-client-api//io.fabric8.kubernetes.client.dsl.internal.BaseOperation.requireFromServer(BaseOperation.java:193)
	at PluginClassLoader for kubernetes-client-api//io.fabric8.kubernetes.client.dsl.internal.BaseOperation.get(BaseOperation.java:149)
	at PluginClassLoader for kubernetes-client-api//io.fabric8.kubernetes.client.dsl.internal.BaseOperation.get(BaseOperation.java:98)
	at PluginClassLoader for kubernetes//org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher.launch(KubernetesLauncher.java:160)
	at hudson.slaves.SlaveComputer.lambda$_connect$0(SlaveComputer.java:298)
	at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
	at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:80)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)

As an attempt to resolve the issue, I restarted the controller. I doubt it will resolve the issue, but it didn't seem that it would do much harm to try.

[MarkEWaite]

I restarted the controller and the agents were allocated without any issue.

[dduportal]

Status(apiVersion=v1, code=401

That is... unexpected. Looks like a temporary failure in EC2 IMDBs system (we use credential-less authentication on ci.jenkins.io for acquiring connection to EKS API).