Controller loses API connection after token expiry on Azure Kubernetes Service (AKS)

[waqas-anonymco]

dask-operator sometimes fails to create Dask Jobs on Azure Kubernetes Service (AKS) 1.33.5:

Related issues: #913

It looks like issue is not completely fixed. We have seen two occurrences of this issues in last two weeks.

Also, see nolar/kopf#980 (comment) report for details.

Logs:

[2025-12-01 22:20:46,800] kr8s._auth           [DEBUG   ] Reloading credentials
...
[2025-12-01 22:20:46,822] httpcore.http11      [DEBUG   ] send_request_headers.started request=<Request [b'GET']>
[2025-12-01 22:20:46,822] httpcore.http11      [DEBUG   ] send_request_headers.complete
...
[2025-12-01 22:20:46,901] httpx                [INFO    ] HTTP Request: GET https://10.0.0.1/apis/kubernetes.dask.org/v1/namespaces/join/daskclusters/job-d90c9c3c-0ae2-48aa-ba00-c80da3bce657 "HTTP/1.1 401 Unauthorized"

Minimal Complete Verifiable Example:

Install dask-operator on AKS 1.33.5
Wait an hour for the authentication token to expire.
Create a DaskJob resource.

dask-operator will not create the DaskJob because dask-operator's kubernetes authentication token has expired and kopf's watchers are no longer connected to kubeapi. A bug in kopf prevents kopf from refreshing the authentication token.

This only occurs on AKS 1.30+ because that is AKS now sets --service-account-extend-token-expiration to false.

Environment:
Dask operator version: 2025.7.0

[jacobtomlinson]

We use two libraries together for Kubernetes communication, kopf and kr8s. They both handle authentication independently.

You only shared a small slice of the logs but it looks like it might be kr8s failing to reauthenticate, not kopf. Can you share the full logs output, it might give a better clue.

[waqas-anonymco]

Thanks for looking into this. Here are some more logs:

[2025-12-01 22:20:46,775] kopf.objects         [DEBUG   ] [join/job-d90c9c3c-0ae2-48aa-ba00-c80da3bce657] Updating diff: (('add', ('status',), None, {'jobStatus': 'JobCreated'}),)
[2025-12-01 22:20:46,776] kopf.objects         [DEBUG   ] [join/job-d90c9c3c-0ae2-48aa-ba00-c80da3bce657] Handler 'daskjob_create_components/status.jobStatus' is invoked.
[2025-12-01 22:20:46,778] kopf.objects         [INFO    ] [join/job-d90c9c3c-0ae2-48aa-ba00-c80da3bce657] Creating Dask job components.
[2025-12-01 22:20:46,800] kr8s._auth           [DEBUG   ] Reloading credentials
[2025-12-01 22:20:46,801] kr8s._auth           [DEBUG   ] Loading kubeconfig
[2025-12-01 22:20:46,803] kr8s._auth           [DEBUG   ] Loading service account
[2025-12-01 22:20:46,805] kr8s._auth           [DEBUG   ] Found new token
[2025-12-01 22:20:46,807] httpcore.connection  [DEBUG   ] close.started
[2025-12-01 22:20:46,808] httpcore.connection  [DEBUG   ] close.complete
[2025-12-01 22:20:46,808] httpcore.connection  [DEBUG   ] connect_tcp.started host='10.0.0.1' port=443 local_address=None timeout=None socket_options=None
[2025-12-01 22:20:46,811] httpcore.connection  [DEBUG   ] connect_tcp.complete return_value=<httpcore._backends.anyio.AnyIOStream object at 0x7fbd1dfa4100>
[2025-12-01 22:20:46,811] httpcore.connection  [DEBUG   ] start_tls.started ssl_context=<ssl.SSLContext object at 0x7fbd15d5aac0> server_hostname='10.0.0.1' timeout=None
[2025-12-01 22:20:46,821] httpcore.connection  [DEBUG   ] start_tls.complete return_value=<httpcore._backends.anyio.AnyIOStream object at 0x7fbd1dfa7cd0>
[2025-12-01 22:20:46,822] httpcore.http11      [DEBUG   ] send_request_headers.started request=<Request [b'GET']>
[2025-12-01 22:20:46,822] httpcore.http11      [DEBUG   ] send_request_headers.complete
[2025-12-01 22:20:46,822] httpcore.http11      [DEBUG   ] send_request_body.started request=<Request [b'GET']>
[2025-12-01 22:20:46,823] httpcore.http11      [DEBUG   ] send_request_body.complete
[2025-12-01 22:20:46,823] httpcore.http11      [DEBUG   ] receive_response_headers.started request=<Request [b'GET']>
[2025-12-01 22:20:46,900] httpcore.http11      [DEBUG   ] receive_response_headers.complete return_value=(b'HTTP/1.1', 401, b'Unauthorized', [(b'Audit-Id', b'c1a40f21-485e-4c4f-9aeb-b24b216864e3'), (b'Cache-Control', b'no-cache, private'), (b'Content-Type', b'application/json'), (b'Date', b'Mon, 01 Dec 2025 22:20:46 GMT'), (b'Content-Length', b'129')])
[2025-12-01 22:20:46,901] httpx                [INFO    ] HTTP Request: GET https://10.0.0.1/apis/kubernetes.dask.org/v1/namespaces/join/daskclusters/job-d90c9c3c-0ae2-48aa-ba00-c80da3bce657 "HTTP/1.1 401 Unauthorized"

[jacobtomlinson]

Thanks for sharing the logs. This looks very similar to the problem described in kr8s-org/kr8s#536.

The workaround there was to avoid defining kr8s objects at the module level because it somehow blocked the session from refreshing.

I'm surprised kopfs watchers also disconnect. As they do auth separately to kr8s this suggests the same bug exists in kubernetes or pykube (kopf can use either and it depends on your config).