Description
Code of Conduct
- I agree to follow this project's Code of Conduct
Search before asking
- I have searched in the issues and found no similar issues.
Describe the bug
Fault: KubenetesClientException - Unauthorized. http code=401
My kyuubi server was running on Azure Kubenetes Service(简写：AKS) , and we met this error in below pic.
Note: Kyuubi server worked well at the beginning, the error happened after 1 hour. AKS said that it was caused by the token expiration. We suspected that kyuubi won't auto reload the serviceAccount token in the pod.
Root Cause
The AKS serviceaccount token expires in 1 hour by default. Thus, the kyuubi will have an peaceful hour after starting, but after that, the token will expires, a continuous KubenetesClientException with 401 http code will be threw out.
Kubenetes client token refresh mechanism
Dive into the code of Kubenetes Client, we found that it has already integrated a TokenRefreshInterceptor to handle the 401 http status code issue, but it doesn't work at all in kyuubi. Here's why:
Kyuubi load the serviceAccount token by itself when initializing kube client, which is NOT RECOMMENDED. In fact, the kubeclient will auto configure the desired token, and take care of the token expiration time on its own.
Check out the TokenRefreshInterceptor code in Kubenetes client repo, we can see that the refreshToken method will skip refreshing token if you have specified a oauthToken by yourself.
Fix
We simply removed the code of setting oauthToken in the kyuubi, and it works very well now.
In addition, if you want to provide some flexible ways for the the authentication configuration, consider using tokenProvider instead of a token value. I would be very willing to commit a PR if you guys agree with my opinion, thanks.
Affects Version(s)
1.7.0~1.9.3
Kyuubi Server Log Output
No response
Kyuubi Engine Log Output
No response
Kyuubi Server Configurations
No response
Kyuubi Engine Configurations
No response
Additional context
No response
Are you willing to submit PR?
- Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
- No. I cannot submit a PR at this time.