Open
Description
Code of Conduct
- I agree to follow this project's Code of Conduct
Search before asking
- I have searched in the issues and found no similar issues.
Describe the bug
The Kyuubi server fails to release the activeSession, preventing the server from shutting down properly.
Details:
We’re utilizing the hot upgrade feature, but noticed that the process doesn’t terminate as expected.
Upon examining the logs and source code, we observed that although ZooKeeper has already released the znode, the activeSessionCount remains greater than zero.
This issue is easily reproducible. Simply run zkCli.sh -server [host] while the server is processing a basic query like show databases;.
We believe this might be a bug on the Kyuubi server side. Could you kindly look into this?
Thank you.
Affects Version(s)
1.9.2
Kyuubi Server Log Output
2025-04-08 13:28:53.336 KyuubiSessionManager-exec-pool: Thread-113 INFO org.apache.kyuubi.operation.ExecuteStatement: Processing jpz40462's query[cc906dab-b30a-482b-8f08-a132ee073d7b]: PENDING_STATE -> RUNNING_STATE, statement:
show databases
2025-04-08 13:28:53.521 KyuubiSessionManager-exec-pool: Thread-113 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[cc906dab-b30a-482b-8f08-a132ee073d7b] in FINISHED_STATE
2025-04-08 13:28:53.522 KyuubiSessionManager-exec-pool: Thread-113 INFO org.apache.kyuubi.operation.ExecuteStatement: Processing jpz40462's query[cc906dab-b30a-482b-8f08-a132ee073d7b]: RUNNING_STATE -> FINISHED_STATE, time taken: 0.184 seconds
2025-04-08 13:28:53.795 KyuubiTBinaryFrontendHandler-Pool: Thread-51 INFO org.apache.kyuubi.operation.ExecuteStatement: Processing jpz40462's query[cc906dab-b30a-482b-8f08-a132ee073d7b]: FINISHED_STATE -> CLOSED_STATE, time taken: 0.458 seconds
2025-04-08 13:28:53.824 KyuubiTBinaryFrontendHandler-Pool: Thread-51 INFO org.apache.kyuubi.client.KyuubiSyncThriftClient: TCloseOperationReq(operationHandle:TOperationHandle(operationId:THandleIdentifier(guid:CC 90 6D AB B3 0A 48 2B 8F 08 A1 32 EE 07 3D 7B, secret:C2 EE 5B 97 3E A0 41 FC AC 16 9B D7 08 ED 8F 38), operationType:EXECUTE_STATEMENT, hasResultSet:true)) succeed on engine side
2025-04-08 13:28:53.845 KyuubiTBinaryFrontendHandler-Pool: Thread-51 INFO org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Received request of closing SessionHandle [6e8bce61-610e-4e78-8e7e-1f64e5f0fc0d]
2025-04-08 13:28:53.847 KyuubiTBinaryFrontendHandler-Pool: Thread-51 INFO org.apache.kyuubi.session.KyuubiSessionManager: jpz40462's KyuubiSessionImpl with SessionHandle [6e8bce61-610e-4e78-8e7e-1f64e5f0fc0d] is closed, current opening sessions 0
2025-04-08 13:28:53.848 KyuubiTBinaryFrontendHandler-Pool: Thread-51 INFO org.apache.kyuubi.operation.LaunchEngine: Processing jpz40462's query[dc7a1cb6-3924-4f88-94de-bc94a77f509c]: FINISHED_STATE -> CLOSED_STATE, time taken: 28.092 seconds
2025-04-08 13:28:53.918 KyuubiTBinaryFrontendHandler-Pool: Thread-51 INFO org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Finished closing SessionHandle [6e8bce61-610e-4e78-8e7e-1f64e5f0fc0d]
...
2025-04-08 13:32:10.856 main-EventThread INFO org.apache.kyuubi.ha.client.KyuubiServiceDiscovery: 1 connection(s) are active, delay shutdown
2025-04-08 13:32:20.857 main-EventThread INFO org.apache.kyuubi.ha.client.KyuubiServiceDiscovery: 1 connection(s) are active, delay shutdown
2025-04-08 13:32:30.857 main-EventThread INFO org.apache.kyuubi.ha.client.KyuubiServiceDiscovery: 1 connection(s) are active, delay shutdown
...
2025-04-08 14:18:40.969 main-EventThread INFO org.apache.kyuubi.ha.client.KyuubiServiceDiscovery: 1 connection(s) are active, delay shutdown