[Bug] Kyuubi SparkSQLEngine hang in stopping EngineServiceDiscovery

[ruanwenjun]

Code of Conduct

• I agree to follow this project's Code of Conduct

Search before asking

• I have searched in the issues and found no similar issues.

Describe the bug

We use kyuubi to submit spark sql on k8s.
After the sql finished, the SparkSqlEngine pod doesn't deleted.
I find the below log

26/01/07 06:40:14 INFO SparkSQLSessionManager: Session stopped due to shared level is Connection.
26/01/07 06:40:14 INFO SparkSQLEngine: Service: [SparkTBinaryFrontend] is stopping.
26/01/07 06:40:14 INFO SparkTBinaryFrontendService: Service: [EngineServiceDiscovery] is stopping.

Seems the EngineServiceDiscovery doesn't stopped.

Below is the thread dump, seems the curator is retry

"SparkSQLSessionManager-timeout-checker: Thread-209" #209 daemon prio=5 os_prio=0 cpu=87.24ms elapsed=1089556.23s tid=0x00007f8a4a824000 nid=0x154 in Object.wait() [0x00007f8661926000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at org.apache.kyuubi.shaded.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1407)
        - locked <0x00000007637f5248> (a org.apache.kyuubi.shaded.zookeeper.ClientCnxn$Packet)
        at org.apache.kyuubi.shaded.zookeeper.ZooKeeper.delete(ZooKeeper.java:880)
        at org.apache.kyuubi.shaded.curator.framework.imps.DeleteBuilderImpl$5.call(DeleteBuilderImpl.java:250)
        at org.apache.kyuubi.shaded.curator.framework.imps.DeleteBuilderImpl$5.call(DeleteBuilderImpl.java:244)
        at org.apache.kyuubi.shaded.curator.RetryLoop.callWithRetry(RetryLoop.java:109)
        at org.apache.kyuubi.shaded.curator.framework.imps.DeleteBuilderImpl.pathInForeground(DeleteBuilderImpl.java:241)
        at org.apache.kyuubi.shaded.curator.framework.imps.DeleteBuilderImpl.forPath(DeleteBuilderImpl.java:225)
        at org.apache.kyuubi.shaded.curator.framework.imps.DeleteBuilderImpl.forPath(DeleteBuilderImpl.java:35)
        at org.apache.kyuubi.shaded.curator.framework.recipes.nodes.PersistentNode.deleteNode(PersistentNode.java:347)
        at org.apache.kyuubi.shaded.curator.framework.recipes.nodes.PersistentNode.close(PersistentNode.java:291)
        at org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient.deregisterService(ZookeeperDiscoveryClient.scala:247)
        at org.apache.kyuubi.ha.client.EngineServiceDiscovery.stop(EngineServiceDiscovery.scala:35)

thread.txt

Affects Version(s)

1.9.1

Kyuubi Server Log Output

Kyuubi Engine Log Output

26/01/07 06:40:14 INFO SparkSQLSessionManager: Session stopped due to shared level is Connection.
26/01/07 06:40:14 INFO SparkSQLEngine: Service: [SparkTBinaryFrontend] is stopping.
26/01/07 06:40:14 INFO SparkTBinaryFrontendService: Service: [EngineServiceDiscovery] is stopping.

Kyuubi Server Configurations

Kyuubi Engine Configurations

Additional context

No response

Are you willing to submit PR?

• Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
• No. I cannot submit a PR at this time.

[ruanwenjun]

Seems this is zk bug, I submit jira: https://issues.apache.org/jira/browse/ZOOKEEPER-5016
Maybe we can use an asynchronous thread to stop the EngineServiceDiscovery and set the maximum wait time to avoid the engine hang.

[pan3793]

cc @cxzl25

[pan3793]

by default, kyuubi uses a shaded zk 3.4 client which is very very old, if it's related to zk, does it have changes in new zk versions? if so, we can port such changes to our shaded zk 3.4.
(unfortunately, we still have some customers that uses zk server 3.4, so we'd like to keep using the old zk 3.4 client as much as possible to gain best compatibility)

[cxzl25]

This is a bit strange, can you provide the JDK version used and whether jemalloc is used?

[ruanwenjun]

@cxzl25 I use

openjdk version "1.8.0_322"

I don't use jemalloc

[ruanwenjun]

@pan3793 Thanks for the key information, I find zk has solved this at the post version
jira: ZOOKEEPER-2251
PR: apache/zookeeper#119
but has not fixed at 3.4.x 😢 , I find kyuubi has shade zk 36 https://github.com/apache/kyuubi-shaded/tree/master/kyuubi-relocated-zookeeper-parent/kyuubi-relocated-zookeeper-36, maybe we can change the kyuubi-relocated-zookeeper.artifacts version to kyuubi-relocated-zookeeper-36, but this is incompatible.

[cxzl25]

Judging from the call stack you provided, the main-SendThread seems to be missing. This may prevent the packet from ClientCnxn.submitRequest from being processed, ultimately resulting in a stuck wait state.

Is there thread-level OOM?

main-SendThread

        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.apache.kyuubi.shaded.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
        at org.apache.kyuubi.shaded.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)

[pan3793]

@ruanwenjun, you are allowed to build with zk-3.6 by passing -Pzookeeper-3.6, but as I said, I still prefer to keep using zk-3.4 by default (I apologize for being so extremely conservative here), if you like, you can follow the below PRs to port ZOOKEEPER-2251 (or other patches, as I lean towards agreeing with @cxzl25's analysis) to kyuubi's shaded zk-3.4

• [KYUUBI #6496] Step 1/2: Port ZOOKEEPER-4377 to Kyuubi shaded ZooKeeper 3.4 and 3.6 kyuubi-shaded#46
• [KYUUBI #6496] Step 2/2: Port ZOOKEEPER-4377 to Kyuubi shaded ZooKeeper 3.4 and 3.6 kyuubi-shaded#47

[ruanwenjun]

@pan3793 Ok, got it. I will take a look.

[ruanwenjun]

@cxzl25 Sorry, the send thread has exist due to OOM, OOM may be a common issue within the driver. I will port ZOOKEEPER-2251 to kyuubi-shaded

26/01/06 21:35:14 WARN ZooKeeperThread: Exception occurred from thread main-SendThread(172.28.16.195:2181)
java.lang.OutOfMemoryError: Java heap space