[Bug] Socket Timeout When Connecting to Kyuubi Server

[nqvuong1998]

Code of Conduct

• I agree to follow this project's Code of Conduct

Search before asking

• I have searched in the issues and found no similar issues.

Describe the bug

When attempting to connect to the Kyuubi server, the connection intermittently fails due to a socket timeout issue. Logs indicate that the connection to the ZooKeeper server is lost and then re-established. However, this results in disruptions when attempting to interact with Kyuubi, causing instability in client connections.

Affects Version(s)

master

Kyuubi Server Log Output

[
  {
    "timeMillis": 1743312078141,
    "timestamp": "2025-03-30T05:21:18.141+0000",
    "thread": "main-SendThread(zookeeper.nprod-green-dev-zookeeper.svc.cluster.local:2181)",
    "threadId": 20,
    "threadPriority": 5,
    "level": "INFO",
    "loggerName": "org.apache.kyuubi.shaded.zookeeper.ClientCnxn",
    "message": "Unable to read additional data from server sessionid 0x2007639831c0000, likely server has closed socket, closing socket connection and attempting reconnect",
    "contextMap": {},
    "endOfBatch": false,
    "loggerFqcn": "org.apache.logging.slf4j.Log4jLogger"
  },
  {
    "timeMillis": 1743312080239,
    "timestamp": "2025-03-30T05:21:20.239+0000",
    "thread": "main-SendThread(zookeeper.nprod-green-dev-zookeeper.svc.cluster.local:2181)",
    "threadId": 20,
    "threadPriority": 5,
    "level": "INFO",
    "loggerName": "org.apache.kyuubi.shaded.zookeeper.ClientCnxn",
    "message": "Opening socket connection to server zookeeper.nprod-green-dev-zookeeper.svc.cluster.local/10.97.33.246:2181. Will not attempt to authenticate using SASL (unknown error)",
    "contextMap": {},
    "endOfBatch": false,
    "loggerFqcn": "org.apache.logging.slf4j.Log4jLogger"
  },
  {
    "timeMillis": 1743312080240,
    "timestamp": "2025-03-30T05:21:20.240+0000",
    "thread": "main-SendThread(zookeeper.nprod-green-dev-zookeeper.svc.cluster.local:2181)",
    "threadId": 20,
    "threadPriority": 5,
    "level": "INFO",
    "loggerName": "org.apache.kyuubi.shaded.zookeeper.ClientCnxn",
    "message": "Socket connection established to zookeeper.nprod-green-dev-zookeeper.svc.cluster.local/10.97.33.246:2181, initiating session",
    "contextMap": {},
    "endOfBatch": false,
    "loggerFqcn": "org.apache.logging.slf4j.Log4jLogger"
  },
  {
    "timeMillis": 1743312080252,
    "timestamp": "2025-03-30T05:21:20.252+0000",
    "thread": "main-EventThread",
    "threadId": 21,
    "threadPriority": 5,
    "level": "INFO",
    "loggerName": "org.apache.kyuubi.shaded.curator.framework.state.ConnectionStateManager",
    "message": "State change: RECONNECTED",
    "contextMap": {},
    "endOfBatch": false,
    "loggerFqcn": "org.apache.logging.slf4j.Log4jLogger"
  },
  {
    "timeMillis": 1743312080253,
    "timestamp": "2025-03-30T05:21:20.253+0000",
    "thread": "Curator-ConnectionStateManager-0",
    "threadId": 19,
    "threadPriority": 5,
    "level": "INFO",
    "loggerName": "org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient",
    "message": "Zookeeper client connection state changed to: RECONNECTED",
    "contextMap": {},
    "endOfBatch": false,
    "loggerFqcn": "org.apache.logging.slf4j.Log4jLogger"
  },
  {
    "timeMillis": 1743312080327,
    "timestamp": "2025-03-30T05:21:20.327+0000",
    "thread": "main-EventThread",
    "threadId": 21,
    "threadPriority": 5,
    "level": "WARN",
    "loggerName": "org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient",
    "message": "This Kyuubi instance 0.0.0.0:10009 now receives the NodeDataChanged event",
    "contextMap": {},
    "endOfBatch": false,
    "loggerFqcn": "org.apache.logging.slf4j.Log4jLogger"
  }
]

Kyuubi Engine Log Output

Kyuubi Server Configurations

Kyuubi Engine Configurations

Additional context

No response

Are you willing to submit PR?

• Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
• No. I cannot submit a PR at this time.

[nqvuong1998]

cc @wForget @turboFei

[pan3793]

Logs indicate that the connection to the ZooKeeper server is lost and then re-established. However, this results in disruptions when attempting to interact with Kyuubi, causing instability in client connections.

@nqvuong1998 I don't think the conclusion is correct. The client only needs to get the Kyuubi server endpoint info from Zk at the beginning, after that, the client establishes a connection to the Kyuubi server directly, Kyuubi server reconnecting Zk would NOT affect the established client-server connection.

If you observed bunches of connections between different components were lost at the same time, you should check the stability of your network instead of blaming some components have bugs.

[nqvuong1998]

Hi @pan3793 @turboFei , thanks for your answer.
I tried deploying a Kyuubi pod on Kubernetes and connecting it to the Zookeeper pods. However, I noticed that the connection to Zookeeper is unstable, and the Kyuubi logs indicate repeated reconnections.

As a result, I'm unable to connect to the Kyuubi service due to a socket timeout. The current workaround is to kill and restart the Kyuubi pod, which temporarily resolves the issue.

Would you recommend investigating and fixing the root cause of the issue instead of continuing with the pod restarts?

[turboFei]

I have not meet such kind of issue before.

For me, zookeeper is only used for kyuubi engine discovery.

For kyuubi server discovery, I use the loadbalancer(HLB or SLB) directly.