[Bug] Intermittent OperationHandle 404 when fetching results under concurrent load

[BohanZhang0222]

Code of Conduct

• I agree to follow this project's Code of Conduct

Search before asking

• I have searched in the issues and found no similar issues.

Describe the bug

I am performing a concurrency stress test on Kyuubi and encountered intermittent 404 errors when fetching operation results.

The issue appears to occur when calling get result row set for an operation that has already transitioned to CLOSED_STATE, even though the client did not explicitly close the operation.

Test Setup
Goal

Validate the maximum concurrent Operation capacity supported by 4 Kyuubi nodes under the same environment configuration.

Load Testing Method

Using Locust to simulate client behavior with the following workflow:

Create operation

Poll operation status

Fetch operation result

Repeat continuously

Test Parameters

Concurrency: 60

Duration: 10 minutes continuous execution

Observed Behavior

Intermittent failures when fetching results:

Error getting result row set for operation handle <operation_id>
Example error:

org.apache.kyuubi.KyuubiSQLException: Invalid OperationHandle [a0f5cbc3-08a4-4a54-8e38-e944174fa940]
This corresponds to a 404-like scenario where the operation handle is no longer valid.

Affects Version(s)

1.9.3

Kyuubi Server Log Output

From Kyuubi server logs, the lifecycle of a failed operation is as follows:
2026-03-20 17:36:33.524 INFO  OperationLog - Creating operation log file ...

2026-03-20 17:42:53.342 INFO  ExecuteStatement - PENDING_STATE -> RUNNING_STATE

2026-03-20 17:42:54.572 INFO  ExecuteStatement - RUNNING_STATE -> FINISHED_STATE

2026-03-20 17:44:42.814 INFO  ExecuteStatement - Processing zhou's query[a0f5cbc3-08a4-4a54-8e38-e944174fa940]: FINISHED_STATE -> CLOSED_STATE, time taken: 109.472 seconds

2026-03-20 17:44:42.814 ERROR ApiUtils - Error getting result row set for operation handle a0f5cbc3-08a4-4a54-8e38-e944174fa940
org.apache.kyuubi.KyuubiSQLException: Invalid OperationHandle [a0f5cbc3-08a4-4a54-8e38-e944174fa940]
        at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:69)
        at org.apache.kyuubi.operation.OperationManager.getOperation(OperationManager.scala:106)
        at org.apache.kyuubi.server.BackendServiceMetric.$anonfun$fetchResults$1(BackendServiceMetric.scala:211)
        at org.apache.kyuubi.metrics.MetricsSystem$.timerTracing(MetricsSystem.scala:112)
        at org.apache.kyuubi.server.BackendServiceMetric.fetchResults(BackendServiceMetric.scala:186)
        at org.apache.kyuubi.server.BackendServiceMetric.fetchResults$(BackendServiceMetric.scala:181)
        at org.apache.kyuubi.server.KyuubiServer$$anon$1.fetchResults(KyuubiServer.scala:179)
        at org.apache.kyuubi.server.api.v1.OperationsResource.getNextRowSet(OperationsResource.scala:186)

Additionally, I noticed that before the error occurs, there are usually logs indicating that the ZooKeeper session was closed. I’m not sure whether this is related to the issue.

Also, in the Kyuubi server logs, there are a large number of entries showing frequent connections being established to ZooKeeper.

Kyuubi Engine Log Output

Kyuubi Server Configurations

Kyuubi Engine Configurations

Additional context

Not all operations are affected。

Are you willing to submit PR?

• Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
• No. I cannot submit a PR at this time.

[pan3793]

Using Locust to simulate client behavior with the following workflow:

Create operation

Poll operation status

Fetch operation result

Which exactly APIs? General descriptions do not help understand problems

2026-03-20 17:44:42.814 INFO  ExecuteStatement - Processing zhou's query[a0f5cbc3-08a4-4a54-8e38-e944174fa940]: FINISHED_STATE -> CLOSED_STATE, time taken: 109.472 seconds

this is clear that the op was closed explicitly, so the question is where the signal comes from, two suggestions:

• follow the latest conf/log4j2.xml.template to enable restAudit logs, then check if it is closed by REST API request.
• tune the code of ExecuteStatement where prints the above logs, print the callsite

your screenshot indicates you are using batch operation? this does not match what you described above (I assume you are using interactive sessions)

[BohanZhang0222]

Using Locust to simulate client behavior with the following workflow:
Create operation
Poll operation status
Fetch operation result

Which exactly APIs? General descriptions do not help understand problems

your screenshot indicates you are using batch operation? this does not match what you described above (I assume you are using interactive sessions)

1. POST /sessions
2. POST /sessions/${sessionHandle}/operations/statement
3. GET /operations/${operationHandle}/event
4. Multiple GET requests to /operations/{operationHandle}/rowset

I am indeed using operations, not batch.
From the monitoring screenshot, you can see that interactive queries are being submitted, while the number of batch submissions is 0.