[TASK][CHALLENGE] Support Spark Connect Frontend/Backend

[ulysses-you]

Code of Conduct

• I agree to follow this project's Code of Conduct

Search before creating

• I have searched in the task list and found no similar tasks.

Mentor

• I have sufficient knowledge and experience of this task, and I volunteer to be the mentor of this task to guide contributors to complete the task.

Skill requirements

• Knowledge about Spark Connect
• Knowledge about Kyuubi architecture
• Knowledge about protobuf
• Knowledge about grpc
• Knowledge about thrift

Background and Goals

Make Kyuubi server compatible with Spark Connect protocol, so that people can use Spark Connect client to connect to Kyuubi Server.

Implementation steps

1. Add a new Spark Connect frontend
   1.1 Add basic gRpc server as frontend
   1.2 Compatible with Spark Connect protocol, see https://github.com/apache/spark/blob/master/connector/connect/common/src/main/protobuf/spark/connect/base.proto
   1.3 Support ExecutePlan
   1.4 Support AnalyzePlan
   1.5 Support Config
   1.6 Support AddArtifacts
   1.7 Support ArtifactsStatus
   1.8 Support Interrupt
   1.9 Support ReattachExecute
   1.10 Support ReleaseExecute
   1.11 Serialize the protobuf based request

2. Add a new Spark Connect backend
   2.1 Imprort Sprak-Connect-Server and rewrite SparkConnectService https://github.com/apache/spark/blob/master/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectServer.scala
   2.2 Deserialize response to protobuf based

3. Add IT

4. Add docs

Additional context

Introduction of #6232

[yehere]

I think this is very challenging, but I want to give it a try. Can you assign it to me and help me

[ulysses-you]

sure, thank you @yehere ! This is a kind of umbrella, we can create sub issue one by one later.

[pan3793]

This huge task could be divided into several different level tasks, feel free to go ahead ~ all your contributions will be counted eventually :)

[cfmcgrady]

cc @cfmcgrady

[zhaomin1423]

sure, thank you @yehere ! This is a kind of umbrella, we can create sub issue one by one later.

I'm also interested in it, hope to work together.

[ulysses-you]

thank you @zhaomin1423 , glad to see you are interested in.

[minyk]

how about co-located mode with kyuubi's sparksql engine? separated service is good and basic, but also needs more resources for more spark instances.

[ulysses-you]

@minyk there are in different process, just like Spark thirftserver and connect server. We are going to add a new module and new server for Kyuubi connect. We can do it together if you are interested in.

[davidyuan1223]

@ulysses-you hello,i'm interested with this component,hope work with you

[davidyuan1223]

@yaooqinn @pan3793 @ulysses-you
i found spark have package the connect module to maven repository
https://mvnrepository.com/artifact/org.apache.spark/spark-connect-client-jvm_2.13/3.4.0
can we use thoes package to simplify the code?
maybe we could provide a connectingstr to sparkSession like the code description
https://github.com/apache/spark/blob/master/connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/SparkSession.scala
based spark-connect package, we can reduce grpcServer and proto
what do you think?

[pan3793]

I haven't had deep look at it, my current thought is,

1. for server part, we only need a thin gRPC layer, coping proto files and regenerating gRPC files is fine.
2. for engine part, we can reuse the connect-server module to simplify the code.