Add a configurable plugin for submitting Spark applications via gRPC

[ChenYi015]

What feature you would like to be added?

Add a configurable plugin for submitting Spark applications via gRPC instead of running ${SPARK_HOME}/bin/spark-submit directly.

Why is this needed?

Improve the controller performance when reconciling a large amount of new SparkApplications.

Describe the solution you would like

• Add a spark_submit.proto grpc service definition and use protoc to generate protobuf code.
• Add a new class to implement the SparkApplicationSubmitter interface:

  spark-operator/internal/controller/sparkapplication/submission.go

  Lines 39 to 42 in f53373e

  	// SparkApplicationSubmitter is the interface for submitting a SparkApplication.
  	type SparkApplicationSubmitter interface {
  	Submit(ctx context.Context, app *v1beta2.SparkApplication) error
  	}

Describe alternatives you have considered

• Use golang to reimplement the spark-submit logic.

Additional context

Related issues: #2337 .
Related PRs: #2500 #2629 .
Related docs: https://docs.google.com/document/d/1lICOCFWIpTEHbwKaz45EGG01hH-7gC8nQUWXC4qT1-I/edit?tab=t.0#heading=h.q5r3jnr2ss51

Love this feature?

Give it a 👍 We prioritize the features with most 👍

[rahul810050]

Hi @ChenYi015 I have opened a PR implementing this feature.
It adds the gRPC-based Spark submitter, the spark_submit.proto definition, generated protobuf stubs, and the new SparkApplicationSubmitter implementation as requested.

PR: #2748
Please take a look when you get a chance — happy to make any adjustments if needed!

Thanks!