Which spark configs are in Engine's attributes?

[HaoYang670]

I set many spark configs when creating a Kyuubi Session to create a Spark cluster.
But when I use the ListEngine REST request, I can only find spark.executor.memory and spark.driver.memory in Engine.attributes. Are other Spark configs filtered out?

[HaoYang670]

cc @pan3793 Could you please take a look if you have time? Thanks a lot.

[pan3793]

Yes, we don't want to expose too many engine attributes to the discovery service, unless it truly has common requirements.

The original requirement here is to let Kyuubi server support Web UI proxy, it's a common requirement in Spark on K8s.

And it's allowed to configure additional attributes by

  val ENGINE_SPARK_REGISTER_ATTRIBUTES: ConfigEntry[Seq[String]] =
    buildConf("kyuubi.engine.spark.register.attributes")
      .internal
      .doc("The extra attributes to expose when registering for Spark engine.")
      .version("1.8.0")
      .stringConf
      .toSequence()
      .createWithDefault(Seq("spark.driver.memory", "spark.executor.memory"))

[HaoYang670]

Thanks @pan3793. Would you mind telling where the related code is?
Maybe I need to do some custom changes to adapt my team's requirement.

[pan3793]

Please read the SparkTBinaryFrontendService#attributes

[HaoYang670]

Thanks a lot.
I guess the config "kyuubi.engine.spark.register.attributes" can solve my problem.