[SPARK-56460][CORE] define configs in text files

[cloud-fan]

What changes were proposed in this pull request?

This PR adds a new module common/config which introduces a framework to define configs in prototext files. When the Spark application starts, we load config definitions from all prototext files and put them in a map with config name as the key, and the proto binary of the config definition as the value. These proto-backed configs are also registered in ConfigEntry.knownConfigs via a wrapper ProtoBackedConfigEntry, together with existing Spark configs.

The proto schema defines each config entry with: key, value type, default value, scope (cluster vs session), mutability (static vs dynamic, i.e. whether the config can be changed after system initialization), visibility (public vs internal), binding policy (session/persisted/not_applicable for SQL views, UDFs, and procedures), documentation, and version. Enum values follow proto3 naming conventions with type-name prefixes (e.g. SCOPE_CLUSTER, VALUE_TYPE_BOOL, VISIBILITY_PUBLIC, BINDING_POLICY_SESSION, MUTABILITY_STATIC) to avoid namespace collisions. The BindingPolicy field maps to the existing Scala ConfigBindingPolicy enum. The Mutability field is used by SQLConf.isStaticConfigKey to determine whether a config can be changed at runtime.

The new framework will co-exist with the existing config framework, until we migrate all existing configs to this new framework.

For simple configs that are only accessed in one place, we can hardcode the config name in the place that accesses it, with a new API SQLConf#getConfByKeyStrict to avoid typo. Example in this PR: spark.sql.optimizer.datasourceV2ExprFolding

For configs that are accessed in multiple places and we want to avoid hardcoding config name, or configs that need custom validation, we can still have an entry in object SQLConf to reference the config definition. Examples in this PR: spark.sql.optimizer.maxIterations and spark.sql.shuffledHashJoinFactor.

Why are the changes needed?

Defining configs in various Scala objects is a bad practice:

• Configs are registered when the Scala objects are loaded. To list all configs we must know all these Scala objects and load them.
• It's hard to audit configs as they spread all over the codebase.
• We will hit JVM limitation one day sooner or later, as defining configs in a Scala object is basically doing heavy work in the constructor, which has limitation of 64 kb bytecode size.

Does this PR introduce any user-facing change?

No, it's developer facing

How was this patch tested?

new tests

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code

[szehon-ho]

Interesting, curious the reason for prototext?

[cloud-fan]

@szehon-ho The reason for prototext:

• It generates the java classes for the config definition, saves my time
• We need to keep all config definitions in memory, and protobuf should be more memory-efficient
• The proto schema file of the config definition is a good place for documentation

[pan3793]

Do you mean APPLICATION scope? I think in the context of Spark, "cluster" usually refers to something like a K8s cluster, a Spark standalone cluster, a YARN cluster, etc.

[cloud-fan]

The physical cluster is managed by resource manager and is not within the context of Spark configs. I choose CLUSTER because it's a more general word for distributed engine and it refers to the driver + executors cluster launched for the application.

[pan3793]

My major concern is APPLICATION is a clear concept and has no ambiguity in the context of Spark, for example, there are configs like spark.app.id, spark.app.name, while CLUSTER may introduce additional ambiguity, for example, in the Spark Standalone docs, "cluster" does mean the "Spark Standalone cluster", using CLUSTER introduces unnecessary confusion.

[cloud-fan]

I'll think about it more. My concern is that, with many managed Spark services out there, most users do not have the concept of application. They either submit the spark application as scheduled jobs, or use notebook directly. With Spark Connect, the concept of Spark application becomes even less obvious. Sessions and the Spark cluster under the hood are more general concepts now IMO.

[gengliangwang]

QQ: how to support createWithDefault, enumConf, fallbackConf in this textproto file?

For example

  val STORE_ASSIGNMENT_POLICY =
    buildConf("spark.sql.storeAssignmentPolicy")
     ...
      .enumConf(StoreAssignmentPolicy)
      .createWithDefault(StoreAssignmentPolicy.ANSI)

  val ANSI_ENABLED = buildConf(SqlApiConfHelper.ANSI_ENABLED_KEY)
    ...
    .createWithDefault(!sys.env.get("SPARK_ANSI_SQL_MODE").contains("false"))

[cloud-fan]

See https://github.com/apache/spark/pull/53488/files#diff-82e9b261f86d3cbd94a23abe94aadf45531e0b29603082336784e30ea46c9574R108-R116

We can override certain fields of the config definition at the Scala side, and default value can be supported as well.

For enum conf, one benefit is that it's type safe: the getConf API returns the enum instance directly. This means we need a ConfigEntry at the Scala side, but we can still define the enum in protobuf, and expose the protobuf enum.

[mihailotim-db]

Nice idea. I've recently encountered a number of issues with config discovery in Spark. Most notably, when configs are in objects that are for some reason not loaded (i.e. Hive confs in sql/core) or object is loaded but conf is defined as lazy val. Will the latter also be supported?

[mridulm]

Very promising !
A few high level thoughts:

• Is this specific only to sql ? current formulation appears to assume yes. If yes:
  • Should it be under sql/config instead ?
  • If no (preferable), handling for core, common/* modules would be something to consider.
    • cluster vs session could get impacted by this as well (how to specify/differentiate from common/core configs).
• Since we are relooking at configs - one 'feature' to consider would be enforcement of cluster defaults, which users cannot override (like security configs, event file location, etc).
• It is not clear what the usecase for test_default is.

[cloud-fan]

Hi @mridulm

• This is not for SQL config only. I mentioned this in the README.md: Currently, SQL configs go in sql.textproto. New categories can have their own files.
• As of today, I don't think users can override spark core configs or static sql configs via public APIs. So this "feature" is already there.
• Quite some boolean configs define their default values as Utils.isTesting. To avoid having to add Scala entries (e.g. the ones in SQLConf) for them, test_default is added to handle such use cases with the new text file based framework.

[mridulm]

• This is not for SQL config only. I mentioned this in the README.md: Currently, SQL configs go in sql.textproto. New categories can have their own files.

This is great !

• As of today, I don't think users can override spark core configs or static sql configs via public APIs. So this "feature" is already there.

Users can override spark core configs - both at time of submission, and before session creation (via the SparkConf passed into SparkContext).

• Quite some boolean configs define their default values as Utils.isTesting. To avoid having to add Scala entries (e.g. the ones in SQLConf) for them, test_default is added to handle such use cases with the new text file based framework.

Sounds good !

[cloud-fan]

yea people can set spark core configs before SparkContext is instantiated, maybe I misunderstood the word "override" in your previous comment. Providing configs before creating SparkContext doesn't sound like "override". Maybe you mean override the config property files?

[mridulm]

To put it differently - ability to enforce deployer configs, which users cannot override: for example, for security: similar to final configuration in hadoop

[cloud-fan]

@mridulm maybe we can extend the config visibility: for now it's only PUBLIC and INTERNAL. Maybe we can add one more: PRIVATE. People can only set PRIVATE configs in the config file and it can't be overridden with other config files or SparkContext/SparkSession config APIs. Is it what you need?

[cloud-fan]

@mridulm Thanks for the suggestion! I think the Hadoop final config feature is about specifying a config value (the deployer marks a specific value as final in the config file), not about defining a config (the config author decides at schema level whether it can be locked). So it's orthogonal to what this PR does — this PR focuses on config definitions/schema.

Supporting Hadoop-style final configs for Spark would be a separate effort, likely involving changes to SparkConf/SQLConf to honor a "finalized keys" set and reject overrides from runtime APIs (SET, spark.conf.set(), etc.). Happy to discuss that in a separate JIRA if you're interested!