Pod Template defaults and SparkConfs being ignored

[ggaskins-slack]

What happened?

• ✋ I have searched the open/closed issues and my issue is not listed.

In v2.4.0, pod template file --conf flags were updated to always use a default Spec if user does not provide one in the submitted SparkApplication. This is a breaking change if users were not writing out the entire Driver and Executor specs within their SparkApplication YAML and relying on defaults provided in spark-defaults.conf mounted to SparkOperator pods. The new change will also override any user-provided spark --conf flags for the template variables.

Reproduction Code

1. Set spark.kubernetes.[Driver/Executor].podTemplateFile in spark-defaults.conf mounted to SparkOperator referencing a template mounted to SparkOperator pod or provide template flags as key/values within the sparkConf of the SparkApplication
2. Submit SparkApplication spec without app.Spec.[Driver/Executor].template

Expected behavior

SparkOperator should use default assigned within spark-defaults.conf (if it exists) or use user provided sparkConf values

Actual behavior

SparkOperator writes a blank Spec to a temporary file on SparkOperator and overwrites template vars to the tmp file location

Environment & Versions

• Kubernetes Version: 1.33.3
• Spark Operator Version: 2.4.0
• Apache Spark Version: 3.5.7

Additional context

No response

Impacted by this bug?

Give it a 👍 We prioritize the issues with most 👍

[aalopatin]

Does my feature suggestion explain your problem? #2795

[ChenYi015]

This breaking change was introduced by #2670, which will always add --conf=spark.kubernetes.[driver|executor].podTemplateFile to the spark-submit command even if .spec.[driver|executor].template is not specified.

Does my feature suggestion explain your problem? #2795

Adding --load-spark-defaults to spark-submit command will not fix this issue because arguments provided by spark operator with --conf will take precedence over default properties file.

[BhumikaTechHub]

Thanks for the detailed report and for clarifying the behavior change in v2.4.0.

From the description and the reference to PR #2670, it appears that the operator now always injects --conf=spark.kubernetes.[driver|executor].podTemplateFile into the spark-submit command, even when .spec.[driver|executor].template is not explicitly defined in the SparkApplication.

This effectively changes the precedence model that many users relied on previously. In setups where pod template defaults were defined via spark-defaults.conf mounted into the Spark Operator pod—or where users provided template-related values via sparkConf—those configurations are now overridden by operator-generated --conf arguments pointing to a temporary, blank spec. Since --conf flags passed directly to spark-submit take precedence over defaults files, this results in both default and user-provided template settings being ignored.

From a user perspective, this is a breaking change because it requires explicitly defining full driver and executor templates in every SparkApplication YAML, whereas earlier versions allowed users to rely on centrally managed defaults. This can be particularly impactful in environments with many applications or where template management is intentionally centralized.

[Denyme24]

hey @ChenYi015 @ggaskins-slack, i would like to work on this issue.

The operator always generates pod template files and passes them via --conf flags to spark-submit, even when spec.driver.template or spec.executor.template are not provided. Since --conf has higher precedence than sparkConf and spark-defaults.conf, this unintentionally overrides user-defined pod template configurations. This PR updates driverPodTemplateOption() and executorPodTemplateOption() in submission.go to only generate pod template arguments when templates are explicitly specified, restoring backward compatibility and ensuring user configuration precedence is respected.

[Denyme24]

/assign