[QUESTION] How to disable spark.properties generation?

[meatheadmike]

I can't seem to find a way to disable the built-in /opt/spark/conf/spark.properties. Generating config via configmaps is not really a viable option for me since I'm trying to build a single spark image that can be used in multiple scenarios (kubeflow being one of them).

Kubeflow seems to stomp all over the /opt/spark/conf path. So I've built a container that keeps config in /opt/spark/config instead and set the appropriate env variable (SPARK_CONF_DIR). But that seems to be ignored because "--properties-file /opt/spark/conf/spark.properties" is injected during job submission. Is there a way to override or remove that option?

Thanks.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically closed because it has not had recent activity. Please comment "/reopen" to reopen it.

[ChenYi015]

/reopen
/lifecycle frozen

[google-oss-prow]

@ChenYi015: Reopened this issue.

Details

In response to this:

/reopen
/lifecycle frozen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.