How to create bustable memory configuration for executors

[viniciuserrero]

What question do you want to ask?

Hello Spark Operator Team 👋🏻

We are exploring the memoryLimits feature (available after v2.2.0/#2478) and are very interested in its potential for creating burstable memory configurations.
Our goal is to have our executor pods request a smaller amount of memory while being able to burst up to a higher limit, which would help us run more jobs in our shared, memory-constrained environment.

We've been experimenting with this new field, but have run into a scenario that we're hoping you could clarify for us:

• We set the executor's memory field to define the pod's memory request.
• We set the memoryLimits field to a higher value to define the pod's memory limit.

This successfully creates an executor pod with the desired requests and limits in Kubernetes. However, the Spark executor's JVM doesn't seem to be aware of the higher limit, and its heap size is still based on the lower memory setting.

Our next step was to try and make Spark aware of the extra memory. We attempted to override the spark.executor.memory property during runtime to a value that would utilize the memory up to the memoryLimits.

When we do this, we observe that the executor pod's memory request is raised to match the value of spark.executor.memory, which defeats our original goal of having a lower request.

Our Questions:

• Could you please clarify the intended workflow for using memoryLimits to make burstable memory available to the executor's JVM?
• Is there a specific configuration or best practice we are missing that allows the Spark JVM to utilize the memory between the request and limit?
• Is the behavior where spark.executor.memory defines the pod's memory request (even when memoryLimits is set) the intended design? If so, could you help us understand the rationale or point us toward an alternative approach?

We guess we might be misunderstanding how these parameters are meant to interact.
Thank you!

Additional context

No response

Have the same question?

Give it a 👍 We prioritize the question with most 👍

[MardanovTimur]

@viniciuserrero Take a look #2647

We also tried to implement this, but Spark always eats a lot according to the JVM heap size. You can try to use a dynamic heap size, but I'm afraid there will be drawdowns when allocating additional memory, and the allocation of additional memory will still occur and will occur instantly. I think the game with memoryLimit is not worth the candle. We use Volcano manager and k8s resource-quota to make a bottleneck, cause we are not alone in the Kubernetes cluster and we havent much more memory, I think it is enough to simply control resource consumption (try to reduce them and see how this will affect performance).

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically closed because it has not had recent activity. Please comment "/reopen" to reopen it.

[ChenYi015]

/reopen
/lifecycle frozen

[google-oss-prow]

@ChenYi015: Reopened this issue.

Details

In response to this:

/reopen
/lifecycle frozen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.