Skip to content

[FLINK-34574] Add CPU and memory size autoscaler quota #789

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Apr 19, 2024

Conversation

gaborgsomogyi
Copy link
Contributor

What is the purpose of the change

Flink can report slot sharing group information in the vertex information. This can be used to fine tune the autoscaler algorithm to define quotas. In this PR I've added 2 type of quotas:

  • kubernetes.operator.job.autoscaler.quota.cpu: Task manager CPU quota can be defined
  • kubernetes.operator.job.autoscaler.quota.memory: Task manager memory size quota can be defined

In bot cases scaling is not happening when the total amount of CPU or memory would be greater than the quota.

Brief change log

Added CPU and memory size autoscaler quota.

Verifying this change

Added automated tests + manually on cluster.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): no
  • The public API, i.e., is any changes to the CustomResourceDescriptors: no
  • Core observer or reconciler logic that is regularly executed: no

Documentation

  • Does this pull request introduce a new feature? no
  • If yes, how is the feature documented? not applicable

@gaborgsomogyi gaborgsomogyi force-pushed the quota branch 2 times, most recently from 18e158d to ea9fc78 Compare March 4, 2024 14:59
@gaborgsomogyi gaborgsomogyi changed the title Add CPU and memory size autoscaler quota [FLINK-34574] Add CPU and memory size autoscaler quota Mar 6, 2024
@gaborgsomogyi gaborgsomogyi force-pushed the quota branch 4 times, most recently from 605105e to 176d2bf Compare April 12, 2024 09:31
@gaborgsomogyi
Copy link
Contributor Author

@mxm can you plz have a second look? We've talked it through with @gyfora and the changes are reflecting it.

@gyfora gyfora merged commit baad900 into apache:main Apr 19, 2024
131 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants