Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add VerticalPodAutoscaler to csi controller pod #2536

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

umagnus
Copy link
Contributor

@umagnus umagnus commented Sep 24, 2024

What type of PR is this?

/kind feature

What this PR does / why we need it:

feat: add VerticalPodAutoscaler to csi controller pod

Which issue(s) this PR fixes:

Fixes #

Requirements:

Special notes for your reviewer:

Release note:

none

@k8s-ci-robot k8s-ci-robot added kind/feature Categorizes issue or PR as related to a new feature. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Sep 24, 2024
@k8s-ci-robot k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Sep 24, 2024
@k8s-ci-robot
Copy link
Contributor

Hi @umagnus. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Sep 24, 2024
@andyzhangx andyzhangx removed the request for review from edreed September 25, 2024 06:39
@umagnus
Copy link
Contributor Author

umagnus commented Oct 21, 2024

AKS need to add a label for vpa admission controller webhook to impact csi controller pod in kube-system namespace, referenced to Can admission controller webhooks impact kube-system and internal AKS namespaces, wait for pr:kubernetes/autoscaler#7402 to completed

@voelzmo
Copy link

voelzmo commented Oct 22, 2024

Hey, I came here via the reference in the VPA PR.
I see that you're about to use VPA for all the containers in the CSI controller Pod. My understanding is that the resource usage in those containers is usually pretty low – but I don't know the details of the Azure CSI controller.

I'm not sure if you tried this for a longer period of time already in some beta env or have long-term experience with other components with many containers in a single pod, but keep in mind that this change will lead to many additional interruptions for the Pod, as VPA will evict it when the recommendation for a container changes by more than 10% (which happens quickly for these small absolute values). This also happens if e.g. your initial requests are too high and the containers will be scaled down step by step. I'm not sure if those disruptions could become an issue – but if resource usage isn't expected to change much, it may just be better to not have VPA enabled for this and have fewer disruptions.

@andyzhangx
Copy link
Member

@voelzmo thanks for the tips. we would like to use VPA mainly for CSI driver controller sidecar containers, the memory usage of those containers would grow dramatically when PV num grows in the cluster, current memory limit 500Mi is far from enough when there are a few thousand PVs in the cluster, in that case, the CSI driver would be OOM, that's the reason we want to use VPA.

@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

  • Mark this PR as fresh with /remove-lifecycle stale
  • Close this PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jan 20, 2025
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: umagnus
Once this PR has been reviewed and has the lgtm label, please assign andyzhangx for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Copy link
Member

@andyzhangx andyzhangx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/ok-to-test

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Mar 20, 2025
@andyzhangx andyzhangx removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Mar 21, 2025
repo="https://github.com/kubernetes/autoscaler.git"
echo "Uninstalling VPA ..."
git clone $repo
cd autoscaler/vertical-pod-autoscaler
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there an easy way to install upstream autoscaler without cloning the whole repo? and it's better installing a stable version instead of master branch

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, the doc only provides the installation to download the source code of VPA https://github.com/kubernetes/autoscaler/blob/master/vertical-pod-autoscaler/docs/installation.md#install-command

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: the script currently reads environment variables: $REGISTRY and $TAG. Make sure you leave them unset unless you want to use a non-default version of VPA.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we use 1.3.0 version in yaml file now

containerPolicies:
- containerName: '*'
maxAllowed:
memory: 50Gi
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

50G is crazy, 10Gi should be enough now

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@@ -0,0 +1,29 @@
#!/bin/bash
# Copyright 2020 The Kubernetes Authors.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2025

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@k8s-ci-robot k8s-ci-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Mar 21, 2025
@k8s-ci-robot
Copy link
Contributor

k8s-ci-robot commented Mar 21, 2025

@umagnus: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
pull-azuredisk-csi-driver-verify b030207 link true /test pull-azuredisk-csi-driver-verify

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/feature Categorizes issue or PR as related to a new feature. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants