Using Kamaji together with Vertical Pod Autoscaler (VPA)

[joseluisgonzalezca]

Hey!

I'm considering using Kamaji together with an external Vertical Pod Autoscaler (p.e. https://github.com/kubernetes/autoscaler/tree/master/vertical-pod-autoscaler)

The idea is to let the VPA automatically adjust the CPU and memory requests of the control plane components based on their actual resource usage over time. My question is related to how this external element could "disturb" the Kamaji controller behaviour. Since Kamaji manages the Deployment object for the TCP components:

• What happens when the VPA updates the resource requests of the Deployment?
• Will the Kamaji controller reconcile the Deployment and restore the original resource values defined in the TenantControlPlane, effectively causing a reconciliation loop? Or does Kamaji intentionally leave those fields untouched once the Deployment has been created, allowing the VPA to manage them without conflicts?

Has anyone tried this approach, or is there a recommended way to combine Kamaji with a VPA/HPA solution?

Thank you for your time!

[prometherion]

Hola Jose! I was working on something similar but with HPA via #366: by using a nil pointer on Deployment replicas, Kamaji doesn't set any value of the TenantControlPlane deployment; thus, the HPA will manually update the replicas field, without having Kamaji battle to update it.

However, VPA is working on resource limits, and unfortunately we're hardcoding nil values and enforcing them:

kamaji/internal/builders/controlplane/deployment.go

Line 520 in 08f2c55

podSpec.Containers[index].Resources = corev1.ResourceRequirements{}

We can give it a try by removing those default values, but keep in mind we're just working at the PodSpec containers level: could this be enough for you in implementing a VPA?

[joseluisgonzalezca]

Hi Dario!

I've tested the VPA scenario by installing the controller and creating a VerticalPodAutoscaler resource pointing to a certain TCP. AFAIK, the VPA should detect the pods without issues because the CRD supports the scale subresource. This is my testing configuration file:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: test-vpa
  namespace: sare0000
spec:
  targetRef:
    apiVersion: "kamaji.clastix.io/v1alpha1"
    kind: TenantControlPlane
    name: ptf1234567
  updatePolicy:
    updateMode: "Recreate"  # Off, Initial, Recreate, InPlaceOrRecreate, InPlace

These are the recommedations and eviction events generated by the controller:

Status:
  Conditions:
    Last Transition Time:  2026-07-30T06:31:25Z
    Status:                True
    Type:                  RecommendationProvided
  Observed Generation:     2
  Recommendation:
    Container Recommendations:
      Container Name:  konnectivity-server
      Lower Bound:
        Cpu:     9m
        Memory:  50Mi
      Target:
        Cpu:     11m
        Memory:  50Mi
      Uncapped Target:
        Cpu:     11m
        Memory:  50Mi
      Upper Bound:
        Cpu:           619m
        Memory:        1328430748
      Container Name:  kube-apiserver
      Lower Bound:
        Cpu:     20m
        Memory:  244272298
      Target:
        Cpu:     23m
        Memory:  297164212
      Uncapped Target:
        Cpu:     23m
        Memory:  297164212
      Upper Bound:
        Cpu:           1972m
        Memory:        16744946349
      Container Name:  kube-controller-manager
      Lower Bound:
        Cpu:     9m
        Memory:  50Mi
      Target:
        Cpu:     11m
        Memory:  63544758
      Uncapped Target:
        Cpu:     11m
        Memory:  63544758
      Upper Bound:
        Cpu:           619m
        Memory:        3580692157
      Container Name:  kube-scheduler
      Lower Bound:
        Cpu:     9m
        Memory:  50Mi
      Target:
        Cpu:     11m
        Memory:  50Mi
      Uncapped Target:
        Cpu:     11m
        Memory:  50Mi
      Upper Bound:
        Cpu:           619m
        Memory:        2042867346
Events:
  Type    Reason      Age   From         Message
  ----    ------      ----  ----         -------
  Normal  EvictedPod  12m   vpa-updater  VPA Updater evicted Pod ptf1234567-847c97c985-8d8q4 to apply resource recommendation.
  Normal  EvictedPod  11m   vpa-updater  VPA Updater evicted Pod ptf1234567-847c97c985-7z5ql to apply resource recommendation.
  Normal  EvictedPod  10m   vpa-updater  VPA Updater evicted Pod ptf1234567-847c97c985-s8s2k to apply resource recommendation.

PodSpecs were updated according to the changes requested by the VPA. Even after applying those changes, we haven't observed any issues or reconciliation loops with the Kamaji controller (I've left the service running for approximately one hour to verify that there are no repeated reconciliation cycles and that the values are not being overwritten).

Based on these results, I think we can confirm that the VPA solution works as expected without requiring any changes to the CRDs spec or the reconciliation logic.

[prometherion]

Thanks for the updated and testing it, although I'm a bit confused: is VPA updating Pod Resources or individual Container Resources?

Nevermind, I read it carefully!

Would you be up for writing some documentation this integration? The community will benefit for sure!

[joseluisgonzalezca]

Hey Dario!

Sorry for that. Maybe I was not concise enough with my previous comment. The resources are updated for the pods at the containers level. See for example:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    cni.projectcalico.org/containerID: 623f7a80f36d31ac9ace8c0578403f322c64c6a8c2fdcba9d923c93e203a3601
    cni.projectcalico.org/podIP: 10.42.225.35/32
    cni.projectcalico.org/podIPs: 10.42.225.35/32
    storage.kamaji.clastix.io/certificate: f0bf61d58b93b2d7e9a7db172dba5f23
    storage.kamaji.clastix.io/config: 266db4cc00e6a5ae5ed828d1b7dbef43
    vpaObservedContainers: kube-apiserver, kube-scheduler,
      kube-controller-manager, konnectivity-server
    vpaUpdates: 'Pod resources updated by test-vpa: container 0: cpu request, memory
      request, cpu limit, memory limit; container 1: cpu request, memory request,
      cpu limit, memory limit; container 2: cpu request, memory request, cpu limit,
      memory limit; container 3: cpu request, memory request, cpu limit, memory limit;
      container 4: memory request, cpu request'
  creationTimestamp: "2026-07-30T06:34:25Z"
  generateName: ptf1234567-847c97c985-
  ** TRUNCATED **
  name: ptf1234567-847c97c985-9htj9
  namespace: sare0000
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: ptf1234567-847c97c985
    uid: ed616e1c-72ea-44ec-8d20-527545c8e487
  resourceVersion: "304658906"
  uid: 58b40d6f-9949-493f-83bd-e8e5f7ba261c
spec:
  containers:
  - args:
     ** TRUNCATED **
    resources:
      limits:
        cpu: 280m    <<<<<<< ----- THOSE VALUE HAS BEEN MODIFIED BY VPA
        memory: "2377313696"
      requests:
        cpu: 35m
        memory: "297164212"
** TRUNCATED **

I will happy to provide some documentation about how make VPA work with Kamaji. Give me some time and I will make you a PR.