VPA: Add memory and cpu graphing to benchmark

[maxcao13]

Which component are you using?:

/area vertical-pod-autoscaler

Is your feature request designed to solve a problem? If so describe the problem this feature should solve.:

The benchmark currently only captures updater step latency at a single point in time. There is no visibility into how VPA component resource consumption behaves over the duration of the run.

I want the benchmark to output a graph of memory usage (and optionally CPU) over the run time of the benchmark. This would make it easy to spot memory leaks, unexpected spikes, or growing baseline consumption.

Describe the solution you'd like.:

Periodically sample VPA component memory (and optionally CPU) usage throughout the benchmark run and output a time-series graph alongside the existing latency results.

Related

Followup to #8866

[maxcao13]

/assign

[iamzili]

Based on the description of this issue, my understanding is that you want to capture the average CPU (and memory) usage for a single VPA component (e.g. the updater) per profile (e.g. --profile=medium), store the results in a time-series database, and then visualize the data points. In other words each VPA component per profile would produce a single data point.

Is my understanding correct?

[maxcao13]

Thanks for the question!

Yes your understanding is correct, in my mind each profile for each component should output its own time-series graph from however way we choose to implement this. Like for example, we would expose prometheus metrics on each component, and sample the cpu and memory prometheus with a client from the benchmark, keep this in memory or a small db, and the visualize it using some library (python, go, etc.). And then output the graphs as artifacts.

Ideally, then we can see how VPAs and pods affect the memory and cpu usage of the components and try to prevent regressions as much as possible by comparing them, for example shape of the graph, peaks, averages, etc.

Something I didn't mention was using pprof for profiling, but I think that is optional and can come as a future enhancement since pprof gives more detailed analysis of the overall program and the call stack, heap, etc., while what I'm talking about is a more general view on resource usage over time that VPA end users might care about, rather than devs. Using this would hopefully allow us to also be able to give more general feedback to users on how to tune the VPA (e.g., kube-api-qps/burst, memory/cpu request/limits, etc.) based on their cluster.

[omerap12]

/triage accepter

[k8s-ci-robot]

@omerap12: The label(s) triage/accepter cannot be applied, because the repository doesn't have them.

Details

In response to this:

/triage accepter

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[omerap12]

/triage accpeted

[k8s-ci-robot]

@omerap12: The label(s) triage/accpeted cannot be applied, because the repository doesn't have them.

Details

In response to this:

/triage accpeted

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[omerap12]

/triage accepted