Add memory benchmark CI workflow

[doxxx93]

Closes #1505

Motivation

Ref #1505 — maintainer requested automatic detection of memory profile regressions in PRs. The benchmark code (kube-runtime/benches/memory.rs) was already implemented; this PR adds CI integration to run it automatically and report results.

Solution

• kube-runtime/benches/memory.rs: Output customSmallerIsBetter JSON directly (via serde::Serialize), eliminating the need for shell-based output parsing in CI.
• .github/workflows/memory-bench.yml: New workflow triggered on pull_request that:
  1. Runs cargo bench -p kube-runtime --bench memory
  2. Feeds JSON output to benchmark-action/github-action-benchmark@v1
  3. Comments on PR when memory regression exceeds 150% of baseline

Benchmark covers 5 scenarios (init_listwatch, steady_state, relist, init_without_modify, init_with_modify) × 3 metrics (peak_bytes, total_allocated, alloc_count) = 15 data points per run.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 76.2%. Comparing base (0eef838) to head (bbe5c03).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##            main   #1937     +/-   ##
=======================================
+ Coverage   76.2%   76.2%   +0.1%     
=======================================
  Files         89      89             
  Lines       8487    8487             
=======================================
+ Hits        6464    6466      +2     
+ Misses      2023    2021      -2     

see 4 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[doxxx93]

@clux I took a stab at the full CI integration for the memory benchmarks from #1505. Would love your thoughts on a few things:

What's implemented:

• memory.rs outputs customSmallerIsBetter JSON directly (no shell parsing needed)
• github-action-benchmark compares against baseline, alerts + fails CI on >150% regression
• Baseline stored via actions/cache (no gh-pages branch needed)
• Baseline updates on push to main, PRs compare against it
• Workflow only runs when kube-runtime/kube-client/kube-core/Cargo.toml/Cargo.lock change

Feedback I'd appreciate:

1. dhat → CI glue: dhat's own JSON output (dhat-heap.json) is for its viewer and doesn't match what github-action-benchmark expects ([{name, unit, value}]). So there's a BenchMetric struct with serde in the benchmark code to bridge the gap. dhat does have dhat::assert! for CI, but that's hardcoded thresholds — no relative comparison against previous runs. Is this approach acceptable, or would you prefer something different?

2. Stability/noise: dhat is fairly deterministic (allocation-count based), but minor variations across CI environments are possible. Is 150% the right alert threshold, or should it be tighter/looser?

3. Baseline storage: Currently using actions/cache — simple but the baseline resets if the cache gets evicted. The alternative is a gh-pages branch (persistent but adds a branch to the repo). Any preference?

4. Scenario coverage: Currently 5 scenarios (init_listwatch, steady_state, relist, init_without_modify, init_with_modify) × 3 metrics (peak_bytes, total_allocated, alloc_count). Are there additional scenarios worth covering? Is 10k objects the right scale?

[clux]

hey. awesome work. finally gotten to have a look at this. i think the general setup looks sensible.

i'll put some nits on the actual code, but here's some quick comments on your points:

1. I think this makes sense yeah. I would personally really like to see something like the warning comments in the action.
2. If the thresholdsusage is deterministic, we should be able to get away with thresholds around 10% I would have thought. At any rate, it's fine to start sensitive and tune later if it's too noisy.
3. The kube-rs cache in the org is already close to many github thresholds so the a permanent branch for this makes sense. Happy to set up some branch protection rules to restrict who can push into it / roles if needed. It needs to not run on forks at the very leastbe careful if it's writing into branches though.
4. Not sure about this, but i think it's a reasonable starting point. Prod clusters I engage with often has less than that, but i know people have some monster clusters. I think a good number will depend on the time it takes in CI and how tight it makes the threshold. Ideally it's somewhat tight.

from org/actions/cache;

[doxxx93]

Changes after review

• Switched PR baseline from actions/cache to gh-pages branch (both main and PR now use gh-pages)
• Added permissions block (contents: write, pull-requests: write) — without this, alert comments fail with Resource not accessible by integration
• Created gh-pages branch on this repo

Validation

Tested the full pipeline in doxxx93/memory-bench-test (minimal dhat benchmark, same workflow structure):

Scenario	PR	Result
3x memory regression	#3	CI failed + alert comment
No regression	#4	CI passed

https://doxxx93.github.io/memory-bench-test/bench/ you can watch the benchmark page

Questions

• Is gh-pages fine as the branch name for baseline storage, or would you prefer something else (e.g. benchmark-data)?
• benchmark-data-dir-path is currently bench. Any preference on the path (e.g. memory-bench, dev/bench)?

[clux]

I think both values are good personally.

• gh-pages; people know what to expect if they see it
• benches could have other benches down the line. you've already named the bench memory so

This all looks very sensible to me.

The only thing that stood out to me is how blindingly white the page is, but looks like that looks non-trivial to customize. I see they have some issues on ui redesign and dark mode also so maybe it gets more visually pleasing eventually.

Anyway, happy to merge if you are happy with it!

[doxxx93]

Yeah, the page is quite bright 😄 I think we can gradually improve the UI over time — I'm happy to merge now currently :)

[doxxx93]

By the way — I do have merge permissions, but I've been holding off on using them so far. For example, I approved #1939 but left the merge to you. Would it be okay for me to start merging PRs like these myself, or would you prefer to keep handling that?

[clux]

would you prefer to keep handling that?

nonono, of course go ahead. you have access, and i'd appreciate it. it's usually easier to keep main rolling forwards rather than waiting. especially on trivial things like that that that passes all the checks.

[doxxx93]

@clux The benchmark data is being pushed to gh-pages branch successfully, but GitHub Pages isn't enabled on the repo yet, so https://kube-rs.github.io/kube/bench/ returns 404. Could you enable GitHub Pages in repo Settings (Source: Deploy from branch, Branch: gh-pages / / (root))?

---

hmm, I think it's not necessary.

[clux]

nice. i've added those options for now.

these links seems to work:

• readme on https://kube-rs.github.io/kube/
• bench on https://kube-rs.github.io/kube/bench/

[doxxx93]

looks great! Looking forward to seeing the trends build up with more commits.

Aggregating the 3 metrics per scenario (instead of 15 separate charts) could make it easier to scan, also other ui things :) let's fix it later

[clux]

yeah, agreed. but at least the data is there. the big value is probably going to be the warnings on exceeding thresholds.

we'll see how difficult it is to backtrack to future regressions (guessing each dot will represent a squash merge into main), we'll probably mostly want to look at the details here when the comment fires to figure out what exactly broken/when.

[doxxx93]

The first PR (#1942) has been merged since this was set up, and the benchmark dashboard at https://kube-rs.github.io/kube/bench/ is working as expected — two data points are now showing with stable memory metrics across all 15 benchmarks. Looks good! 🎉