Repository navigation
Document CI cluster selection, CPU : RAM ratio / machine types, and general recommendations specific to prow.k8s.io #34139
Description
What would you like to be added:
We don't have a single place to point to regarding which cluster: you should use and why, and how much resources to use (and how to avoid pointlessly scheduling minuscule amounts of memory per CPU core, which ultimately costs us more when workloads prefer more CPU time to allocating and memory sits unused).
We should do this per-cluster and create a doc somewhere discoverable, perhaps under config/jobs.
We should also consider adding details like:
- kubekins / CI image recommendations
- docker in docker
- Additional pointers to the hacks we have employed in the clusters (like pre-allocating loop devices, tuning sysctls ...).
Why is this needed:
So contributors can understand the Kubernetes specific CI environment and how to effectively schedule to it / write prow.k8s.io specific jobs.
/sig testing k8s-infra
@kubernetes/sig-k8s-infra-leads @kubernetes/sig-testing-leads
These are really not discoverable:
Along with "what is the trusted cluster" etc.
We should also deprecate out the eks-job-migration doc and associated job report results, and we should consider how to balance scheduling to EKS/GKE more generally now that the budgets are similar and all the workloads are running in community accounts. (And also how to approach Azure with the much smaller budget ...)
Activity
BenTheElder commentedon Jan 15, 2025
We should also link out to https://monitoring-eks.prow.k8s.io/?orgId=1 https://monitoring-gke.prow.k8s.io/?orgId=1 for checking actual resource usage
BenTheElder commentedon Feb 26, 2025
@kubernetes/sig-k8s-infra-leads how far are we from being able to tell CI users about the CI cluster machine shapes?
BenTheElder commentedon Feb 26, 2025
I know we wanted to reconsider the EKS machine types, and also on GCP it's not clear if highmem actually makes sense with our current workloads, though any changes would have to be done carefully to avoid breaking jobs that already implicitly depend on the machine sizes.
ameukam commentedon Feb 26, 2025
I think once we are done with instance type selection. We can also communicate what we already have and later update the docs once the instances are switched.
BenTheElder commentedon Mar 26, 2025
@xmudrii @upodroid ?
This is coming up all the time, we should decide and either commit to documenting the EKS cluster as-is, or move on changing it.
It's really difficult to answer these questions about resources available, which cluster to use, etc. currently.
xmudrii commentedon Mar 26, 2025
I'll push on that right after KubeCon, too busy at the moment. 🙃
upodroid commentedon Mar 30, 2025
I recommend the following:
We introduce the following labels to pick the size of the pod for the community users:
prow.k8s.io/machine-size: small- 2 cores 8 GB of memoryprow.k8s.io/machine-size: medium- 4 cores and 16GB of memoryprow.k8s.io/machine-size: large- 7 cores and 32GB of memoryIf the pod requires node isolation, we can also add this label:
prow.k8s.io/dedicated: true|falseWe can use kyverno to mutate pods by:
https://github.com/kubernetes/k8s.io/blob/main/kubernetes/ibm-ppc64le/prow/kyverno.yaml
typical CI machines sizes
https://docs.gitlab.com/ci/runners/hosted_runners/linux/#machine-types-available-for-linux---x86-64
https://cloud.google.com/build/pricing
https://circleci.com/pricing/price-list/
The instance types now become something internal to k8s infra and we can pick the following instances:
GKE
EKS
dims commentedon Mar 30, 2025
@upodroid seems reasonable to me! thanks
BenTheElder commentedon Mar 31, 2025
I don't generally recommend depending on mutating admission webhooks ... that part gives me pause
BenTheElder commentedon Jul 23, 2025
Probably best to follow up on this when 1.35 opens up, or else just document the current state.
The longer we don't change the clusters the more we might as well document them as-is.
xmudrii commentedon Jul 23, 2025
At this time it's the best to wait for 1.35, by then prepare the changes, and then apply them shortly after the Code Thaw.
5 remaining items
alpha/beta race detection: increase memory limits
main (#6)
k8s-triage-robot commentedon Jan 19, 2026
The Kubernetes project currently lacks enough contributors to adequately respond to all issues.
This bot triages un-triaged issues according to the following rules:
lifecycle/staleis appliedlifecycle/stalewas applied,lifecycle/rottenis appliedlifecycle/rottenwas applied, the issue is closedYou can:
/remove-lifecycle stale/closePlease send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
pohly commentedon Jan 20, 2026
/remove-lifecycle stale
Still relevant. This issue is what I look for when I need the Grafana dashboard URLs - they really should be in some documentation.
BenTheElder commentedon Jan 20, 2026
#36121 is the plan of record for the moment -- start enforcing a unified maximum workload size => align the machine shapes and then document everything with guidance.
BenTheElder commentedon Jan 20, 2026
/cc @kubernetes/sig-k8s-infra-leads @kubernetes/sig-testing-leads
k8s-triage-robot commentedon Apr 20, 2026
The Kubernetes project currently lacks enough contributors to adequately respond to all issues.
This bot triages un-triaged issues according to the following rules:
lifecycle/staleis appliedlifecycle/stalewas applied,lifecycle/rottenis appliedlifecycle/rottenwas applied, the issue is closedYou can:
/remove-lifecycle stale/closePlease send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
pohly commentedon Apr 21, 2026
/remove-lifecycle stale
BenTheElder commentedon Apr 21, 2026
/lifecycle frozen