Resource limit / control

[fnetX]

Clear and concise description of the problem

Currently, woodpecker does not limit resource power (often "CI minutes") per user. This can lead to

• resource waste through stuck builds (if people don't manually cancel)
• resource abuse by malicious users / PRs
• resource waste because people don't consider optimizing their pipelines

Suggested solution

Implement

• setting a maximum global resource limit for builds (e.g. 20 minutes)
• allowing a user-definable limit per build (so a user who expects the pipeline to run at most 3 minutes can limit at 5, to avoid using up unnecessary resources if something is stuck)
• tracking the resource usage per user and / or project
• a user and admin dashboard showing the use
• capping the resource usage when reaching a daily / weekly / monthly / yearly limit (don't start new builds) ← ideally configurable for each of the timeframes, but monthly could be a good start
• prominently calculate the trees burned to produce that much energy 😉

Alternative

In #345 there is already the idea to display resource usage, but manual moderation is often not feasible.

Additional context

No response

Validations

• Read the Contributing Guidelines.
• Read the docs.
• Check that there isn't already an issue that request the same feature to avoid creating a duplicate.

[6543]

the info is already in DB
-> #345

[qwerty287]

As for tree calculation 😆 🌳, I just found https://www.epa.gov/energy/greenhouse-gas-equivalencies-calculator which can calculate something like this. They have a calculation reference at https://www.epa.gov/energy/greenhouse-gases-equivalencies-calculator-calculations-and-references#seedlings. Not sure if this can help because it just calculates CO2, but there's a converting calculation on the same page (https://www.epa.gov/energy/greenhouse-gases-equivalencies-calculator-calculations-and-references#kilowatt) to convert kWh to the CO2 emission.

[maltegrosse]

I run into the issue by using Woodpecker on Kubernetes with Docker in Docker (dind). Resource Limits defined by k8s are ignored by the dockerd. After playing around with cgroups I could somehow get it working adding the existing pod resource limits (cgroup) to the docker deamon:

          command: ["dockerd"]
          args: ["--cgroup-parent=$(cat /proc/self/cgroup | cut -d: -f3)"]

(or in the /etc/docker/daemon.json with "runtimeArgs": [], see docker for more arguments)

the cat command returns the cgroup path for that pod which can be found in /sys/fs/cgroup/*:

/kubepods/pod{pod_uid}/{container_runtimeId}

Perhaps someone with cgroups experience has an idea how we can clone the kubepod child cgroup into a new parent group? (@6543 ? :-)

For non-k8s users (assume @fnetX ) , they can create their own cgroups limits either on the host system using systemd or self defined cgroups in the dind with cgcreate. More information on docker.com ...

P.S. I am new to the cgroups topic and I am sure there is a better/proper way to achieve this. As this solution also causes multiple issues on the docker deamon by inspecting the logs - but somehow it works, even the resources are a bit over allocated by +10%. But as the resource limit of the agent as enough cpu/mem spare, it shouldnt be a problem at all. The downside is that dockerd does not report the cpu/mem usage to k8s - thats why enabling the metrics on the dockerd should be considered to monitor the workloads.

P.P.S anyone with GPU (NVIDIA) needs, can give my docker image a try on github or dockerhub: maltegrosse/dind-nvidia:c_11.8.0-d_20.10.22. K8s resources need the nvidia gpu tag, e.g. nvidia.com/mig-2g.20gb: 1 and the pipeline needs for the gpu step: environment: - NVIDIA_VISIBLE_DEVICES=all

[maltegrosse]

I did some more testing and playing around with cgroups. I finally came up with entrypoint.sh following script

#!/bin/sh
cgroup_path="/sys/fs/cgroup/"
new_group_path="$cgroup_path"DIND
echo "setting cgroups"
mkdir -p $new_group_path
if [ -z "${KUBERNETES_SERVICE_HOST}" ]; then
  echo "non-k8s"
  if [ -z "${CPU_MAX}" ] || [ -z "${MEMORY_MAX}"]; then
    echo "env variable missing, skip setting resources"
  else
    echo "$CPU_MAX" > "$new_group_path"/cpu.max
    echo "$MEMORY_MAX" > "$new_group_path"/memory.max
  fi
else
  echo "k8s"
  cgroup_k8s_path=$(cat /proc/self/cgroup | cut -d: -f3)
  full_path="$cgroup_path$cgroup_k8s_path"
  echo $(cat "$full_path"/cpu.max)> "$new_group_path"/cpu.max
  echo $(cat "$full_path"/memory.max)> "$new_group_path"/memory.max
fi
. ./dockerd-entrypoint.sh

It either maps the resource limits from k8s, or two env variable can be set: CPU_MAX & M̀EMORY_MAX (in cgroup format)

keep in mind that running two agents on the same host will take the same cgroup...

P.S. k8s uses https://github.com/containerd/cgroups to dynamic create/delete the child cgroups...

[anbraten]

requested features

• timeout for pipeline (already implemented and a maximum / default can be set by server admins)
• maximum pipeline time per month per org
• maximum of concurrent pipelines per org
• api endpoints to set quotas (this will allow to have some management ui or even a payment / credit system)
• UI for orgs to see available, used and remaining quotas
• UI for admins to see usage of orgs (maybe as a list sorted by most usage at the top)

Currently I guess setting those limits per repo would probably be quite some maintenance afford so mainly ignoring it and suggesting to have it on the org level.