Add gitlab-orphaned-job-canceller cronjob - #1415
Conversation
We recently noticed some GitLab CI jobs get stuck in a "Running" state beyond their normal job timeout. We traced this back to jobs running on our Kubernetes runners that lose their backing pod without GitLab ever finding out. In at least one instance, this was caused by Karpenter wrongly deleting a node it thinks is empty: kubernetes-sigs/karpenter#2916. This commit adds a new cronjob to mitigate this issue. It correlates running jobs against Kubernetes pods and cancels those whose pod is missing or already terminal. This new cronjob only cancels jobs (it doesn't restart them), and it only ever touches jobs from Spack group runners (not instance runners). This corresponds to our Kubernetes runners but not our on-premises runners at the University of Oregon.
|
It would also be good to include canceling any jobs in general that exceed the job timeout, regardless of where they are started from. |
I like this idea, but I don't think the GitLab API actually tells us when a job would timeout 🫠 |
| failed_projects = [] | ||
| for project in projects: | ||
| try: | ||
| cancel_orphaned_jobs(v1, project, args.grace_period_minutes) |
There was a problem hiding this comment.
Is it worth collecting these jobs and raising an error with them for sentry to report?
There was a problem hiding this comment.
Good idea, I added a sentry warning for orphaned jobs that get canceled & retried, and made it an error if we exhaust our retry cap. @jjnesbitt FYI. I think this is a rare enough defect that it won't flood sentry but I wanted to make you aware.
| period (to avoid racing normal pod-scheduling delays), and its backing | ||
| pod either no longer exists, or has already reached a terminal phase | ||
| without GitLab having found out.""" | ||
| if not is_kubernetes_executor_job(job): |
There was a problem hiding this comment.
I think we can also check here if the runtime is longer than the expected max timeout of 12h?
Then, jobs orphaned by any non-k8s runners can be cancelled. I don't think I have seen these, but it seems to be possible during updates/unexpected downtime this type of thing can happen.
|
Since this doesn't retry any jobs, if a job that would have been retried on failure is cancelled due to overrun, it won't be retried at all by any other service, including gitlab. Correct? Are we just assuming these jobs will be retried manually? It may be hard to clearly identify which jobs require this, since they won't be obviously overrunning their timeout. |
Yes, that is my understanding as well.
The issue that this PR is mitigating is that jobs are stuck in a running state indefinitely. Because of our use of deferred pipelines, when this happens on a develop pipeline (like it did this morning) it has the affect of preventing any new PRs from getting tested/merged until we manually intervene. At the risk of stating the obvious, jobs that run forever don't get automatically retried. That being said, I can look into adding auto-retry logic to this new cronjob if you think that's a good idea. |
The 30 minute grace period probably handles most of these cases where gitlab would have retried the job itself, but since this is going to catch issues much earlier the pipeline is probably still useful the the developer so retrying would be good. |
I was mainly just trying to make sure we weren't taking a half-measure w.r.t the mitigation. Since the goal is to allow deferred pipelines to run, I think limiting the scope as you've done is perfectly fine. |
Log a warning to sentry when a job is canceled and retried, and log an error when the retry cap is reached.
Our f2f conversation about canceled jobs being confusing to users was enough to convince me that this is probably a good idea. This script now attempts to retry canceled jobs up to a limit of 3 total retries per job. |
3 participants
We recently noticed some GitLab CI jobs get stuck in a "Running" state beyond their normal job timeout. We traced this back to jobs running on our Kubernetes runners that lose their backing pod without GitLab ever finding out. In at least one instance, this was caused by Karpenter wrongly deleting a node it thinks is empty: kubernetes-sigs/karpenter#2916.
This commit adds a new cronjob to mitigate this issue. It correlates running jobs against Kubernetes pods and cancels those whose pod is missing or already terminal. This new cronjob only cancels jobs (it doesn't restart them), and it only ever touches jobs from Spack group runners (not instance runners). This corresponds to our Kubernetes runners but not our on-premises runners at the University of Oregon.