Add ability to prioritize GitHub Action runners

[gajus]

Describe the enhancement

Self-hosted GitHub actions should have an attribute (weight) that allows to prioritize them, i.e. If there are multiple idle runners with matching labels, then the weight attribute would determine which runner to use first, e.g. prioritized in ascending order.

Additional information

For context, the reason this is needed is because the current implementation randomly picks an available runner. However, imagine that you are scaling up and down runners depending on how long they have been idle. Using random allocation mechanism, there is no way to determine (efficiently) how long the runner was not in use. As a result, we have a large portion of VMs runnings that are not in use most of the time.

Prioritization would allow more efficient resource packing.

[ruvceskistefan]

Hi @gajus,
Thanks for the reported issue and idea. I'm adding a future label, so that we can work on developing this in near future.

[gajus]

@ruvceskistefan Do you have any update on this? This severely impacts our ability to scale GitHub Runners, costing literally tends of thousands monthly.

[ethomson]

Hi @gajus - can you tell me more about this? It sounds like you're rolling your own autoscaling solution? Does the ephemeral runner option not give you enough control here?

[gajus]

The original issue text already includes description of how we orchestrate runners.

Whether we use ephemeral option or not, the problem is that there is no way to prioritize which runners will be picked up first. This means that if we have 100 idle runners and we have 20 jobs, then we have no way to say that these 20 should idle runners should be used first. Without weight of such for sorting, this means that a large number of idle runners is just sitting waiting for jobs because they keep getting random jobs (which they would otherwise not get if there was an order assigned).

[ethomson]

What I don't understand yet is whether these 20 runners in your example are different in some meaningful way that you really want to route to these 20? In other words, does it matter which 20 runners are getting jobs, or do you just want to scale down to any 20 runners? You mention "more efficient resource packing" but I don't feel like I have the full picture yet.

[al2114]

Chiming in that we have a pool of machines that we use as runners, though some machines run significantly faster than others (reducing build time significantly). Ideally we want to be able to prioritize allocating jobs to the faster machines to reduce build times, but want to keep the slower machines active so they could pickup jobs while the faster ones are busy. Prioritization would be very beneficial here.

[ethomson]

Thanks @al2114 - are you running static runners? I can understand the need for some more advanced routing there, but I'm trying to better understand the need for weights when auto scaling, or when using some sort of control plane.

[gajus]

What I don't understand yet is whether these 20 runners in your example are different in some meaningful way that you really want to route to these 20?

No. All machines are identical.

Here is a simple task. You have 100 machines. You have 20 jobs that start every minute and complete in a minute.

What happens in the current setup?

Every minute a random 20 machines will get picked from the pool.

Why is that bad?

Machines that have not been used for 10 minutes are automatically removed from the pool. If resources are randomly assigned, then machines that otherwise would not need to have been used are being used. Therefore, you will always have 100 machines running even though 20 would suffice.

What's desired?

A way to prioritize which machines should get picked first. This way, the oldest machines (as an example) will always get used first and the rest will soon timeout and disconnect from the pool.

[naikrovek]

your autoscaling solution is probably a bit too naive.

your assumption that the jobs are assigned is also wrong (I'm pretty sure). jobs are pulled by the runners, and not pushed to them. each runner periodically queries to see if any work is available. the first runner to pull that info after it is available gets it, and does the work. in order for a priority system to be implemented, all runners you host would need to talk to each other to know who should poll next.

the GitHub Actions Runner system, in its entirety, is making the assumption that the runner virtual machines are used a single time, then destroyed and replaced with a fresh VM when needed. solutions which do not keep that assumption in mind are going to have a difficult time adjusting to how GHA works.

I wrote an orchestrator in Go which uses workflow_job payloads exclusively to know when to destroy runner VMs and to know when to bring more online. everything is very smooth since I accepted that single-use runners (GitHub call them ephemeral runners) were the correct approach, and stopped fighting it.

[jharris-tc]

I would like something kind of relevant to this.

I would like to be able to give the runners the ability to opt out of polling, based on some health check. In my instance, I have a a few VMs, each with a few runners running, and I want a runner to be able to recognize that there is, lets say 95% memory usage, and not pick up a job. This will allow a runner on a less congested VM to pick it up. Right now, sometimes a congested VM will still pick up the job, and then oom.

This would not require runners communicating with each other, but basically just polling some endpoint, either through like curl or some file/socket, and if the number is 1 then pick up the job, if 0 then don't

Would be super helpful in distributing jobs

[idyll]

As another example for this, we have M1 and Intel self hosted Mac runners.

The M1s are so much faster that we'd love a way to give them priority over the intel runners and only send jobs to the intel runners if all the M1 runners are busy.

The weight solution would work but really anything that allows us to set precedence would be great.

[gerhard]

I am also interested in this. This is my generic question which lead me here: https://github.com/orgs/community/discussions/30693

This issue seems like a great place to add more context and make it specific so that you can better determine if this is a legit +1

Our Test Universe workflow is highly variable, but it usually takes ~20 minutes to complete. When we run it locally, it completes within 5 minutes:

When we parallelise, it completes in less than 2 minutes. That is a significant 10x speed-up. We cannot parallelise it in GitHub Actions because we hit the 7GB memory limit (context). We would like to use self-hosted GitHub Runners in order to achieve this 10x speed-up.

If our own self-hosted GitHub runners are not available (busy, offline, etc.), free GitHub Runners should pick up those jobs. Currently, if we were to use jobs.<job_id>.runs-on: self-hosted, we would be excluding the free GitHub Runners.

I have two questions:

1. Would it make sense to add jobs.<job_id>.runs-on-preferred: [self-hosted, ubuntu-latest] ?
2. Can you think of a different way of achieving the above without this feature?

Thank you!

[gerhard]

The new larger GitHub Actions hosted runners makes my previous comment a non-issue. This new feature made a huge positive different for us already: dagger/dagger#3277 (comment). Great job everyone! 🤘