It would be a great addition if the autoscaler could be configured to listen on multiple queue names.
For instance, I have a queue named private that has been in place for a long time and a lot of repos target this queue. Now, I have introduced private-arm64 as we expand into multi-arch builds.
I have used the BuildkiteAgentTags input on the elastic-ci stack to add additional queue tags to private: queue: private-amd64 and queue: private-x86_64. This mostly works. If a job is created and asks for private-amd64 it will be assigned. But if there are no private queue workers running none will be created because the autoscaler is only watching for BUILDKITE_AGENT: private and not private-amd64.
Also, being able to target queue: private-{{matrix.arch}} is VERY useful to avoid duplication in pipelines.
This means I either need to:
- Switch over to queue names such as
private-amd64 and private-arm64, which would require updating a lot of repos from private -> private-amd64
- Introduce an entirely new
private-amd64 queue. Slightly inefficient on resource usage with extra agents hanging around.
I think the ideal would be for the autoscaler to support multiple queue names to trigger a scale event, eg: BUILDKITE_QUEUE: private,private-amd64,private-x86_64. This would also match the behavior of the buildkite-agent which is able to listen for and accept jobs on multiple queue tags.
It would be a great addition if the autoscaler could be configured to listen on multiple queue names.
For instance, I have a queue named
privatethat has been in place for a long time and a lot of repos target this queue. Now, I have introducedprivate-arm64as we expand into multi-arch builds.I have used the
BuildkiteAgentTagsinput on the elastic-ci stack to add additional queue tags toprivate:queue: private-amd64andqueue: private-x86_64. This mostly works. If a job is created and asks forprivate-amd64it will be assigned. But if there are noprivatequeue workers running none will be created because the autoscaler is only watching forBUILDKITE_AGENT: privateand notprivate-amd64.Also, being able to target
queue: private-{{matrix.arch}}is VERY useful to avoid duplication in pipelines.This means I either need to:
private-amd64andprivate-arm64, which would require updating a lot of repos fromprivate->private-amd64private-amd64queue. Slightly inefficient on resource usage with extra agents hanging around.I think the ideal would be for the autoscaler to support multiple queue names to trigger a scale event, eg:
BUILDKITE_QUEUE: private,private-amd64,private-x86_64. This would also match the behavior of thebuildkite-agentwhich is able to listen for and accept jobs on multiplequeuetags.