Improve CICD Test Build Matrix Generation #4546
Description
Today, our CI/CD workflows randomly assign test systems to PRs and most single-use build/test workflows through a naive random number generator. This approach leads to an inefficient use of our test systems and can create frustrating scenarios where 3 separate PRs are all assigned to a single system, causing a delay of 2+ hours before a build and test cycle completes.
We should change the matrix generator and locking mechanism to be smarter about using system resources; picking open systems where available, and when all systems are in-use, creating a backlog/queue of system reservations visible to all other builds. Ideally, locks and queues include an estimated time to complete the job, so this can also factor into the system reservation choice.
This issue will require work in this repository's matrix generator, zowe-action's locking mechanism, and potentially a new action which checks locks/queues before building the workflow matrix.