Queue AWX experiments on a cluster

[Red-GV]

Currently, our assumption is that there are X runner clusters for X experiment. However, we would like to change the algorithm such that if there are Y clusters (Y < X), we can still run X experiments on them.

[Red-GV]

Partial implementation added via #327 . Still needs to consider the jobs waiting for a previous run to consider complete.

[Red-GV]

@rohanarora

So, I'm not seeing anything obvious in the AWX module or API that allows the pipeline to wait for one workflow to end before starting up a new one. You can wait for multiple instances of the same workflow, but not the results of a different workflow. At least, not without probably drastically expanding the nodes.

The current way I see it, we can have two options:

1. Keep things as is. The number of scenarios to run == the number of runners (clusters) and we just make an explicit assertion to warn people about that. If someone wants to run more scenarios than they have clusters, they can probably make an additional playbook or script to wait until the previous results and then do more scenarios.

2. We could make a design for that, but it would have us do a bunch of waiting. This would cause the playbook to stay active for a decent chunk of time. Not sure if that the user experience we want to support.