Overhaul Testing Strategy #350
Description
Maybe incorporating things like the Python Unit Test module or other suitable methodology (ex. PyTest), it would be nice if given a Kubernetes cluster:
- A comprehensive unit test was run on the local machine
- This unit test strategy was consumable by the integration tests
Right now, there is no unit test structure in ITBench and the integration tests are the bare minimum in smoke testing (ensuring that the playbook fires and exits cleanly). However, these tests are also very time consuming and resource consuming. The smoke tests take several VMs to run (the scenarios take like 15) and a single test can go upwards to 40 minutes.
Having a unit test version would also the user to check against their local machine and may be able to employ better testing techniques which can be used. Readout of the results of the playbook should be presented in the case of an error. Also, it should be easy to determine which test failed without searching too much.
Hopefully, through these changes, we can test more efficiently and effectively, while not using too many VMs on GitHub actions as with the current load, it is quite easy to load up too many and cause them to stay in the queue state - exacerbating how long it takes to get a result back.