more fine grained categorisation for tests

[DaanHoogland]

as a tester, one would want a more fine grained categorisation than "smoke", "component", "integration" to be able to quickly test a PR against features it touches on.

ISSUE TYPE

• Improvement Request
• Enhancement Request
• Feature Idea
• Other : testing aid

COMPONENT NAME

Marvin

CLOUDSTACK VERSION

CONFIGURATION

OS / ENVIRONMENT

SUMMARY

when a PR is tested it is usually tested against a set of smoke tests in travis or a private cloud with s.moke tests and than maybe avery feature specific (often manual) test. A better categorisation would help in testing more related code and would allow for bypassing any tests that have no baring on the subject of the change.
A blueorangutan smoke test run takes anywhere between 8 and 15 hours and fails only once after the run, albeit with multiple reports. It would be helpful to run a smaller set of basic tests. if this set passes than a more high level batch could be choosen for a next step.

STEPS TO REPRODUCE

go-test smoke
go-test accounts-and-roles
go-test instance-groups
go-test ...

go-test could be any test infra structure like blueorangutan. It is used to illustrate the categorisation, not the test-infrastructure.

EXPECTED RESULTS

ACTUAL RESULTS

[nvazquez]

@DaanHoogland is this still on hold?

[DaanHoogland]

yes, becoming more actual, but might not be ready for 4.16

[rohityadavcloud]

Moving it to unplanned based on @DaanHoogland 's comment.

[rohityadavcloud]

Closing this - this has been open for 2+yrs; any categorisation effort won't speed up the CI/CD, will create new PRs/issues to track work towards improving CI/CD timings; one idea I want to explore is if it's possible to even run tests in parallel (or certain groups that can run in parallel).

[DaanHoogland]

this is valid as long as we use integration tests and really not yet implemented to satisfaction