document expected test runtime limit

[ktdreyer]

I heard a rumor that all tests are supposed to complete within 5 minutes, but I could not find any clear documentation or source that explains such a rule in this repository. I've seen the Timeout class, but there does not appear to be a single documented policy about maximum allowed runtime for tests, nor a reference to a 5-minute universal rule.

Is there a guideline about maximum test runtime or timeouts for this repo?

What is needed

• If a formal timeout policy exists, document it in a clear place (e.g., a README).
• If there is no such policy, should we have one?

Reason for request

To help contributors know whether to submit long-running tests to this project.

[fege]

@ktdreyer I do not think that we have such a strict rule, in this repo we have also different test tier (smoke, sanity, tier1 ... etc) and we are in discussion to decide which type of test is going where and which are the rules for each tier.
Said that, 5 minutes for all the test to complete I think it is not realistic, happy to discuss more about it

[ktdreyer]

Thanks for clarifying!

[dbasunag]

@ktdreyer, We don't have any hard rule, as @fege mentioned. However, tests should be granular. They should not be written in a way that they try to do many things, in one test. e.g. if your test is about, say registering a model, it should not try to pack checking ability to create model versions, artifact validation as a part of the same test. They should be separated. They can be parameterized, made into a bunch of smaller dependent set of tests.

A component decides what kind of tests they want to add to the repo or marker they want to use for their tests. I am afraid, one hard rule won't fit all.

Please note: we use timeout sampler so that we can wait for something in a test/fixture/utility etc. This helps us to be able to wait for something to happen, instead of relying on hard coded time.sleep() in our tests. The wait time is reliant on product behavior. e.g. if you have a reconciliation behavior on the component you are testing and you know that it can take upto 3 minutes for your resource to reconcile after an action against it, then we expect your code to have a wait for the reconciliation with a wait time of 3 minutes.

Please let us know if you need anything else from our side. If you are good with it, please feel free to resolve the issue.

[ktdreyer]

Thanks!