smoke: Clarify smoke tests scenarios and support 2 minors

[carsonip]

• It is unclear what scenarios are the smoke tests covering. Clarify and document them in the repo.
• Update smoke tests to test 2 minor versions, as we have 2 active minors now. Run smoke-tests-ess for 8.16 and 8.17 #15885
• Ensure that SNAPSHOT versions are tested

[simitt]

Please let me know your thoughts about the following suggestion for what to test in the current state.

Testing Goals:

• test next patch version of released and supported minor versions
• test next minor version of released and supported major version
• test next major version if the development for the next major version has started
• test version in FF and in developer branch
• do not test released versions but versions under development

Currently released versions:

• 7.17.28
• 8.16.5
• 8.17.3

Upcoming releases:

• 8.16.6
• 8.17.4
• 8.18.0
• 8.19.0
• 9.0.0
• 9.1.0

OS Support Tests:

• 7.17.29-SNAPSHOT
• 8.16.0-SNAPSHOT
• 8.17.0-SNAPSHOT
• 8.18.0-SNAPSHOT
• 8.19.0-SNAPSHOT
• 9.0.0-SNAPSHOT
• 9.1.0-SNAPSHOT

Upgrade Tests:

• 7.17.29-SNAPSHOT -> 8.18.0-SNAPSHOT (somewhat arbitrary to which of the 8.x versions to upgrade to)
• 8.16.6-SNAPSHOT -> 8.17.4-SNAPSHOT
• 8.17.4-SNAPSHOT -> 8.18.0-SNAPSHOT
• 8.18.0-SNAPSHOT -> 8.19.0-SNAPSHOT
• 8.18.0-SNAPSHOT -> 9.0.0-SNAPSHOT
• 8.19.0-SNAPSHOT -> 9.0.0-SNAPSHOT
• 9.0.0-SNAPSHOT -> 9.1.0-SNAPSHOT

[ericywl]

For reference, currently available tests

Legacy Managed:

• 7.17.28 standalone -> 7.17.28 managed
• 7.17.29-SNAPSHOT standalone -> 7.17.29-SNAPSHOT managed

Legacy Standalone to Managed:

• 7.17.28 standalone -> 8.18.0 standalone -> 8.18.0 managed
• 7.17.29-SNAPSHOT standalone -> 9.1.0-SNAPSHOT standalone -> 9.1.0-SNAPSHOT managed (doesn't work because we can't jump from 7.x to 9.x)

OS Tests (if fixed):

• 8.16.6-SNAPSHOT
• 8.17.4-SNAPSHOT
• 8.18.0-SNAPSHOT
• 8.19.0-SNAPSHOT
• 9.0.0-SNAPSHOT
• 9.1.0-SNAPSHOT

Upgrade Tests:

• 7.17.27 -> 7.17.28
• 8.15.5 -> 8.16.5
• 8.16.5 -> 8.17.3
• 8.17.3 -> 8.18.0
• 8.18.0 -> 9.0.0

Next Actions

1. Update to use SNAPSHOT versions where necessary
2. Document the scenarios

[simitt]

Legacy Standalone to Managed:

7.17.28 standalone -> 8.18.0 standalone -> 8.18.0 managed
7.17.29-SNAPSHOT standalone -> 9.1.0-SNAPSHOT standalone -> 9.1.0-SNAPSHOT managed (doesn't work because we can't jump from 7.x to 9.x)

Since we are testing standalone and managed (which is great!), WDYT about also adding the following:
7.17.28 standalone -> 8.18.0 standalone -> 9.0.0 standalone -> 9.0.0 managed

[ericywl]

Since we are testing standalone and managed (which is great!), WDYT about also adding the following:
7.17.28 standalone -> 8.18.0 standalone -> 9.0.0 standalone -> 9.0.0 managed

Sounds good! I will need to take a look at how the versions are retrieved first.

[ericywl]

do not test released versions but versions under development

@simitt Does this mean that we should replace released versions from our tests currently? Or should we test both released and not-yet released versions?

For example, in "Upgrade Tests", we use released versions (e.g. 8.16.5 -> 8.17.3) instead of not-yet released versions (e.g. 8.16.6 -> 8.17.4). We can either:

1. Replace them all with not-yet released SNAPSHOTs
2. Add on the not-yet released SNAPSHOTs, keep existing ones

Same goes for "Legacy Managed" and "Legacy Standalone to Managed" tests.

7.17.29-SNAPSHOT -> 8.18.0-SNAPSHOT (somewhat arbitrary to which of the 8.x versions to upgrade to)

Also what about 7.17.28-SNAPSHOT -> 7.17.29-SNAPSHOT? Slightly unrelated, but 7.17.28-SNAPSHOT fails to deploy in ECH for some reason.

[ericywl]

Marc also brought up a point:

I don't have anything against this but we should be mindful that we won't be testing against the exact artefact that we release to customers, since there may have been additional unreleased commits post the "public" release. If we're fine with that we should merge this since I still think there are tangible benefits to always testing with SNAPSHOT.