smoke tests: v2

[endorama]

This issue aims at consolidating multiple recent issues related to smoke tests and their Golang siblings, functional tests. Will act as meta issue to bridge the gaps between current smoke tests (v1) and v2 (v2 will reuse the current setup prepared for functionaltests. functionaltests will be migrated to smoke tests v2 and will be removed.)

We know that our current smoke tests framework has some limitations that slowed down some bug fixes and some feature implementation.

In parallel we started building some more complex tests after 8.15.0 to test ILM and DSL behavior in different APM Server versions. For these tests we decided to build a new Go test framework that we anticipate could replace the smoke tests one over time.

We are at a stage where most of the features in smoke tests are implemented in functional tests and we can think of moving to a v2 of smoke test, integrating the 2 series of tests in a single place. We can partition running tests using standard go test flags and patterns.

We can also take the chance to consolidate our current smoke testing strategy. Issues linked here still require to be uniformed/reworked to align with the direction, as of now there is some confusion around functionaltests and what they cover.

Approach:

1. consolidate our current smoke tests, fixing what's broken and covering all use cases defined that do not require additional feature development
2. migrate v1 smoke test cases to v2, one at a time functionaltests: Migrate existing smoke tests to functional tests #16130
3. migrate current relevant functional tests functionalities to testing/smoke/v2

Requirements to make the migration successful:

1. we need to fetch latest versions functionaltests: Add ability to fetch stack versions #16200
2. we need to define all the scenarios we expect functionaltests: Add ability to fetch stack versions #16200
3. we need a way to create custom ILM policies to apply to APM data streams, this is still

Benefits from this work:

1. Adding new tests scenarios functionaltests: Adding new test scenarios #16146
   1. we want to test the reroute processor use case Expand smoke tests to include reroute processors #14061
   2. we want to test ILM behaviour New indexes created for datastreams after update to 8.15.0 are without lifecycle policies #13898 (comment)
2. manage ES deprecation info functionaltests: 7.x standalone -> 8.x standalone -> 9.x standalone -> 9.x managed #16199 (comment)

Where we are now:

• the set of tests to run is discussed in functionaltests: Add ability to fetch stack versions #16200 and mostly finalized
• on smoke tests:
  • we need to address smoketest: basic-upgrade 8.18.0 -> 9.0.0 assertion fail #16179
  • we need to address functionaltests: 7.x standalone -> 8.x standalone -> 9.x standalone -> 9.x managed #16199
• on functional tests:
  • we need to add tests for ILM upgrades/use use case New indexes created for datastreams after update to 8.15.0 are without lifecycle policies #13898
  • we need to automate fetching latest version functionaltests: Add ability to fetch stack versions #16200
  • we need to add APM standalone tests from 7.x, WIP
  • functionaltests: add 8.17 tests #15310 is probably superseded by functionaltest: Add one re-route processor test #15960 but we should recover the apm-data migration test case

[simitt]

@endorama when originally discussing the functional tests, one of the goals was that we would be able to run certain tests across several versions quickly and in-expensively. E.g with switching to apm data in ES and switching to DSL we experienced different behavior when upgrading from v7.17 -> 8.15, v8.0-8.15, v8.7+ -> 8.15, v8.13+ -> 8.15. This was due to some changes in apm-server behavior during the 8.x lifetime.
The classic smoke tests are usually running upgrades from the last major/minor versions to the current versions, we do not test upgrades from much older minors of the same major. When merging the functionaltests, that were specifically focusing at the data template, mapping and lifecycle setup, with general smoke tests, what is your proposal for achieving the test coverage we were aiming for with the functional tests while not making the overall smoke tests inefficient?