Introduce dedicated smoke tests suite

[achyuthajoy]

Description

With #5586, we have introduced building a plugin zip file and running E2E - Pull Request tests against that package when release, pre-release or build-zip workflows are triggered. However, instead of the existing E2E tests, we should add a dedicated smoke testing suite.

Acceptance criteria

The smoke tests suite should run when:

• A zip file is created
• A release PR is created
• A pre-release is created

[haszari]

@achyuthajoy What does "smoke tests" mean in this case?

To my mind, smoke tests are manual tests of very high level functions – e.g. checkout page renders payment form, payment can be completed.

If we are proposing automated tests to assist with release, I think we should use a different term, so we're really clear about what's different about this suite of tests. E.g. if they are automated, why do we not run all tests? Or all tests on critical flows? etc.

[haszari]

However, instead of the existing E2E tests, we should add a dedicated smoke testing suite.

@achyuthajoy What does "smoke tests" mean in this case?

To my mind, smoke tests are manual tests of very high level functions – e.g. checkout page renders payment form, payment can be completed.

If we are proposing automated tests to assist with release, I think we should use a different term, so we're really clear about what's different about this suite of tests. E.g. if they are automated, why do we not run all tests? Or all tests on critical flows? etc.

Totally agree with @vbelolapotkov here. We need to define the why of this different test suite.

I have voted 👍 based on the assumption that we want:

• A smaller set of tests
• Of highly critical, fundamentally important behaviour
• That we would run on all releases
• And would run much faster than the existing e2e test suite which is comprehensive

Perhaps we can rename this issue, one of the following:

• Small, focused suite of essential "smoke" tests
• Hierarchy labels to split test suite into fast, reliable, critical tests and comprehensive secondary tests