Add support for TEST_SKIP_AFTER_FAILURE_COUNT

[tdiesler]

Borrowed from maven surefire - count test failures and skip subsequent tests when failure count > $TEST_SKIP_AFTER_FAILURE_COUNT
The default is TEST_SKIP_AFTER_FAILURE_COUNT=3

This can significantly speed up validation of bad PRs

[squakez]

Not sure to understand this change request. Consider that when you're running in GH actions, each workflow would execute each test suite individually. For instance, it does make test-common. So, in such case, this new variable is never took in consideration. The only case where it can make sense here would be in the test-smoke. However, such a test is only executed at night time before nightly release, and, it only have 4 possible failures, so, we'd really skip almost nothing.

[tdiesler]

make test-smoke takes > 30min on my box. If something goes wrong at the beginning, it already safes quite some time when set to 1. We can expand this to the other test suites later, if wanted

[squakez]

make test-smoke takes > 30min on my box. If something goes wrong at the beginning, it already safes quite some time when set to 1. We can expand this to the other test suites later, if wanted

Okey, then, we better make the variable scope local to this specific action. Otherwise we can get mislead and think that this is a global scope and would affect all test suite executions.

[christophd]

How about a TEST_FAIL_FAST=true option so the test suite stops after the 1st failure. I don't see why somebody would be interested in stopping execution after >1 failing tests when asking for fast feedback.

[tdiesler]

In surefire, it defaults to 0. Folks set it higher to see e.g. "the first three failures"
Would you like to expand this to the other test suites i.e. if smoke fails, no need to execute the others?

[tdiesler]

Ready to merge

[squakez]

Ready to merge

Please, see this comment #5475 (comment) - if we want this variable, then, we need to make it in the specific test scope to avoid users to mislead its meaning. Thanks.

[christophd]

just another nitpick: I have started to document the available E2E test env vars in https://github.com/apache/camel-k/blob/main/e2e/README.md#environment-variables

Maybe we can add a section on how to run tests locally with Makefile and the possible env var settings

[tdiesler]

Okey, then, we better make the variable scope local to this specific action. Otherwise we can get mislead and think that this is a global scope and would affect all test suite executions.

This is done

[tdiesler]

just another nitpick: I have started to document the available E2E test env vars in https://github.com/apache/camel-k/blob/main/e2e/README.md#environment-variables

Maybe we can add a section on how to run tests locally with Makefile and the possible env var settings

I now promoted the env var table to the top and the skip var is now local to the smoke tests like this ...

test-smoke: do-build
	TEST_SKIP_AFTER_FAILURE_COUNT="$(TEST_SKIP_AFTER_FAILURE_COUNT)"; \
	if [[ $$TEST_SKIP_AFTER_FAILURE_COUNT = "" ]]; then \
		TEST_SKIP_AFTER_FAILURE_COUNT=0; \
	fi; \
	FAILED=0; STAGING_RUNTIME_REPO="$(STAGING_RUNTIME_REPO)"; \

Folks wanting to see more than one smoke test failure are likely familiar with the Makefile anyway - so I guess it sufficiently documents itself here.

[tdiesler]

could we pls run this again?

[tdiesler]

fails unrelated, pls try again

[tdiesler]

repeatedly fails in

❌ TestOLMOperatorUpgrade (16m29.34s)
❌ TestOLMOperatorUpgrade/OLM_upgrade (10m0.03s)

[squakez]

Yes, the checks were failing because it required a rebase after 2.3.2 release. Merging anyway.