Skip to content

Commit 38865bb

Browse files
tdieslertdiesler
committed
Add support for TEST_SKIP_AFTER_FAILURE_COUNT
1 parent 6ef8112 commit 38865bb

File tree

2 files changed

+57
-47
lines changed

2 files changed

+57
-47
lines changed

‎e2e/README.md

+27-27
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,33 @@
22

33
This directory contains the suite of test that are run on a CI to ensure the stability of the product and no regression are introduced at each PR. The full documentation can be found at https://camel.apache.org/camel-k/next/contributing/e2e.html
44

5+
## Environment variables
6+
7+
You can set some environment variables to change the behavior of the E2E test suite.
8+
9+
| Env | Default | Description |
10+
|-----------------------------------------|-----------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------|
11+
| CAMEL_K_TEST_SKIP_PROBLEMATIC | false | Skips tests that are marked to be problematic (flaky) on certain environments (e.g. on OpenShift). |
12+
| CAMEL_K_TEST_SAVE_FAILED_TEST_NAMESPACE | false | Used to not remove the temporary test namespaces after the test run. Enables better analysis of resources after the test |
13+
| CAMEL_K_TEST_LOG_LEVEL | info | Logging level used to run the tests and used in Maven commands run by the operator (if level is `debug` the Maven commands use `-X` option). |
14+
| CAMEL_K_TEST_MAVEN_CLI_OPTIONS | {} | Maven CLI options used to run Camel K integrations during the tests. |
15+
| CAMEL_K_TEST_OPERATOR_IMAGE | docker.io/apache/camel-k:2.4.0-SNAPSHOT | Camel K operator image used in operator installation. |
16+
| CAMEL_K_TEST_OPERATOR_IMAGE_PULL_POLICY | - | Operator image pull policy. |
17+
| CAMEL_K_TEST_IMAGE_NAME | docker.io/apache/camel-k | Camel K operator image name used in operator installation. |
18+
| CAMEL_K_TEST_IMAGE_VERSION | 2.4.0-SNAPSHOT | Camel K operator image version used in operator installation. Value is retrieved from `pkg/util/defaults/defaults.go` |
19+
| CAMEL_K_TEST_NO_OLM_OPERATOR_IMAGE | docker.io/apache/camel-k:2.4.0-SNAPSHOT | Camel K operator image used in non OLM based operator installation. |
20+
| CAMEL_K_TEST_RUNTIME_VERSION | 3.8.1 | Camel K runtime version used for the integrations. Value is retrieved from `pkg/util/defaults/defaults.go` |
21+
| CAMEL_K_TEST_BASE_IMAGE | eclipse-temurin:17 | Camel K runtime base image used for the integrations. Value is retrieved from `pkg/util/defaults/defaults.go` |
22+
| CAMEL_K_TEST_TIMEOUT_SHORT | 1 | Customize the timeouts (in minutes) used in test assertions. |
23+
| CAMEL_K_TEST_TIMEOUT_MEDIUM | 5 | Customize the timeouts (in minutes) used in test assertions. |
24+
| CAMEL_K_TEST_TIMEOUT_LONG | 15 | Customize the timeouts (in minutes) used in test assertions. |
25+
| CAMEL_K_TEST_MAVEN_CA_PEM_PATH | - | Optional Maven certificate path. |
26+
| CAMEL_K_TEST_COPY_CATALOG | true | Enable/disable the optimization to copy the Camel Catalog from default operator namespace for each test namespace. |
27+
| CAMEL_K_TEST_COPY_INTEGRATION_KITS | true | Enable/disable the optimization to copy integration kits from default operator namespace for each test namespace. |
28+
| CAMEL_K_TEST_NS | - | Custom test namespace name used to create temporary namespaces. |
29+
| CAMEL_K_TEST_MAKE_DIR | - | Used in Helm and Kustomize install tests as Makefile root dir. |
30+
| CAMEL_K_TEST_MAKE_ARGS | - | Used in Helm and Kustomize install tests as Makefile arguments. |
31+
532
## Structure of the directory
633

734
NOTE: dear contributor, please, keep this organization as clean as you can, updating any documentation if any change is done.
@@ -46,30 +73,3 @@ Test suite that cover the features associated with Telemetry feature. The test e
4673
### Yaks
4774

4875
Test suite that cover certain Knative features together with YAKS operator.
49-
50-
## Environment variables
51-
52-
You can set some environment variables to change the behavior of the E2E test suite.
53-
54-
| Env | Default | Description |
55-
|-----------------------------------------|-----------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------|
56-
| CAMEL_K_TEST_SKIP_PROBLEMATIC | false | Skips tests that are marked to be problematic (flaky) on certain environments (e.g. on OpenShift). |
57-
| CAMEL_K_TEST_SAVE_FAILED_TEST_NAMESPACE | false | Used to not remove the temporary test namespaces after the test run. Enables better analysis of resources after the test |
58-
| CAMEL_K_TEST_LOG_LEVEL | info | Logging level used to run the tests and used in Maven commands run by the operator (if level is `debug` the Maven commands use `-X` option). |
59-
| CAMEL_K_TEST_MAVEN_CLI_OPTIONS | {} | Maven CLI options used to run Camel K integrations during the tests. |
60-
| CAMEL_K_TEST_OPERATOR_IMAGE | docker.io/apache/camel-k:2.4.0-SNAPSHOT | Camel K operator image used in operator installation. |
61-
| CAMEL_K_TEST_OPERATOR_IMAGE_PULL_POLICY | - | Operator image pull policy. |
62-
| CAMEL_K_TEST_IMAGE_NAME | docker.io/apache/camel-k | Camel K operator image name used in operator installation. |
63-
| CAMEL_K_TEST_IMAGE_VERSION | 2.4.0-SNAPSHOT | Camel K operator image version used in operator installation. Value is retrieved from `pkg/util/defaults/defaults.go` |
64-
| CAMEL_K_TEST_NO_OLM_OPERATOR_IMAGE | docker.io/apache/camel-k:2.4.0-SNAPSHOT | Camel K operator image used in non OLM based operator installation. |
65-
| CAMEL_K_TEST_RUNTIME_VERSION | 3.8.1 | Camel K runtime version used for the integrations. Value is retrieved from `pkg/util/defaults/defaults.go` |
66-
| CAMEL_K_TEST_BASE_IMAGE | eclipse-temurin:17 | Camel K runtime base image used for the integrations. Value is retrieved from `pkg/util/defaults/defaults.go` |
67-
| CAMEL_K_TEST_TIMEOUT_SHORT | 1 | Customize the timeouts (in minutes) used in test assertions. |
68-
| CAMEL_K_TEST_TIMEOUT_MEDIUM | 5 | Customize the timeouts (in minutes) used in test assertions. |
69-
| CAMEL_K_TEST_TIMEOUT_LONG | 15 | Customize the timeouts (in minutes) used in test assertions. |
70-
| CAMEL_K_TEST_MAVEN_CA_PEM_PATH | - | Optional Maven certificate path. |
71-
| CAMEL_K_TEST_COPY_CATALOG | true | Enable/disable the optimization to copy the Camel Catalog from default operator namespace for each test namespace. |
72-
| CAMEL_K_TEST_COPY_INTEGRATION_KITS | true | Enable/disable the optimization to copy integration kits from default operator namespace for each test namespace. |
73-
| CAMEL_K_TEST_NS | - | Custom test namespace name used to create temporary namespaces. |
74-
| CAMEL_K_TEST_MAKE_DIR | - | Used in Helm and Kustomize install tests as Makefile root dir. |
75-
| CAMEL_K_TEST_MAKE_ARGS | - | Used in Helm and Kustomize install tests as Makefile arguments. |

‎script/Makefile

+30-20
Original file line numberDiff line numberDiff line change
@@ -272,45 +272,55 @@ endif
272272
#
273273
test-common: do-build
274274
FAILED=0; STAGING_RUNTIME_REPO="$(STAGING_RUNTIME_REPO)"; \
275-
go test -timeout 90m -v ./e2e/common/... -tags=integration -parallel=$(TEST_COMMON_PARALLEL_COUNT) $(TEST_INTEGRATION_COMMON_LANG_RUN) $(GOTESTFMT) || FAILED=1; \
275+
go test -timeout 90m -v ./e2e/common/... -tags=integration -parallel=$(TEST_COMMON_PARALLEL_COUNT) $(TEST_INTEGRATION_COMMON_LANG_RUN) $(GOTESTFMT) || ((FAILED++)); \
276276
exit $${FAILED}
277277

278278
#
279279
# Smoke tests are a subset of common tests used to quickly verify the operator
280280
#
281281
test-smoke: do-build
282+
TEST_SKIP_AFTER_FAILURE_COUNT="$(TEST_SKIP_AFTER_FAILURE_COUNT)"; \
283+
if [[ $$TEST_SKIP_AFTER_FAILURE_COUNT = "" ]]; then \
284+
TEST_SKIP_AFTER_FAILURE_COUNT=0; \
285+
fi; \
282286
FAILED=0; STAGING_RUNTIME_REPO="$(STAGING_RUNTIME_REPO)"; \
283-
go test -timeout 10m -v ./e2e/common/main_test.go -tags=integration $(TEST_INTEGRATION_COMMON_LANG_RUN) $(GOTESTFMT) || FAILED=1; \
284-
go test -timeout 30m -v ./e2e/common/languages -tags=integration -parallel=$(TEST_COMMON_PARALLEL_COUNT) $(TEST_INTEGRATION_COMMON_LANG_RUN) $(GOTESTFMT) || FAILED=1; \
285-
go test -timeout 30m -v \
286-
./e2e/common/misc/cron_test.go \
287-
./e2e/common/misc/kamelet_test.go \
288-
./e2e/common/misc/pipe_test.go \
289-
-tags=integration -parallel=$(TEST_COMMON_PARALLEL_COUNT) $(TEST_INTEGRATION_COMMON_LANG_RUN) $(GOTESTFMT) || FAILED=1; \
290-
go test -timeout 30m -v \
291-
./e2e/common/traits/camel_test.go \
292-
./e2e/common/traits/container_test.go \
293-
./e2e/common/traits/openapi_test.go \
294-
./e2e/common/traits/service_test.go \
295-
-tags=integration -parallel=$(TEST_COMMON_PARALLEL_COUNT) $(TEST_INTEGRATION_COMMON_LANG_RUN) $(GOTESTFMT) || FAILED=1; \
296-
exit $${FAILED}
287+
go test -timeout 10m -count=1 -v ./e2e/common/main_test.go -tags=integration $(TEST_INTEGRATION_COMMON_LANG_RUN) $(GOTESTFMT) || ((FAILED++)); \
288+
if [ $$FAILED -le $$TEST_SKIP_AFTER_FAILURE_COUNT ]; then \
289+
go test -timeout 30m -count=1 -v ./e2e/common/languages -tags=integration -parallel=$(TEST_COMMON_PARALLEL_COUNT) $(TEST_INTEGRATION_COMMON_LANG_RUN) $(GOTESTFMT) || ((FAILED++)); \
290+
fi; \
291+
if [ $$FAILED -le $$TEST_SKIP_AFTER_FAILURE_COUNT ]; then \
292+
go test -timeout 30m -count=1 -v \
293+
./e2e/common/misc/cron_test.go \
294+
./e2e/common/misc/kamelet_test.go \
295+
./e2e/common/misc/pipe_test.go \
296+
-tags=integration -parallel=$(TEST_COMMON_PARALLEL_COUNT) $(TEST_INTEGRATION_COMMON_LANG_RUN) $(GOTESTFMT) || ((FAILED++)); \
297+
fi; \
298+
if [ $$FAILED -le $$TEST_SKIP_AFTER_FAILURE_COUNT ]; then \
299+
go test -timeout 30m -count=1 -v \
300+
./e2e/common/traits/camel_test.go \
301+
./e2e/common/traits/container_test.go \
302+
./e2e/common/traits/openapi_test.go \
303+
./e2e/common/traits/service_test.go \
304+
-tags=integration -parallel=$(TEST_COMMON_PARALLEL_COUNT) $(TEST_INTEGRATION_COMMON_LANG_RUN) $(GOTESTFMT) || ((FAILED++)); \
305+
fi; \
306+
exit $$FAILED
297307

298308
#
299309
# Common tests that require some particular operator setting or need to be installed in multiple namespaces
300310
#
301311
test-advanced: do-build
302312
FAILED=0; STAGING_RUNTIME_REPO="$(STAGING_RUNTIME_REPO)"; \
303-
go test -timeout 90m -v ./e2e/advanced -tags=integration -parallel=$(TEST_ADVANCED_PARALLEL_COUNT) $(TEST_INSTALL_RUN) $(GOTESTFMT) || FAILED=1; \
313+
go test -timeout 90m -v ./e2e/advanced -tags=integration -parallel=$(TEST_ADVANCED_PARALLEL_COUNT) $(TEST_INSTALL_RUN) $(GOTESTFMT) || ((FAILED++)); \
304314
exit $${FAILED}
305315

306316
#
307317
# Installation procedure test using CLi and Kustomize methodologies
308318
#
309319
test-install: do-build
310320
FAILED=0; STAGING_RUNTIME_REPO="$(STAGING_RUNTIME_REPO)"; \
311-
go test -timeout 40m -v ./e2e/install/cli -tags=integration $(TEST_INSTALL_RUN) $(GOTESTFMT) || FAILED=1; \
312-
go test -timeout 40m -v ./e2e/install/kustomize -tags=integration $(TEST_INSTALL_RUN) $(GOTESTFMT) || FAILED=1; \
313-
go test -timeout 40m -v ./e2e/install/helm -tags=integration $(TEST_INSTALL_RUN) $(GOTESTFMT) || FAILED=1; \
321+
go test -timeout 40m -v ./e2e/install/cli -tags=integration $(TEST_INSTALL_RUN) $(GOTESTFMT) || ((FAILED++)); \
322+
go test -timeout 40m -v ./e2e/install/kustomize -tags=integration $(TEST_INSTALL_RUN) $(GOTESTFMT) || ((FAILED++)); \
323+
go test -timeout 40m -v ./e2e/install/helm -tags=integration $(TEST_INSTALL_RUN) $(GOTESTFMT) || ((FAILED++)); \
314324
exit $${FAILED}
315325

316326
#
@@ -332,7 +342,7 @@ test-install-upgrade: do-build
332342
#
333343
test-knative: do-build
334344
STAGING_RUNTIME_REPO="$(STAGING_RUNTIME_REPO)"; \
335-
go test -timeout 60m -v ./e2e/knative -tags=integration $(TEST_KNATIVE_RUN) $(GOTESTFMT) || FAILED=1; \
345+
go test -timeout 60m -v ./e2e/knative -tags=integration $(TEST_KNATIVE_RUN) $(GOTESTFMT) || ((FAILED++)); \
336346
exit $${FAILED}
337347

338348
#

0 commit comments

Comments
 (0)