Skip to content

Commit 544ee58

Browse files
scottkurzscottkurz
authored
Merge pull request #107 from awisniew90/regression_test
Add basic test infrastructure
2 parents 9a6acc1 + 0a04742 commit 544ee58

File tree

5 files changed

+357
-0
lines changed

5 files changed

+357
-0
lines changed

‎.github/workflows/stack-regression-tests.yml

+77
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
name: stack-regression-tests
2+
on:
3+
pull_request:
4+
branches:
5+
- main
6+
jobs:
7+
stack-tests:
8+
runs-on: ubuntu-latest
9+
steps:
10+
11+
- name: Checkout
12+
uses: actions/checkout@v2
13+
14+
- name: Get stack image diff
15+
id: stack-image-diff
16+
uses: softprops/diffset@v1
17+
with:
18+
base: main
19+
stackimage_files: |
20+
stackimage/*
21+
templates/stackimage/*
22+
env:
23+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
24+
25+
- name: Get outer loop diff
26+
id: outer-loop-diff
27+
uses: softprops/diffset@v1
28+
with:
29+
base: main
30+
outerloop_files: |
31+
stackimage/*
32+
templates/stackimage/*
33+
templates/outer-loop/*
34+
env:
35+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
36+
37+
- name: Setup Minikube
38+
uses: manusa/[email protected]
39+
with:
40+
minikube version: 'v1.11.0'
41+
kubernetes version: 'v1.17.0'
42+
github token: ${{ secrets.GITHUB_TOKEN }}
43+
start args: '--addons=registry --addons=ingress'
44+
45+
- name: Wait for nodes to be ready
46+
run: |
47+
while [[ $(kubectl get nodes -o 'jsonpath={..status.conditions[?(@.type=="Ready")].status}') != "True" ]]; do
48+
echo "waiting for nodes" && sleep 1;
49+
done
50+
51+
- name: Install odo
52+
run: |
53+
sudo curl -L https://mirror.openshift.com/pub/openshift-v4/clients/odo/latest/odo-linux-amd64 -o /usr/local/bin/odo
54+
sudo chmod +x /usr/local/bin/odo
55+
56+
- name: Print version info
57+
run: |
58+
set -x
59+
docker version
60+
kubectl version
61+
odo version
62+
minikube version
63+
set +x
64+
65+
- name: Build stack
66+
run: ./test/utils.sh buildStack
67+
68+
- name: Build stack image
69+
if: steps.stack-image-diff.outputs.stackimage_files
70+
run: ./test/utils.sh buildStackImage
71+
72+
- name: Inner loop test
73+
run: ./test/stack-test-inner-loop.sh
74+
75+
- name: Outer loop test
76+
if: steps.outer-loop-diff.outputs.outerloop_files
77+
run: ./test/stack-test-outer-loop.sh

‎test/README.md

+74
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
## Test Overview
2+
3+
The application-stack repo is tested using Github Actions. The main workflow is defined in `stack-regression-tests.yml` and performs the following steps:
4+
5+
### Install Minikube
6+
7+
This is done via the `manusa/actions-setup-minikube` actions plugin
8+
9+
### Install ODO
10+
11+
The latest version is installed
12+
13+
### Build stack
14+
15+
This runs the `build.sh` script to generate all stack content in the `generated/` dir based on the PRs branch.
16+
17+
### Build stack image
18+
19+
The stack image is built if changes are detected to `stackimage/` or `templates/stackimage`
20+
21+
### Run inner loop tests
22+
23+
Basic setup:
24+
25+
Clone application-stack-intro, create, push.
26+
27+
Tests against following endpoints:
28+
29+
1. /health/live
30+
1. /health/ready
31+
1. /api/resource
32+
33+
### Run outer loop tests
34+
35+
Outerloop tests are only done if changes are detected to `templates/outer-loop` or any stack image content (`stackimage/` or `templates/stackimage`)
36+
Basic setup:
37+
38+
Clone application-stack-intro, build docker image, install OL operator, deploy.
39+
40+
Tests against following endpoints:
41+
42+
1. /health/live
43+
1. /health/ready
44+
1. /api/resource
45+
46+
47+
## Additional features
48+
49+
1. Tests are triggered on each PR or update to a PR
50+
1. All local docker images are available for use by the Minikube cluster since Minikube is installed/started with `driver=none` (bare metal)
51+
1. Changes to specific files (which gates what steps are run) is controlled via the `softprops/diffset@v1` actions plugin
52+
53+
54+
## Known issues
55+
56+
1. Intermittent Ingress connection refused errors:
57+
58+
```
59+
Applying URL changes
60+
✗ Failed To Update Config To Component Deployed.
61+
Error: unable to create ingress: error creating ingress: Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": Post https://ingress-nginx-controller-admission.kube-system.svc:443/extensions/v1beta1/ingresses?timeout=30s: dial tcp 10.96.251.20:443: connect: connection refused
62+
```
63+
64+
If this occurs, the tests will need to be rerun.
65+
66+
1. Warning message due to known "bug" in `softprops/diffset@v1` plugin (https://github.com/softprops/diffset/issues/5):
67+
68+
```
69+
Unexpected input(s) 'stackimage_files', valid inputs are ['base']
70+
```
71+
72+
## Future Tests
73+
74+
1. Validate version changes in build.sh to make sure proper incrementation (i.e. you changed the outerloop Dockerfile but didnt increment the outerloop version)

‎test/stack-test-inner-loop.sh

+67
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
#!/bin/bash
2+
3+
mkdir inner-loop-test-dir
4+
cd inner-loop-test-dir
5+
6+
echo -e "\n> Clone application-stack-intro project"
7+
git clone https://github.com/OpenLiberty/application-stack-intro.git
8+
cd application-stack-intro
9+
10+
echo -e "\n> Copy devfile"
11+
cp ../../generated/devfile.yaml devfile.yaml
12+
13+
echo -e "\n> Create new odo project"
14+
odo project create inner-loop-test
15+
16+
echo -e "\n> Create new odo component"
17+
odo create my-ol-component
18+
19+
echo -e "\n> Create URL with Minikube IP"
20+
odo url create --host $(minikube ip).nip.io
21+
22+
echo -e "\n> Push to Minikube"
23+
odo push
24+
25+
echo -e "\n> Check for server start"
26+
count=1
27+
while ! odo log | grep -q "CWWKF0011I: The defaultServer server"; do
28+
echo "waiting for server start... " && sleep 3;
29+
count=`expr $count + 1`
30+
if [ $count -eq 20 ]; then
31+
echo "Timed out waiting for server to start"
32+
exit 12
33+
fi
34+
done
35+
36+
echo -e "\n> Test liveness endpoint"
37+
livenessResults=$(curl http://my-ol-component.$(minikube ip).nip.io/health/live)
38+
if echo $livenessResults | grep -qF '{"checks":[{"data":{},"name":"SampleLivenessCheck","status":"UP"}],"status":"UP"}'; then
39+
40+
echo "Liveness check passed!"
41+
else
42+
echo "Liveness check failed. Liveness endpoint returned: "
43+
echo $livenessResults
44+
exit 12
45+
fi
46+
47+
echo -e "\n> Test readiness endpoint"
48+
readinessResults=$(curl http://my-ol-component.$(minikube ip).nip.io/health/ready)
49+
if echo $readinessResults | grep -qF '{"checks":[{"data":{},"name":"SampleReadinessCheck","status":"UP"}],"status":"UP"}'; then
50+
51+
echo "Readiness check passed!"
52+
else
53+
echo "Readiness check failed! Readiness endpoint returned: "
54+
echo $readinessResults
55+
exit 12
56+
fi
57+
58+
echo -e "\n> Test REST endpoint"
59+
restResults=$(curl http://my-ol-component.$(minikube ip).nip.io/health/live)
60+
if ! echo $restResults | grep -qF 'Hello! Welcome to Open Liberty'; then
61+
62+
echo "REST endpoint check passed!"
63+
else
64+
echo "REST endpoint check failed. REST endpoint returned: "
65+
echo $restResults
66+
exit 12
67+
fi

‎test/stack-test-outer-loop.sh

+102
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,102 @@
1+
#!/bin/bash
2+
3+
OUTER_LOOP_TEST_NAMESPACE='outer-loop-test'
4+
5+
mkdir outer-loop-test-dir
6+
cd outer-loop-test-dir
7+
8+
echo -e "\n> Clone application-stack-intro project"
9+
git clone https://github.com/OpenLiberty/application-stack-intro.git
10+
cd application-stack-intro
11+
12+
echo -e "\n> Copy Dockerfile"
13+
cp ../../generated/Dockerfile Dockerfile
14+
15+
echo -e "\n> Copy app-deploy.yaml"
16+
cp ../../templates/outer-loop/app-deploy.yaml app-deploy.yaml
17+
18+
echo -e "\n> Create new odo project"
19+
odo project create ${OUTER_LOOP_TEST_NAMESPACE}
20+
21+
### This is only needed if the Minikube driver is "docker". As of now, we are using diver=none or "bare metal"
22+
### which allows Minikube to use the local docker registry
23+
#echo -e "\n Use the Minikube image registry"
24+
#eval $(minikube docker-env)
25+
26+
echo -e "\n> Build Docker image"
27+
sed -i '/COPY --from=compile/a RUN true' Dockerfile
28+
docker build -t outerloop/application-stack-intro:1.0 .
29+
30+
echo -e "\n> Replace variables in app-deploy.yaml"
31+
sed -i 's/{{\.PORT}}/9080/g' app-deploy.yaml
32+
sed -i 's/{{\.COMPONENT_NAME}}/my-ol-deployment/g' app-deploy.yaml
33+
sed -i 's/{{\.CONTAINER_IMAGE}}/outerloop\/application-stack-intro:1\.0/g' app-deploy.yaml
34+
35+
echo -e "\n> Install Open Liberty Operator"
36+
kubectl apply -f https://raw.githubusercontent.com/OpenLiberty/open-liberty-operator/master/deploy/releases/0.7.0/openliberty-app-crd.yaml
37+
curl -L https://raw.githubusercontent.com/OpenLiberty/open-liberty-operator/master/deploy/releases/0.7.0/openliberty-app-operator.yaml \
38+
| sed -e "s/OPEN_LIBERTY_WATCH_NAMESPACE/${OUTER_LOOP_TEST_NAMESPACE}/" \
39+
| kubectl apply -n ${OUTER_LOOP_TEST_NAMESPACE} -f -
40+
41+
echo -e "\n> Wait for operator pod to start"
42+
count=1
43+
while [[ $(kubectl get pods -l name=open-liberty-operator -o 'jsonpath={..status.conditions[?(@.type=="Ready")].status}') != "True" && $count -lt 20 ]]; do
44+
kubectl get pods
45+
echo "waiting for operator pod" && sleep 3;
46+
count=`expr $count + 1`
47+
done
48+
if [ $count -eq 20 ]; then
49+
echo "Timed out waiting for operator pod to start"
50+
kubectl describe pods -l name=open-liberty-operator
51+
exit 12
52+
fi
53+
54+
echo -e "\n> Deploy image"
55+
cat app-deploy.yaml
56+
kubectl apply -f app-deploy.yaml
57+
58+
echo -e "\n> Wait for pod to start"
59+
count=1
60+
while [[ $(kubectl get pods -l app.kubernetes.io/name=my-ol-deployment -o 'jsonpath={..status.conditions[?(@.type=="Ready")].status}') != "True" && $count -lt 20 ]]; do
61+
kubectl get pods
62+
echo "waiting for pod" && sleep 3;
63+
count=`expr $count + 1`
64+
done
65+
if [ $count -eq 20 ]; then
66+
echo "Timed out waiting for pod to start"
67+
kubectl describe pods -l app.kubernetes.io/name=my-ol-deployment
68+
exit 12
69+
fi
70+
71+
echo -e "\n> Test liveness endpoint"
72+
livenessResults=$(curl http://my-ol-deployment.$(minikube ip).nip.io/health/live)
73+
if echo $livenessResults | grep -qF '{"checks":[{"data":{},"name":"SampleLivenessCheck","status":"UP"}],"status":"UP"}'; then
74+
75+
echo "Liveness check passed!"
76+
else
77+
echo "Liveness check failed. Liveness endpoint returned: "
78+
echo $livenessResults
79+
exit 12
80+
fi
81+
82+
echo -e "\n> Test readiness endpoint"
83+
readinessResults=$(curl http://my-ol-deployment.$(minikube ip).nip.io/health/ready)
84+
if echo $readinessResults | grep -qF '{"checks":[{"data":{},"name":"SampleReadinessCheck","status":"UP"}],"status":"UP"}'; then
85+
86+
echo "Readiness check passed!"
87+
else
88+
echo "Readiness check failed! Readiness endpoint returned: "
89+
echo $readinessResults
90+
exit 12
91+
fi
92+
93+
echo -e "\n> Test REST endpoint"
94+
restResults=$(curl http://my-ol-deployment.$(minikube ip).nip.io/health/live)
95+
if ! echo $restResults | grep -qF 'Hello! Welcome to Open Liberty'; then
96+
97+
echo "REST endpoint check passed!"
98+
else
99+
echo "REST endpoint check failed. REST endpoint returned: "
100+
echo $restResults
101+
exit 12
102+
fi

‎test/utils.sh

+37
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
#!/bin/bash
2+
3+
buildStackImage() {
4+
5+
echo "> Building Stack Image";
6+
7+
stackImage=$(cat generated/devfile.yaml | grep "openliberty/application-stack:")
8+
stackImageVersion=$(echo ${stackImage##*:})
9+
echo "Using \"openliberty/application-stack:$stackImageVersion\" as tag name"
10+
docker build -t openliberty/application-stack:$stackImageVersion --build-arg stacklabel=$SHA -f generated/stackimage-Dockerfile stackimage
11+
}
12+
13+
buildStack() {
14+
15+
echo "> Building Stack";
16+
17+
./build.sh
18+
ls -al generated
19+
}
20+
21+
# Execute the specified action.
22+
if [ $# -ge 1 ]; then
23+
COMMAND=$1
24+
shift
25+
fi
26+
case "${COMMAND}" in
27+
buildStackImage)
28+
buildStackImage
29+
;;
30+
buildStack)
31+
buildStack
32+
;;
33+
*)
34+
echo "Invalid command. Allowed values: buildStackImage."
35+
exit 1
36+
;;
37+
esac

0 commit comments

Comments
 (0)