fix(recipe): fix issues due to job to recipe change (#100)

This commit has changes that fixes the manifests & code that were
still referring to old job custom resource.

In addition, this commit provides a way to trigger E to E testing
as a make target. This has also been added to github actions to
proactively verify any errors injected by changes during the review
phase itself.

E to E testing make use of local docker registry and K3s to test
D-operators' controllers.

README has been updated to highlight that fact that D-operators uses
itself to perform end to end tests on its controllers. In other words
Recipe custom resource is used to test Recipe & other resources.

Signed-off-by: AmitKumarDas <[email protected]>

Author: Amit Kumar Das

.github/workflows/release-docker.yaml

@@ -1,17 +1,17 @@
 ---
-name: Release docker image
+name: Release Docker Image
 on:
   push:
     tags:
       - 'v[0-9]+.[0-9]+.[0-9]+'
 jobs:
   release-docker:
-    name: Release docker image
+    name: Release Docker Image
     runs-on: ubuntu-18.04
     steps:
-      - name: Checkout
+      - name: Checkout Code
         uses: actions/checkout@v2
-      - name: Deploy docker image
+      - name: Setup Docker
         uses: docker/build-push-action@v1
         with:
           username: ${{ secrets.DOCKER_USERNAME }}

.github/workflows/test-release.yaml

@@ -1,39 +1,57 @@
 ---
-name: Test and Release (if on master)
+name: Unit Test, E to E Test and Github Release (if on master)
 on: [push, pull_request]
 jobs:
-  tests:
+  unittest:
     runs-on: ubuntu-18.04
     strategy:
       matrix:
         test: ['test']
     name: ${{ matrix.test }}
     steps:
-      - name: Checkout
+      - name: Checkout Code
         uses: actions/checkout@v2
-      - name: setup env
+      - name: Setup GOPATH
         run: |
           echo "::set-env name=GOPATH::$(go env GOPATH)"
           echo "::add-path::$(go env GOPATH)/bin"
-      - name: Setup go
+      - name: Setup Golang
         uses: actions/setup-go@v1
         with:
           go-version: 1.13.5
       - run: make ${{ matrix.test }}
+  e2etest:
+    runs-on: ubuntu-18.04
+    strategy:
+      matrix:
+        test: ['e2e-test']
+    name: ${{ matrix.test }}
+    steps:
+      - name: Checkout Code
+        uses: actions/checkout@v2
+      - name: Setup GOPATH
+        run: |
+          echo "::set-env name=GOPATH::$(go env GOPATH)"
+          echo "::add-path::$(go env GOPATH)/bin"
+      - name: Setup Golang
+        uses: actions/setup-go@v1
+        with:
+          go-version: 1.13.5
+      - run: sudo make ${{ matrix.test }}
   release:
-    name: Release
+    name: Make Github Release
     runs-on: ubuntu-18.04
-    needs: ['tests']
+    needs: ['unittest', 'e2etest']
     steps:
-      - name: Checkout
+      - name: Checkout Code
         uses: actions/checkout@v1
       - name: Setup Node.js
         uses: actions/setup-node@v1
         with:
           node-version: 12
-      - name: Install dependencies
+      - name: Install NPM Dependencies to Make Release
         run: npm install ci
-      - name: Release
+      - name: Make Semantic Release
         env:
           GH_TOKEN: ${{ secrets.PAT }}
         run: npx semantic-release

.gitignore

@@ -1,4 +1,5 @@
 /vendor/
 d-operators
 test/bin/
-test/kubebin/
+test/kubebin/
+test/e2e/uninstall-k3s.txt

Dockerfile

@@ -28,7 +28,6 @@ COPY common/ common/
 COPY config/ config/
 COPY controller/ controller/
 COPY pkg/ pkg/
-COPY test/ test/
 COPY types/ types/
 
 # test d-operators
@@ -65,11 +64,6 @@ COPY controller/ controller/
 COPY pkg/ pkg/
 COPY types/ types/
 
-# we run the test once again since this is one of the
-# ways to remind copying new source packages into this 
-# build stage
-RUN make test
-
 # build binary
 RUN make
 

Makefile

@@ -41,11 +41,14 @@ test:
 testv:
 	@go test ./... -cover -v -args --logtostderr -v=2
 
+.PHONY: e2e-test
+e2e-test: 
+	@cd test/e2e && ./suite.sh
+
 .PHONY: image
 image: $(GIT_TAGS)
 	docker build -t $(IMG_REPO):$(PACKAGE_VERSION) .
 
-
 .PHONY: push
 push: image
-	docker push $(IMG_REPO):$(PACKAGE_VERSION)
+	docker push $(IMG_REPO):$(PACKAGE_VERSION

README.md

@@ -1,20 +1,42 @@
-## d-operators
+## D-operators
 D-operators define various declarative patterns to write kubernetes controllers. This uses [metac](https://github.com/AmitKumarDas/metac/) under the hood. Users can _create_, _delete_, _update_, _assert_, _patch_, _clone_, & _schedule_ one or more kubernetes resources _(native as well as custom)_ using a yaml file. D-operators expose a bunch of kubernetes custom resources that provide the building blocks to implement a higher order controller.
 
 D-operators follow a pure intent based approach to writing specifications **instead of** having to deal with yamls that are cluttered with scripts, kubectl, loops, conditions, templating and so on.
 
 ### Programmatic vs. Declarative
-It is important to understand that these declarative patterns are built upon programmatic ones. The low level constructs _(read native k8s resources & custom resources)_ might be implemented in programming language(s) of one's choice. Use d-controller's YAMLs to aggregate these low level resources in a particular way to build a completely new kubernetes controller.
+It is important to understand that these declarative patterns are built upon programmatic ones. The low level constructs _(read native Kubernetes resources & custom resources)_ might be implemented in programming language(s) of one's choice. Use d-controller's YAMLs to aggregate these low level resources in a particular way to build a completely new kubernetes controller.
 
-### When to use d-operators
-D-operators is not meant to build complex controllers like Deployment, StatefulSet or Pod in a declarative yaml. However, if one needs to use Deployment, StatefulSet, Pod, etc. to build new k8s controller(s) then d-operators' declarative constructs _(read custom resources)_ should be considered to build one. 
+### When to use D-operators
+D-operators is not meant to build complex controller logic like Deployment, StatefulSet or Pod in a declarative yaml. However, if one needs to use available Kubernetes resources to build new k8s controller(s) then d-operators should be considered to build one. D-operators helps implement the last mile automation needed to manage applications & infrastructure in Kubernetes clusters.
 
-However, any controller which is complex and at the same time is **generic** enough to be used along with other kubernetes resources, can be implemented as a core d-operator custom resource.
+### E to E testing
+D-operators make use of d-operators _(i.e. its own self)_ to test its controllers. It does not need kubectl, bash, sed, awk etc to test its controllers. In addition, it does not depend on writing go code to write tests. It makes use of declarative YAMLs to test its controllers.
 
-### Available controllers
+_NOTE: One can make use of these YAMLs (kind: Recipe) to test any Kubernetes controllers declaratively_
+
+Navigate to test/experiments to learn more on these YAMLs.
+
+```sh
+# Following runs the e2e test suite
+#
+# NOTE: test/e2e/suite.sh does the following:
+# - d-operators' image known as 'dope' is built
+# - a docker container is started & acts as the image registry
+# - dope image is pushed to this image registry
+# - k3s is installed with above image registry
+# - d-operators' manifests are applied
+# - experiments _(i.e. test code written as YAMLs)_ are applied
+# - experiments are asserted
+# - if all experiments pass then e2e is a success else it failed
+# - k3s is un-installed
+# - local image registry is stopped
+sudo make e2e-test
+```
+
+### Available Kubernetes controllers
 - [x] kind: Recipe
 - [ ] kind: HTTP
-- [ ] kind: Chef
+- [ ] kind: HTTPPipe
 - [ ] kind: Command
 - [ ] kind: DaemonJob
-- [ ] kind: MasterChef
+- [ ] kind: UberLoop

config/metac.yaml

@@ -1,14 +1,14 @@
 apiVersion: metac.openebs.io/v1alpha1
 kind: GenericController
 metadata:
-  name: sync-job
+  name: sync-recipe
   namespace: dope
 spec:
-  watch:
-    apiVersion: metacontroller.app/v1
-    resource: jobs
+  watch: # kind: Recipe custom resource is watched
+    apiVersion: dope.metacontroller.io/v1
+    resource: recipes
   hooks:
     sync:
       inline:
-        funcName: sync/job
+        funcName: sync/recipe
 ---