Skip to content
This repository was archived by the owner on Dec 7, 2024. It is now read-only.

README.md

Latest commit

 

History

History
583 lines (401 loc) · 22.1 KB

README.md

File metadata and controls

583 lines (401 loc) · 22.1 KB
Raw

Building CI/CD Pipelines

Table of Contents
  1. Introduction
  2. Course Information
  3. Information about the Project
  4. What I have done as part of the project
  5. Getting Started
  6. Contact

Introduction

This repository was created as part of IBM's "Continuous Integration and Delivery (CI/CD)" course.
It's a project about building a CI/CD pipeline.
Much of the code was cloned from the IBM repository: https://github.com/ibm-developer-skills-network/wtecc-CICD_PracticeCode

Preview images of the project:

7 start pipelinerun

1 Tekton Hub git clone

3 running pipeline yaml

(back to top)



Course Information

Title: Continuous Integration and Delivery (CI/CD)
Type: Practice Project
Course Provider: IBM

(back to top)



Information about the Project

General

  • Client: Myself
  • Project Goal: Building a CI/CD pipeline. Gain a deeper understanding of Continous Integration and Continous Delivery.
  • Number of Project Participants: 1 (Cloned repository of IBM. Developed the rest on my own)
  • Time Period: November - December, 2024
  • Industry / Area: DevOps
  • Role: Developer
  • Languages: English
  • Result: CI/CD pipeline successfully built. Improved understanding of implementing Continous Integration and Continous Delivery.

Tech Stack

With regard to my role:

  • CI/CD Tool: GitHub Actions
  • CI/CD Tool: Tekton
  • Container Orchestration: Kubernetes
  • Container Orchestration: Red Hat OpenShift
  • IBM Cloud IDE (based on Theia and Container)

(back to top)



What I have done as part of the project

Task 1 - Setup of GitHub Actions and implement a workflow

First implemented the setup for GitHub Actions: The creation of a .github/workflows folder with a workflow YML file.
The name of the workflow as well as the events, the job and the runner including the container were defined.

Implement empty workflow

If you check the Actions tab on the GitHub website, you will see failed workflows, as the workflow has been triggered but not yet fully implemented.

Failed workflows

After the workflow file was created, the steps were added:

3 Adding Steps to workflow

The changes were pushed into the repository. This triggered the workflow again.
It is now running successfully thanks to the implemented steps:

4 Workflow successful

After clicking on the build job, exact details of the individual steps can be viewed:

5 Details of Workflow Steps

(back to top)



Task 2 - Building a Tekton Pipeline

The first step of the task was to create a traditional Hello World program / pipeline in Tekton.

The Tekton object "Task" is used for this.
According to the Tekton documentation, a Task is a collection of Steps that you define and arrange in a specific order of execution as part of your continuous integration flow.

The corresponding Task has been implemented in the tasks.yaml file:

Implement tasks yaml

It is then applied to the cluster with the following command:

kubectl apply -f tasks.yaml

The Task has been successfully created:

2 Task created

Next, a Tekton object called "Pipeline" is created that uses and executes the Task object.
The implementation is in the pipeline.yaml:

3 Implement pipeline yaml

It is then applied to the cluster with the following command:

kubectl apply -f pipeline.yaml

The Pipeline has been successfully created:

4 Pipeline created

The Pipeline is now executed with the following command:

tkn pipeline start --showlog hello-world-pipeline

Output of the Pipeline:

5 Run pipeline and display output

Parameters are now added to make the pipeline more flexible.
Both the task and the pipeline are changed for this.

The modified tasks.yaml file:

6 Add params to task

The changes are then applied to the cluster with the following command:

kubectl apply -f tasks.yaml

The Task has been successfully created:

7 Task created

The modified pipeline.yaml file:

7 5 Add params to pipeline yaml

The changes are then applied to the cluster with the following command:

kubectl apply -f pipeline.yaml

The Pipeline has been successfully created:

8 Pipeline update configured

The Pipeline is now executed with the following command:

tkn pipeline start hello-world-pipeline \
  --showlog \
  -p message="Hello Tekton! I added params to pipeline!"

Output of the Pipeline:

9 Output of updated pipeline

To embellish the whole thing a little, another Pipeline is added: cd-pipeline with checkout Task.
The checkout task with parameters and the clone command have been added to the tasks.yaml file:

10 Add task checkout to tasks yaml

The output after applying to the cluster with the following command:

kubectl apply -f tasks.yaml

11 Apply new tasks yaml

The pipeline.yaml file has been extended by the cd-pipeline:

12 Add cd pipeline

The output after applying to the cluster with the following command:

kubectl apply -f pipeline.yaml

13 Apply new pipeline

If you execute the new Pipeline with the following command, the output appears:

tkn pipeline start cd-pipeline \
    --showlog \
    -p repo-url="https://github.com/christian-schw/cicd-practice-project.git" \
    -p branch="main"

14 Output new pipeline checkout

As a final exercise in this section, four placeholder Tasks (lint, tests, build and deploy) are added to the cd-pipeline.
The echo Task already created is used as a placeholder.
The placeholders are replaced in the following sections below.

The modified pipeline.yaml:

15 Add placeholder echo tasks to pipeline

The output after applying the changes to the cluster and executing the following command:

tkn pipeline start cd-pipeline \
    --showlog \
    -p repo-url="https://github.com/christian-schw/cicd-practice-project.git" \
    -p branch="main"

16 Output placeholder echo pipeline

(back to top)



Task 3 - Adding GitHub Triggers

Since it is tedious and offers limited possibilities to run the pipeline manually, Tekton objects EventListener, TriggerBinding and TriggerTemplate are created.
This makes it possible for the pipeline to be started automatically by external events (e.g. changes in the GitHub repository).
The corresponding files are located in the folder: ../labs/02_add_git_trigger/

First, the EventListener was implemented to be able to listen to events on GitHub:

1 Add eventlistener yaml

The output after applying to the cluster with the following command:

kubectl apply -f eventlistener.yaml

2 Verify tekton eventlistener

Next, the TriggerBinding was implemented to be able to bind the incoming data from the event to pass on to the pipeline:

3 Add triggerbinding yaml

The output after applying to the cluster with the following command:

kubectl apply -f triggerbinding.yaml

4 Verify tekton triggerbinding

The TriggerTemplate takes the parameters passed in from the TriggerBinding and creates a PipelineRun to start the pipeline:

5 Implement triggertemplate yaml

The output after applying to the cluster with the following command:

kubectl apply -f triggertemplate.yaml

6 Verify tekton triggertemplate

To test the Pipeline Run, the curl command is used (Note: localhost port of IBM Cloud IDE is 8090):

curl -X POST http://localhost:8090 \
  -H 'Content-Type: application/json' \
  -d '{"ref":"main","repository":{"url":"https://github.com/christian-schw/cicd-practice-project.git"}}'

The output:

7 start pipelinerun

The red marked PipelineRun is the pipeline in this task. Everything works!

(back to top)



Task 4 - Using Tekton Continous Delivery Catalog

The self-created clone task is very small and hardly robust.
To improve the whole thing, a prefabricated task in the Tekton Hub is integrated into our pipeline.

1 Tekton Hub git clone

With the help of the Tekton CLI, this task is installed in our cluster in the current active namespace:

2 Tekton Hub install git clone

The task requires two inputs:

  • The URL of a Git repo to clone ('url' param)
  • A workspace / disk volume that can be shared across tasks ('output' param)

To create the workspace, the PersistentVolumeClaim file (pvc.yaml) already predefined by the IBM course is used:

3 provided pvc yaml

The output after applying to the cluster with the following command:

kubectl apply -f pvc.yaml

4 verifying pvc

This persistent volume can now be referenced by its name 'pipelinerun-pvc' when creating workspaces for the Tekton tasks.

The pipeline.yaml is then changed so that the Tekton Hub Task is now used instead of the self-created task.
At the same time, the workspace is defined (as a required parameter according to Tekton Hub):

5 changing pipeline yaml

The output after applying to the cluster with the following command:

kubectl apply -f pipeline.yaml

6 verifying pipeline

A PipelineRun is then created to run the pipeline:

tkn pipeline start cd-pipeline \
    -p repo-url="https://github.com/christian-schw/cicd-practice-project.git" \
    -p branch="main" \
    -w name=pipeline-workspace,claimName=pipelinerun-pvc \
    --showlog

The logs already show significantly more information than before:

7 starting the pipeline

The pipeline was successfully executed and the Tekton Hub Task integrated:

8 pipeline succeeded

(back to top)



Task 5 - Integrating Linting and Unit Test Automation

Two further tasks in the pipeline (folder: /labs/04_unit_test_automation/) are replaced, which previously acted as placeholders: lint and tests.

First the lint task.
Flake8 is used for the linting. There is already a predefined task for this in the Tekton Hub, which will be installed:

1 installing flake8

The existing task reference in the pipeline is adapted and the corresponding required task parameters are implemented:

2 implemented pipeline yaml labs04

The output after applying to the cluster with the following command:

kubectl apply -f pipeline.yaml

3 apply pipeline yaml

If you run the pipeline with the following command, you will see that the new task is working and is being executed:

tkn pipeline start cd-pipeline \
    -p repo-url="https://github.com/christian-schw/cicd-practice-project.git" \
    -p branch="main" \
    -w name=pipeline-workspace,claimName=pipelinerun-pvc \
    --showlog

4 running pipeline with flake8 task

Linting is done, now the unit test task will be implemented.

The testing framework nose is used.
However, there is nothing ready-made for this in the Tekton Hub.
This means that I have to define this task myself:

5 adding nose task

The existing task reference in the pipeline is adapted and the corresponding required task parameters are implemented:

6 changing taskref in pipeline yaml

The changes are then applied to the cluster with the following command:

kubectl apply -f pipeline.yaml

The output after running the pipeline with the following command:

tkn pipeline start cd-pipeline \
    -p repo-url="https://github.com/christian-schw/cicd-practice-project.git" \
    -p branch="main" \
    -w name=pipeline-workspace,claimName=pipelinerun-pvc \
    --showlog

7 running pipeline

The status of the PipelineRun object is succeeded.
Everything works as intended!

8 status succeeded pipelinerun

(back to top)



Task 6 - Building an Image

The next placeholder task in the pipeline to be replaced is the build task.
Before the application can be deployed, it's necessary to build a Docker image and push it to an image registry (in this case: OpenShift Image Registry).
The Tekton Hub buildah task is used for this.

First of all, it is necessary to check whether the task has already been installed in the cluster.
This step was previously omitted in the previous tasks (for learning reasons).

Tasks that are available for pipelines in the entire cluster are called ClusterTasks in Tekton.
According to the Tekton documentation, ClusterTasks are deprecated and cluster resolvers should be used instead.
A ClusterTask is used / required in the practice project, so a ClusterTask is also used here.

The output after running the pipeline with the following command:

tkn clustertask ls

1 Check for buildah ClusterTask

The buildah task is listed. This means that it has already been installed and can be used for the pipeline.
The task reference in the pipeline is then changed to the buildah task (folder: /labs/05_build_an_image/):

2 part 1 of changing pipeline yaml

2 part 2 of changing pipeline yaml

The changes are then applied to the cluster with the following command:

kubectl apply -f pipeline.yaml

It has also been checked whether the PVC for the workspace exists.
The output:

3 applying pipeline and verifying pvc

The extended pipeline is now executed with the following command and the logs are examined:

tkn pipeline start cd-pipeline \
    -p repo-url="https://github.com/christian-schw/cicd-practice-project.git" \
    -p branch=main \
    -p build-image=image-registry.openshift-image-registry.svc:5000/$SN_ICR_NAMESPACE/tekton-lab:latest \
    -w name=pipeline-workspace,claimName=pipelinerun-pvc \
    --showlog

4 running pipeline

To make sure that everything worked, we also check whether the image exists in the OpenShift image registry - and yes, it does! Everything fits.

5 verifying image in openshift image registry

(back to top)



Task 7 - Deploy to OpenShift

The last step of this pipeline: Deploy the Docker image to an OpenShift cluster.
There is an already defined task for this in the Tekton Hub: openshift-client.

First check whether the (cluster) task openshift-client is already installed.
The output after the following command:

tkn clustertask ls

1 check for openshift-client clustertask

It's already installed.
The task reference in the pipeline is then changed to the openshift-client task (folder: /labs/06_deploy_to_kubernetes/):

2 part 1 change pipeline yaml

2 part 2 change pipeline yaml

The extended pipeline is now executed with the following command and the logs are examined:

tkn pipeline start cd-pipeline \
    -p repo-url="https://github.com/christian-schw/cicd-practice-project.git" \
    -p branch=main \
    -p app-name=hitcounter \
    -p build-image=image-registry.openshift-image-registry.svc:5000/$SN_ICR_NAMESPACE/tekton-lab:latest \
    -w name=pipeline-workspace,claimName=pipelinerun-pvc \
    --showlog

3 running pipeline yaml

Finally, check whether the deployment has worked.

4 verifying successful deployment

The pod of the deployment is running which means the application has been successfully deployed!
The pipeline is now fully implemented and the practice project is complete.

(back to top)



Getting Started

The code was only executable with the help of the IBM Cloud IDE.
The IDE had many necessary tools and accesses preconfigured. These include, for example, Kubernetes installation or IBM Cloud Container Registry.

(back to top)



Contact

If you have any questions, please feel free to reach out via email: christian-schwanse (at) gmx.net

(back to top)