Welcome to my GitHub repository!
This repository serves as the baseline for my lab, encompassing various experiments and setups. Here, you'll find a collection of tools and scripts that form the foundation for my OpenShift testing.
- Prerequisite
- bootstrap/cluster folder
- Ansible playbooks
- KVM Host Setup
- Mirror Registry Setup
- OC-Mirror
- Agent Based Install and build image
- Bootstrap Cluster and Operators
- The KVM host is running with CentOS 9-Stream. Also, we will use it for registry VM.
- In this lab, mirror registry for Red Hat OpenShift is used. Download mirror-registry.tar.gz.
- Install oc-mirror.
- Downalod and install the agent-based-installer.
Download CentOS-Stream-GenericCloud-9-20220718.0.x86_64.qcow2 and mirror-registry.tar.gz, and locate them under the git folder after clone.
The cluster folder serves as a central location for housing operators and configurations related to the hub cluster. As the hub cluster evolves over time, it's essential to maintain and update this folder with new operators and configurations, while also ensuring that corresponding repository updates are made, which are described as following chapters.
cluster/
├── configuration
│ ├── advanced-cluster-management
│ ├── fakefish
│ ├── odf-operator
│ ├── openshift-local-storage
│ ├── patch-operator
│ └── topology-aware-lifecycle-manager
└── operators
├── advanced-cluster-management
├── odf-operator
├── openshift-local-storage
├── patch-operator
└── topology-aware-lifecycle-manager
Four playbooks are included.
Those four playbooks drive the bootstrap up to the point that ArgoCD gitops is ready for cluster management.
| Playbook | Description |
|---|---|
| baremetal_setup_playbook.yaml | This playbook will configure bridge interface, install and configure libvirt and sushy on the KVM host |
| registry_vm_setup_playbook.yaml | This playbook will launch CentOS VM for the private registry with 500GB disk size on the KVM host. |
| registry_setup_playbook.yaml | This playbook will install mirror-registry into the registry VM with self-signed cert. |
| vm_setup_playbook.yaml | This playbook will launch Openshift cluster based on the image generated by Agent-based-installer. |
Currently I have modifications and manual steps between playbooks in order to make the setup generic and simplified. However, as the lab evolves, my intention is to automate these manual steps to improve reproducibility and eliminate human error.
First step is setting up a KVM host using an Ansible playbook. This playbook will create a bridge interface, install libvirtd and configure Sushy, a tool used to emulate baremetal nodes, by leveraging the containerized Sushy tool based on Brandon's configuration.
$ ansible-playbook -i inventry baremetal_setup_playbook.yaml
To configure access for a CentOS instance using cloud-init, you need to create a cloud-init ISO image containing the necessary user data and meta-data.
$ mkisofs -output init.iso -volid cidata -joliet -rock {user-data,meta-data}
Proceed with setting up a private registry for our clusters. Launch a CentOS VM using the Ansible playbook.
$ ansible-playbook -i inventry registry_vm_setup_playbook.yaml
Before configuring the mirror-registry, we need to set up the SSL certificate to secure our private registry. Follow the steps in the instructions under the "ssl" folder.
To install and configure the mirror-registry with SSL certificate, run the Ansible playbook as follows.
$ ansible-playbook -i inventry registry_setup_playbook.yaml
We use the oc-mirror OpenShift CLI (oc) plugin to mirror images to the mirror registry. To push images to the mirror-registry with oc-mirror, follow these steps:
-
In the "mirror" folder, you should find an "image-setconfig.yaml" file. Edit this file according to your needs to specify the operators and their channels that you want to mirror. Refer to the readme in the mirror folder.
-
Run the following command to push the specified images to your mirror-registry:
$ cd mirror
$ oc-mirror --config=./imageset-config.yaml docker://registry1.cotton.blue:8443/mirror --continue-on-error
Under the "installer" folder, you should find the "install-config.yaml" and "agent-config.yaml" files. Make sure these files are properly configured according to your desired OpenShift installation settings and agent configurations.
Run the following command to build the install image:
openshift-install --dir manifests agent create image
Copy the agent.x86_64.iso image to the KVM host (veterans) using rsync:
rsync --rsync-path="sudo rsync" agent.x86_64.iso mabe@veterans:/var/lib/libvirt/images/
If you want to include extra manifests in the install image, you can follow the steps from my blog to add those manifests to the image.
This playbook will create VMs with the specified configurations for 3 master nodes and 3 worker nodes. The agent ISO will be attached to each VM, initiating the installation process for the OpenShift Cluster. The cluster installation process will begin after starting the VMs.
ansible-playbook -i inventory vm_setup_playbook.yaml -K
Great!
Now that our OpenShift Cluster is up and running, we can bootstrap the Advanced Cluster Management (ACM) and other components using the GitOps operator. \
The provided command will apply the necessary configurations to deploy the App of Apps pattern application on top of Gitops operator:
oc apply -k cluster-bootstrap/
If the oc apply -k cluster-bootstrap/ command appears to be stuck or takes an unusually long time to complete, apply each configuration separately using the -k option to narrow down the cofiguration for specific need and operator.
ansible-playbook -i inventory baremetal_setup_playbook.yaml --check -K
ansible all -i "localhost," -c local -m template -a "src=./templates/vm_master.xml.j2 dest=./test.txt" --extra-vars='{"masters": ["master1", "master2", "master3"]}'