|
1 | | -# longhorn_kubernetes |
2 | | -Ansible role to install Longhorn cloud native distributed block storage for Kubernetes |
| 1 | +<!-- |
| 2 | +Copyright (C) 2023 Robert Wimmer |
| 3 | +SPDX-License-Identifier: GPL-3.0-or-later |
| 4 | +--> |
| 5 | + |
| 6 | +ansible-role-longhorn_kubernetes |
| 7 | +================================ |
| 8 | + |
| 9 | +This Ansible role installs [Longhorn](https://longhorn.io/) cloud native distributed block storage on a Kubernetes cluster. Behind the doors it uses the official [Helm chart](https://charts.longhorn.io). Currently procedures like installing, upgrading and deleting the `Longhorn` deployment are supported. |
| 10 | + |
| 11 | +Versions |
| 12 | +-------- |
| 13 | + |
| 14 | +I tag every release and try to stay with [semantic versioning](http://semver.org). If you want to use the role I recommend to checkout the latest tag. The master branch is basically development while the tags mark stable releases. But in general I try to keep master in good shape too. A tag `0.1.0+1.4.0` means this is release `0.1.0` of this role and it contains Longhorn chart version `1.4.0` (which normally is the same as the Longhorn version itself). If the role itself changes `X.Y.Z` before `+` will increase. If the Longhorn chart version changes `X.Y.Z` after `+` will increase too. This allows to tag bugfixes and new major versions of the role while it's still developed for a specific Longhorn release. |
| 15 | + |
| 16 | +Requirements |
| 17 | +------------ |
| 18 | + |
| 19 | +You need to have [Helm 3](https://helm.sh/) binary installed on that host where `ansible-playbook` is executed or on that host where you delegated the playbooks to (e.g. by using `longhorn_delegate_to` variable). You can either |
| 20 | + |
| 21 | +- use your favorite package manager if your distribution includes `helm` in its repository (for Archlinux use `sudo pacman -S helm` e.g.) |
| 22 | +- or use one of the Ansible `Helm` roles (e.g. [helm](https://galaxy.ansible.com/gantsign/helm) - which gets also installed if you use `ansible-galaxy role install -vr requirements.yml` |
| 23 | +- or directly download the binary from [Helm releases)[https://github.com/helm/helm/releases]) and put it into `/usr/local/bin/` directory e.g. |
| 24 | + |
| 25 | +A properly configured `KUBECONFIG` is also needed (which is located at `${HOME}/.kube/config` by default). Normally if `kubectl` works with your K8s cluster then everything should be already fine in this regards. |
| 26 | + |
| 27 | +Additionally the Ansible `kubernetes.core` collection needs to be installed. This can be done by using the `collections.yml` file included in this role: `ansible-galaxy install -r collections.yml`. At least `kubernetes.core >= v2.3.0` is needed. |
| 28 | + |
| 29 | +And of course you need a Kubernetes Cluster ;-) |
| 30 | + |
| 31 | +Additional remarks |
| 32 | +------------------ |
| 33 | + |
| 34 | +By default Longhorn uses the path `/var/lib/longhorn` on the K8s worker nodes to store data or replicas of data. First it's a good idea to just keep that path if possible. The same is true for the default namespace `longhorn-system`. This might save you from some trouble later. Second since Longhorn doesn’t currently support sharding between the different disks, its recommend using LVM to aggregate all the disks for Longhorn into a single partition, so it can be easily extended in the future. This role doesn't manage LVM but you can use my [LVM role](https://github.com/githubixx/ansible-role-lvm) or any other Ansible LVM role or whatever automation tool you want or just configure everything manually. The Molecule test (see below) contains an example LVM setup. |
| 35 | + |
| 36 | +Changelog |
| 37 | +--------- |
| 38 | + |
| 39 | +See [CHANGELOG.md](https://github.com/githubixx/ansible-role-longhorn-kubernetes/blob/master/CHANGELOG.md) |
| 40 | + |
| 41 | +Role variables |
| 42 | +-------------- |
| 43 | + |
| 44 | +```yaml |
| 45 | +# Helm chart version |
| 46 | +longhorn_chart_version: "1.4.0" |
| 47 | + |
| 48 | +# Helm release name |
| 49 | +longhorn_release_name: "longhorn" |
| 50 | + |
| 51 | +# Helm repository name |
| 52 | +longhorn_repo_name: "longhorn" |
| 53 | + |
| 54 | +# Helm chart name |
| 55 | +longhorn_chart_name: "{{ longhorn_repo_name }}/{{ longhorn_release_name }}" |
| 56 | + |
| 57 | +# Helm chart URL |
| 58 | +longhorn_chart_url: "https://charts.longhorn.io" |
| 59 | + |
| 60 | +# Kubernetes namespace where Longhorn resources should be installed |
| 61 | +longhorn_namespace: "longhorn-system" |
| 62 | + |
| 63 | +# By default all tasks that needs to communicate with the Kubernetes |
| 64 | +# cluster are executed on your local host (127.0.0.1). But if that one |
| 65 | +# doesn't have direct connection to this cluster or should be executed |
| 66 | +# elsewhere this variable can be changed accordingly. |
| 67 | +longhorn_delegate_to: "127.0.0.1" |
| 68 | + |
| 69 | +# Shows the "helm" command that was executed if a task uses Helm to |
| 70 | +# install, update/upgrade or deletes such a resource. |
| 71 | +longhorn_helm_show_commands: false |
| 72 | + |
| 73 | +# Without "longhorn_action" variable defined this role will only render a file |
| 74 | +# with all the resources that will be installed or upgraded. The rendered |
| 75 | +# file with the resources will be called "template.yml" and will be |
| 76 | +# placed in the directory specified below. The default will create a directory |
| 77 | +# "longhorn/template" in users "$HOME" directory. |
| 78 | +longhorn_template_output_directory: "{{ '~/longhorn/template' | expanduser }}" |
| 79 | + |
| 80 | +# The Ansible group name of the nodes where Longhorn will store data |
| 81 | +# or replicas of the data. As the role needs to install a few OS packages |
| 82 | +# on that nodes the role needs to know on which hosts the tools/packages |
| 83 | +# are needed. E.g. if Longhorn should only be installed on nodes with a |
| 84 | +# specific label or taint then only these nodes need the packages installed. |
| 85 | +longhorn_nodes: "k8s_longhorn" |
| 86 | + |
| 87 | +# Enable multipathd blacklist. For more information see: |
| 88 | +# https://longhorn.io/kb/troubleshooting-volume-with-multipath/ |
| 89 | +# longhorn_multipathd_blacklist_directory: "/etc/multipath/conf.d" |
| 90 | +# longhorn_multipathd_blacklist_directory_perm: "0755" |
| 91 | +# longhorn_multipathd_blacklist_file: "10-longhorn.conf" |
| 92 | +# longhorn_multipathd_blacklist_file_perm: "0644" |
| 93 | +``` |
| 94 | + |
| 95 | +Usage |
| 96 | +----- |
| 97 | + |
| 98 | +Before you start installing Longhorn you REALLY want to read the [The Longhorn Documentation](https://longhorn.io/docs/1.4.0/)! As data is the most valuable thing you can have you should understand how Longhorn works and don't forget to add backups later ;-). Esp. have a look at the [best practices](https://longhorn.io/docs/1.4.0/best-practices/). |
| 99 | + |
| 100 | +That said: The first thing to do is to check `templates/longhorn_values_default.yml.j2`. This file contains the values/settings for the Longhorn Helm chart that are partly default anyways (just to avoid that someone changes the defaults) or different to the default ones which are located [here](https://github.com/longhorn/longhorn/blob/v1.4.0/chart/values.yaml). All settings can be found in the [Settings Reference](https://longhorn.io/docs/1.4.0/references/settings/). |
| 101 | + |
| 102 | +To use your own values just create a file called `longhorn_values_user.yml.j2` and put it into the `templates` directory. Then this Longhorn role will use that file to render the Helm values. You can use `templates/longhorn_values_default.yml.j2` as a template or just start from scratch. As mentioned above you can modify all settings for the Longhorn Helm chart that are different to the default ones which are located [here](https://github.com/longhorn/longhorn/blob/v1.4.0/chart/values.yaml). |
| 103 | + |
| 104 | +After the values file (`templates/longhorn_values_default.yml.j2` or `templates/longhorn_values_user.yml.j2`) is in place and the `defaults/main.yml` values are checked and maybe adjusted accordingly, the role can be installed. Quite a few tasks need to communicate with the Kubernetes API server or executing [Helm](https://helm.sh/) commands. By default these commands are executed on the host where the `ansible-playbook` gets executed and the current user is used. But you can delegate this kind of tasks to a different host by using `longhorn_delegate_to` variable (see above). |
| 105 | + |
| 106 | +The default action is to just render the Kubernetes resources YAML file after replacing all Jinja2 variables and stuff like that. In the `Example Playbook` section below there is an `Example 2 (assign tag to role)`. The role `githubixx.longhorn_kubernetes` has a tag `role-longhorn-kubernetes` assigned and all further `ansible-playbook` examples below will refer to `Example 2`! |
| 107 | + |
| 108 | +Assuming that the values for the Helm chart should be rendered (nothing will be installed in this case) and the playbook is called `k8s.yml` execute the following command: |
| 109 | + |
| 110 | +```bash |
| 111 | +ansible-playbook --tags=role-longhorn-kubernetes k8s.yml |
| 112 | +``` |
| 113 | + |
| 114 | +To render the template into a different directory change `longhorn_template_output_directory` variable value e.g.: |
| 115 | + |
| 116 | +```bash |
| 117 | +ansible-playbook --tags=role-longhorn-kubernetes --extra-vars longhorn_template_output_directory="/tmp/longhorn" k8s.yml |
| 118 | +``` |
| 119 | + |
| 120 | +If you want to see the `helm` commands and the parameters which were executed in the logs you can also specify `--extra-vars longhorn_helm_show_commands=true`. |
| 121 | + |
| 122 | +One of the final tasks is called `TASK [githubixx.longhorn_kubernetes : Write templates to file]`. This renders the template with the resources that will be created into the directory specified in `longhorn_template_output_directory`. The file will be called `template.yml`. The directory/file will be placed either on your local machine or on the host specified with `longhorn_delegate_to`. |
| 123 | + |
| 124 | +If the rendered output contains everything you need, the role can be installed which finally deploys Longhorn: |
| 125 | + |
| 126 | +```bash |
| 127 | +ansible-playbook --tags=role-longhorn-kubernetes --extra-vars longhorn_action=install k8s.yml |
| 128 | +``` |
| 129 | + |
| 130 | +To check if everything was deployed use the usual `kubectl` commands like `kubectl -n <longhorn_namespace> get pods -o wide`. The first installation will take quite some time if your internet connection isn't the fastest one. Lots of container images need to be downloaded. |
| 131 | + |
| 132 | +As Longhorn gets updates/upgrades every few weeks/months the role also can do upgrades. For updates/upgrades (esp. major upgrades) have a look at `tasks/upgrade.yml` to see what's happening before, during and after the update. Of course you should consult Longhorn's [upgrade guide](https://longhorn.io/docs/1.4.0/deploy/upgrade/) (the link is for upgrading to Longhorn `v1.4.0`) to check for major changes and stuff like that before upgrading. Now is also a good time to check if the backups are in place and if the backups are actually valid ;-) |
| 133 | + |
| 134 | +After consulting Longhorn's [upgrade guide](https://longhorn.io/docs/1.4.0/deploy/upgrade/) you basically only need to change `longhorn_chart_version` variable e.g. from `1.4.0` to `1.4.1` for a patch release or from `1.4.0` to `1.5.0` for a major upgrade. And of course the Helm values need to be adjusted for potential breaking changes (if any are mentioned in the upgrade guide e.g.). |
| 135 | + |
| 136 | +You can also use the upgrade method if you keep the version number and just want to change some Helm values or other settings. But please be aware that changing some of settings might have some serious consequences if you already have volumes deployed! Not all Longhorn settings can be changed just by changing a number or a string. So you really want to consult the [Settings reference](https://longhorn.io/docs/1.4.0/references/settings/) to figure out what might happen if you change this or that setting or what you need to do before you apply a changed setting! |
| 137 | + |
| 138 | +That said to actually do the update/upgrade run |
| 139 | + |
| 140 | +```bash |
| 141 | +ansible-playbook --tags=role-longhorn-kubernetes --extra-vars longhorn_action=upgrade k8s.yml |
| 142 | +``` |
| 143 | + |
| 144 | +And finally if you REALLY want to get rid of Longhorn you can delete all Longhorn resources again. Make sure that you have backups! Nobody will be able to recover/restore data that was deleted and there is no backup! Make sure that all Longhorn volumes are gone before you delete everything! That said you have to provide two variables in order do be able to delete Longhorn resources via this role e.g.: |
| 145 | + |
| 146 | +```bash |
| 147 | +ansible-playbook \ |
| 148 | + --tags=role-longhorn-kubernetes \ |
| 149 | + --extra-vars longhorn_action=delete \ |
| 150 | + --extra-vars longhorn_delete=true \ |
| 151 | + k8s.yml |
| 152 | +``` |
| 153 | + |
| 154 | +Longhorn has a [Deleting Confirmation Flag](https://longhorn.io/docs/1.4.0/references/settings/#deleting-confirmation-flag) which is set to `false` by default. In this case Longhorn refuses to be uninstalled. By setting `--extra-vars longhorn_delete=true` the Ansible role will set this flag to `true` and afterwards the Longhorn resources can be deleted by the role. Without `longhorn_delete` variable the role will refuse to finish uninstallation. |
| 155 | + |
| 156 | +Example Playbook |
| 157 | +---------------- |
| 158 | + |
| 159 | +Example 1 (without role tag): |
| 160 | + |
| 161 | +```yaml |
| 162 | +--- |
| 163 | +- name: Setup Longhorn |
| 164 | + hosts: longhorn |
| 165 | + tasks: |
| 166 | + - name: Include Longhorn role |
| 167 | + ansible.builtin.include_role: |
| 168 | + name: githubixx.longhorn_kubernetes |
| 169 | +``` |
| 170 | +
|
| 171 | +Example 2 (assign tag to role): |
| 172 | +
|
| 173 | +```yaml |
| 174 | +--- |
| 175 | +- |
| 176 | + hosts: longhorn |
| 177 | + roles: |
| 178 | + - |
| 179 | + role: githubixx.longhorn_kubernetes |
| 180 | + tags: role-longhorn-kubernetes |
| 181 | +``` |
| 182 | +
|
| 183 | +Testing |
| 184 | +------- |
| 185 | +
|
| 186 | +This role has a (full blown) Kubernetes test setup that is created using [Molecule](https://github.com/ansible-community/molecule), libvirt (vagrant-libvirt) and QEMU/KVM. Please see my blog post [Testing Ansible roles with Molecule, libvirt (vagrant-libvirt) and QEMU/KVM](https://www.tauceti.blog/posts/testing-ansible-roles-with-molecule-libvirt-vagrant-qemu-kvm/) how to setup. The test configuration is [here](https://github.com/githubixx/ansible-role-longhorn-kubernetes/tree/master/molecule/default). |
| 187 | +
|
| 188 | +Afterwards molecule can be executed. The following command will setup a K8s cluster with 8 VMs (Three controller and four Kubernetes worker and one Ansible controller node) and render a template of the resources (default "longhorn_action" see above) that will be created (so Longhorn won't be installed by default): |
| 189 | +
|
| 190 | +```bash |
| 191 | +molecule converge |
| 192 | +``` |
| 193 | + |
| 194 | +To actually install `Longhorn` and the required resources use this command: |
| 195 | + |
| 196 | +```bash |
| 197 | +molecule converge -- --extra-vars longhorn_action=install |
| 198 | +``` |
| 199 | + |
| 200 | +Upgrading `Longhorn` or changing parameters: |
| 201 | + |
| 202 | +```bash |
| 203 | +molecule converge -- --extra-vars longhorn_action=upgrade |
| 204 | +``` |
| 205 | + |
| 206 | +Deleting `Longhorn` and its resources: |
| 207 | + |
| 208 | +```bash |
| 209 | +molecule converge -- --extra-vars longhorn_action=delete --extra-vars longhorn_delete=true |
| 210 | +``` |
| 211 | + |
| 212 | +To clean up run |
| 213 | + |
| 214 | +```bash |
| 215 | +molecule destroy |
| 216 | +``` |
| 217 | + |
| 218 | +License |
| 219 | +------- |
| 220 | + |
| 221 | +GNU GENERAL PUBLIC LICENSE Version 3 |
| 222 | + |
| 223 | +Author Information |
| 224 | +------------------ |
| 225 | + |
| 226 | +[http://www.tauceti.blog](http://www.tauceti.blog) |
0 commit comments