Skip to content

Commit 1db51d4

Browse files
Millefeuille42Millefeuille42
committed
cmd: add linstor witness script
Signed-off-by: Mathieu Labourier <mathieu.labourier@vates.tech>
1 parent 4f7a785 commit 1db51d4

7 files changed

Lines changed: 1010 additions & 0 deletions

File tree

‎src/linstor_witness/README.md‎

Lines changed: 288 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,288 @@
1+
# XOSTOR Witness scripts
2+
3+
In any case, the witness VM should not run on a node of the LINSTOR pool, but on a separate host.
4+
It is recommended to use a dedicated, possibly a remote host for the witness VM.
5+
6+
The witness VM should have access to the LINSTOR pool network and be able to communicate with the LINSTOR nodes.
7+
The witness VM will act as a TieBreaker and will help to keep the quorum of the LINSTOR pool in the case of a pool with only two nodes.
8+
9+
If the ssh keys are not provided, ssh will ask for the root password to connect to each remote host (witness VM and LINSTOR node).
10+
The script will first ask for the current password of the root user on the remote hosts when copying the ssh key if provided.
11+
Then the ssh key should be used.
12+
13+
The script should be run from the `./provision` directory.
14+
15+
## Table of contents
16+
17+
- [Deploying the witness VM](#deploying-the-witness-vm)
18+
* [On an existing VM](#on-an-existing-vm)
19+
* [From a template VM](#from-a-template-vm)
20+
- [Creating a template VM](#creating-a-template-vm)
21+
- [Updating the witness VM](#updating-the-witness-vm)
22+
* [Updating in-place](#updating-in-place)
23+
* [Using a new VM](#using-a-new-vm)
24+
+ [On an existing VM](#on-an-existing-vm-1)
25+
+ [From a template VM](#from-a-template-vm-1)
26+
- [Setting up paths and interfaces](#setting-up-paths-and-interfaces)
27+
* [Preparing the network path with a `PrefNIC` set on the nodes](#preparing-the-network-path-with-a-prefnic-set-on-the-nodes)
28+
* [Alternative: Preparing the network path removing the `PrefNIC` from the nodes](#alternative-preparing-the-network-path-removing-the-prefnic-from-the-nodes)
29+
* [If there is no `PrefNIC` set on any node](#if-there-is-no-prefnic-set-on-any-node)
30+
31+
## Deploying the witness VM
32+
33+
### On an existing VM
34+
35+
This script will provision a witness VM on an existing VM.
36+
It requires an existing LINSTOR pool and an AlmaLinux >=9 VM.
37+
38+
```bash
39+
./provision-witness from-existing \
40+
--witness-ip <IP of the target VM> \
41+
--cluster-ip <IP of one of the nodes of the linstor pool> \
42+
--cluster-ssh-key <Optional: Path to the ssh key of the linstor pool> \
43+
--set-hostname <Optional: hostname. defaults to witness> \
44+
--set-ssh-key-path <Optional: path to the public ssh key to send to the witness> \
45+
--set-password <Optional: password for the root user> \
46+
--skip-setup <Optional: if set, the script will skip the setup of the witness VM> \
47+
--skip-drbd <Optional: if set, the script will skip the DRBD installation on the witness VM> \
48+
--skip-linstor <Optional: if set, the script will skip the LINSTOR satellite installation on the witness VM>
49+
```
50+
51+
### From a template VM
52+
53+
This script will create a witness VM from a template VM.
54+
It requires an existing LINSTOR pool and a template VM that has been provisioned with the LINSTOR client installed.
55+
56+
To create a template VM, please refer to the [Creating a template VM](#creating-a-template-vm) section below.
57+
58+
```bash
59+
./provision-witness from-template \
60+
--template-xva-path <Path to the template XVA> \
61+
--vm-name <Name of the target VM> \
62+
--vm-network <UUID of the network to use for the witness VM> \
63+
--vm-sr <UUID of the SR to create the target VM in> \
64+
--cluster-ip <IP of one of the nodes of the linstor pool> \
65+
--cluster-ssh-key <Optional: Path to the ssh key of the linstor pool> \
66+
--host-ip <Optional: IP of the witness VM host> \
67+
--host-ssh-key-path <Optional: path to the ssh key to access the host> \
68+
--set-hostname <Optional: hostname. defaults to witness> \
69+
--set-ssh-key-path <Optional: path to the public ssh key to send to the witness> \
70+
--set-password <Optional: password for the root user>
71+
```
72+
73+
If `--host-ip` is not provided, the script will assume the current host is the one that will host the witness VM.
74+
75+
76+
## Creating a template VM
77+
78+
To create a template VM, use the `make-template` command on an existing AlmaLinux >=9 VM
79+
and then use the newly provisioned VM as a template for future witness VMs.
80+
81+
The command will follow the same steps as for provisioning the witness VM from an existing VM, but will also
82+
strip identifying information from the VM to make it reusable as a template. Then it will shut the vm down.
83+
It will also skip the step of adding the VM to the LINSTOR pool as it is not necessary for a template VM.
84+
85+
```bash
86+
./provision-witness make-template \
87+
--witness-ip <IP of the target VM> \
88+
--set-hostname <Optional: hostname. defaults to witness> \
89+
--set-ssh-key-path <Optional: path to the public ssh key to send to the witness> \
90+
--set-password <Optional: password for the root user> \
91+
--skip-setup <Optional: if set, the script will skip the setup of the witness VM> \
92+
--skip-drbd <Optional: if set, the script will skip the DRBD installation on the witness VM> \
93+
--skip-linstor <Optional: if set, the script will skip the LINSTOR satellite installation on the witness VM>
94+
```
95+
96+
The resulting VM is now ready to be converted as a template or exported as an XVA to be used as a template for future witness VMs.
97+
98+
## Updating the witness VM
99+
100+
Updating the witness VM can be done in two ways:
101+
- Updating the witness VM in place, which will keep the same VM and update its configuration.
102+
- Using a new VM, which will provision a new witness VM and migrate the witness resource to
103+
the new VM.
104+
105+
The update command can be used to update the witness VM or to change its configuration,
106+
for example to change the ssh key or the password.
107+
108+
### Updating in-place
109+
110+
The script will follow the same steps as for provisioning the witness VM from an existing VM
111+
except for those differences:
112+
113+
- Unless any configuration has changed, the `--set-*` options
114+
can be omitted to keep the current configuration of the witness VM.
115+
- Since it is done in place, the step were the node is added to the LINSTOR pool will be skipped
116+
and the `--cluster-*` options are not used.
117+
118+
To update in place:
119+
120+
```bash
121+
./provision-witness update-in-place \
122+
--witness-ip <IP of the target VM>
123+
# The other options are available, but wouldn't make sense in most cases
124+
```
125+
126+
### Using a new VM
127+
128+
The script will follow the same steps as either the provisioning the witness VM from an existing VM
129+
or from a template VM.
130+
131+
The two following flags are added to the command,
132+
to respectively specify the name of the old witness VM and if it should be deleted.
133+
134+
It is important that the new witness node doesn't have the same name
135+
as the old witness node to avoid conflicts in the LINSTOR pool.
136+
137+
The process is as follows:
138+
- A new witness VM is provisioned/created
139+
- The new witness VM is added to the LINSTOR pool
140+
- The old witness VM is evacuated, which will migrate the witness resource to the new witness VM.
141+
- Optionally, the old witness VM is deleted.
142+
143+
#### On an existing VM
144+
```bash
145+
./provision-witness update-from-new from-existing \
146+
--witness-ip <IP of the target VM> \
147+
--evacuate-node <Name of the old witness VM to evacuate> \
148+
--cluster-ip <IP of one of the nodes of the linstor pool> \
149+
--cluster-ssh-key <Optional: Path to the ssh key of the linstor pool> \
150+
--remove-node <Optional: if set, the old witness VM will be deleted after the evacuation> \
151+
--set-hostname <Optional: hostname. defaults to witness> \
152+
--set-ssh-key-path <Optional: path to the public ssh key to send to the witness> \
153+
--set-password <Optional: password for the root user> \
154+
--skip-setup <Optional: if set, the script will skip the setup of the witness VM> \
155+
--skip-drbd <Optional: if set, the script will skip the DRBD installation on the witness VM> \
156+
--skip-linstor <Optional: if set, the script will skip the LINSTOR satellite installation on the witness VM>
157+
```
158+
159+
#### From a template VM
160+
```bash
161+
./provision-witness update-from-new from-template \
162+
--template-xva-path <Path to the template XVA> \
163+
--vm-name <Name of the target VM> \
164+
--vm-network <UUID of the network to use for the witness VM> \
165+
--vm-sr <UUID of the SR to create the target VM in> \
166+
--cluster-ip <IP of one of the nodes of the linstor pool> \
167+
--evacuate-node <Name of the old witness VM to evacuate> \
168+
--cluster-ssh-key <Optional: Path to the ssh key of the linstor pool> \
169+
--remove-node <Optional: if set, the old witness VM will be deleted after the evacuation> \
170+
--host-ip <Optional: IP of the witness VM host> \
171+
--host-ssh-key-path <Optional: path to the ssh key to access the host> \
172+
--set-hostname <Optional: hostname. defaults to witness> \
173+
--set-ssh-key-path <Optional: path to the public ssh key to send to the witness> \
174+
--set-password <Optional: password for the root user>
175+
```
176+
177+
178+
## Setting up paths and interfaces
179+
180+
As it is recommended to have a dedicated 10Gbps network for the LINSTOR nodes,
181+
and that the witness VM should not run on the same pool as the LINSTOR nodes,
182+
it is likely that the witness VM doesn't have access to the 10Gbps network.
183+
Therefore, it is necessary to set up the paths and interfaces
184+
for the witness VM to be able to communicate with the LINSTOR nodes.
185+
186+
Let's imagine a setup with the following network topology:
187+
188+
![topology.png](images/topology.png)
189+
190+
In a typical setup before adding the witness, there are two likely situations:
191+
- Either a `PrefNic` is set
192+
- Either a node-connection path is set
193+
194+
To check which situation is in place:
195+
196+
- Check for existing interfaces on each node:
197+
```bash
198+
linstor node interface list lin-1
199+
╭──────────────────────────────────────────────────────────────────╮
200+
┊ r620-s1 ┊ NetInterface ┊ IP ┊ Port ┊ EncryptionType ┊
201+
╞══════════════════════════════════════════════════════════════════╡
202+
┊ + StltCon ┊ default ┊ 10.0.0.11 ┊ 3366 ┊ PLAIN ┊
203+
┊ + ┊ 10g ┊ 192.168.1.11 ┊ ┊ ┊
204+
╰──────────────────────────────────────────────────────────────────╯
205+
```
206+
```bash
207+
linstor node interface list lin-1
208+
╭──────────────────────────────────────────────────────────────────╮
209+
┊ r620-s1 ┊ NetInterface ┊ IP ┊ Port ┊ EncryptionType ┊
210+
╞══════════════════════════════════════════════════════════════════╡
211+
┊ + StltCon ┊ default ┊ 10.0.0.12 ┊ 3366 ┊ PLAIN ┊
212+
┊ + ┊ 10g ┊ 192.168.1.12 ┊ ┊ ┊
213+
╰──────────────────────────────────────────────────────────────────╯
214+
```
215+
216+
- Check for a `PrefNIC` set on each node:
217+
```bash
218+
linstor node list-properties lin-1
219+
╭───────────────────────────╮
220+
┊ Key ┊ Value ┊
221+
╞═══════════════════════════╡
222+
┊ CurStltConnName ┊ default ┊
223+
┊ NodeUname ┊ lin-1 ┊
224+
┊ PrefNIC ┊ 10g ┊
225+
╰───────────────────────────╯
226+
```
227+
```bash
228+
linstor node list-properties lin-2
229+
╭───────────────────────────╮
230+
┊ Key ┊ Value ┊
231+
╞═══════════════════════════╡
232+
┊ CurStltConnName ┊ default ┊
233+
┊ NodeUname ┊ lin-1 ┊
234+
┊ PrefNIC ┊ 10g ┊
235+
╰───────────────────────────╯
236+
```
237+
238+
If there is a `PrefNIC` set on any node, the witness VM will use it to communicate with the LINSTOR nodes.
239+
It will then be necessary to set up the corresponding path on the witness VM to use the default interface.
240+
241+
If there is no `PrefNIC` set on any node, the witness VM will use the default interface to communicate with the LINSTOR nodes.
242+
243+
- Check for existing paths on each node:
244+
```bash
245+
linstor node-connection path list lin-1 lin-2
246+
╭────────────────────────────────────╮
247+
┊ Key ┊ Value ┊
248+
╞════════════════════════════════════╡
249+
┊ Paths/drbd/lin-1 ┊ 10g ┊
250+
┊ Paths/drbd/lin-2 ┊ 10g ┊
251+
╰────────────────────────────────────╯
252+
```
253+
254+
If there is a path set between the two nodes, the two nodes will use the specified interface for DRBD traffic.
255+
If there is no path set between the two nodes, the two nodes will use the PrefNic/default interface for DRBD traffic.
256+
257+
### Preparing the network path with a `PrefNIC` set on the nodes
258+
259+
These steps must be done after adding the witness node to the pool.
260+
Assuming the witness VM is called `witness`.
261+
262+
Set up the corresponding path on the witness VM to use the default interface:
263+
```bash
264+
linstor node-connection path create lin-1 witness default default
265+
```
266+
This step is to be repeated for each node with a `PrefNIC` set.
267+
268+
### Alternative: Preparing the network path removing the `PrefNIC` from the nodes
269+
270+
It is also possible to remove the `PrefNIC` from the nodes and set up the paths on the nodes.
271+
This will allow the witness VM to use the default interface to communicate with the LINSTOR nodes.
272+
273+
Create the path between the two hosts:
274+
```bash
275+
linstor node-connection path create lin-1 lin-2 default default
276+
```
277+
278+
Remove the `PrefNIC` from the nodes:
279+
```bash
280+
linstor node set-property lin-1 PrefNIC
281+
# This step is to be repeated for each node with a `PrefNIC` set.
282+
```
283+
284+
This order is important to avoid using the default interface with the nodes when removing the `PrefNIC`.
285+
286+
### If there is no `PrefNIC` set on any node
287+
288+
Nothing needs to be done on the witness VM as it will use the default interface to communicate with the LINSTOR nodes.

‎src/linstor_witness/images/topology.png‎

54.3 KB
Loading

0 commit comments

Comments
 (0)