Skip to content

HArmadillium.md

Latest commit

 

History

History
596 lines (492 loc) · 19.2 KB

HArmadillium.md

File metadata and controls

596 lines (492 loc) · 19.2 KB

Hyperledger Debian Ubuntu

Support UniversalBit Project -- Disambiguation --

📊 High Availability Cluster Setup Flowchart

flowchart TD
    A[High Availability Cluster Setup Guide]:::main

    subgraph Prerequisites
        direction TB
        A1[Required Packages: python3, corosync, pacemaker, etc.]
        A2[Static IP for each node]
        A3[Edit /etc/hosts on each node]
        A4[UFW Firewall Rules]
        A5[Enable SSH/OpenSSH]
    end

    subgraph Corosync_Setup["Corosync"]
        direction TB
        B1[Edit /etc/corosync/corosync.conf]
        B2[Generate corosync-keygen on armadillium01]
        B3[Copy authkey to other nodes via scp]
        B4[Move authkey to /etc/corosync on all nodes]
        B5[Set correct permissions]
        B6[Start corosync service]
    end

    subgraph Pacemaker_Setup["Pacemaker & PCMK"]
        direction TB
        C1[Install pacemaker]
        C2[Create /etc/corosync/service.d/pcmk file]
        C3[Add pacemaker service config to pcmk]
        C4[Run update-rc.d pacemaker defaults]
    end

    subgraph PCS_Setup["PCS"]
        direction TB
        D1[Start pcsd service]
        D2[Set hacluster password]
        D3[Localhost pcs authentication]
        D4[Authorize all cluster nodes]
        D5[Disable stonith]
        D6[Set no-quorum-policy ignore]
        D7[Install resource-agents-extra]
        D8[Create nginx webserver resource]
        D9[Create virtual_ip resource]
        D10[Add colocation and order constraints]
        D11[Start and enable cluster on all nodes]
    end

    subgraph Web_Server["Web Server Setup"]
        direction TB
        E1[Nginx: install and configure reverse proxy]
        E2[Create self-signed certificate with OpenSSL]
        E3[Edit nginx site config]
        E4[Start nginx service]
        E5[Alternative: Apache install, ssl, config, start]
    end

    subgraph VIP["Virtual IP (VIP)"]
        direction TB
        F1[Setup single VIP ]
        F2[Configure VIP resource in PCS]
    end

    subgraph Troubleshooting
        direction TB
        G1[Error: Unable to authenticate/known-hosts]
        G2[Fix: start pcsd service]
        G3[Check pcs cluster status]
        G4[View cluster property list]
    end

    %% Relationships
    A --> Prerequisites
    Prerequisites --> Corosync_Setup
    Corosync_Setup --> Pacemaker_Setup
    Pacemaker_Setup --> PCS_Setup
    PCS_Setup --> Web_Server
    PCS_Setup --> VIP
    Web_Server --> VIP
    PCS_Setup --> Troubleshooting
    Troubleshooting --> G3
    Troubleshooting --> G4

    classDef main fill:#e2e2e2,stroke:#333,stroke-width:2px;
Loading

Required Packages: Lists necessary software like python3, corosync, pacemaker, fence-agents, crmsh, pcs, nginx, and more.

This document complements the ha_cluster_setup.sh script by detailing the manual configurations and additional setups needed to complete the HA cluster configuration process.

Learn More About High Availability

Python3

note: --Deadsnakes PPA has already updated its support for Ubuntu 24.04 (Noble)

sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt update
sudo apt install python3.11
sudo python3 -m ensurepip --upgrade
Alternative Python Setup:

For Ubuntu 24.04 LTS (Noble), install the following packages:

sudo apt install corosync pacemaker fence-agents crmsh pcs* cluster-glue ufw nginx haveged heartbeat openssh-server openssh-client

Static IP

Setup Static IP Address

Ensure that each node is configured with a static IP address by following the setup guide linked above.

Host

Edit the Host File for Each Node
To configure the host file on each node, use the following command:

sudo nano /etc/hosts

Reference:

Note: Ensure that the host file is properly edited and configured on every node.

UFW Firewall Rules for Each Node

The Uncomplicated Firewall (UFW) is a user-friendly front-end for managing iptables, simplifying the process of configuring a Netfilter firewall. It provides a command-line interface with syntax inspi[...]

Commands for Configuration:

sudo ufw allow from 192.168.1.141
sudo ufw allow from 192.168.1.142
sudo ufw allow from 192.168.1.143
sudo ufw allow from 192.168.1.144
sudo ufw allow ssh

Note:
Ensure that these firewall rules are applied to each node to maintain proper network access and security.

SSH Connection to Communicate with All Nodes

OpenSSH
Ensure that each node has SSH enabled to allow secure communication between nodes. OpenSSH is a widely-used tool for managing secure shell (SSH) connections, providing encryption for data transfer and[...]

References:

Note:
To maintain proper connectivity, verify that SSH is enabled and properly configured on all nodes.

Corosync

  • Corosync cluster engine daemon and utilities
The Corosync Cluster Engine is a Group Communication System with additional features for implementing high availability within applications.
The project provides four C Application Programming Interface features:
  • A closed process group communication model with virtual synchrony guarantees for creating replicated state machines.
  • A simple availability manager that restarts the application process when it has failed.
  • A configuration and statistics in-memory database that provide the ability to set, retrieve, and receive change notifications of information.
  • A quorum system that notifies applications when quorum is achieved or lost.

Corosync Configuration File: repeat this TO each node

sudo rm /etc/corosync/corosync.conf
sudo nano /etc/corosync/corosync.conf

corosync configuration file:

totem {
  version: 2
  cluster_name: HArmadillium
  transport: udpu
  interface {
   ringnumber: 0
   bindnetaddr: 192.168.1.140
   broadcast: yes
   mcastport: 5405
 }
}
nodelist {
  node {
    ring0_addr: 192.168.1.141
    name: armadillium01
    nodeid: 1
  }
  node {
    ring0_addr: 192.168.1.142
    name: armadillium02
    nodeid: 2
  }
  node {
    ring0_addr: 192.168.1.143
    name: armadillium03
    nodeid: 3
  }
  node {
    ring0_addr: 192.168.1.144
    name: armadillium04
    nodeid: 4
  }
}
logging {
  to_logfile: yes
  logfile: /var/log/corosync/corosync.log
  to_syslog: yes
  timestamp: on
}
service {
  name: pacemaker
  ver: 1
}
sudo service corosync start

Corosync-keygen Authorize

  • FROM armadillium01 create corosync key :
#armadillium01 
sudo corosync-keygen
  • secure copy(ssh) corosync authkey FROM armadillium01 TO #armadillium02 #armadillium03 #armadillium04 IN /tmp directory
sudo scp /etc/corosync/authkey armadillium02@192.168.1.142:/tmp #02
sudo scp /etc/corosync/authkey armadillium03@192.168.1.143:/tmp #03
sudo scp /etc/corosync/authkey armadillium04@192.168.1.144:/tmp #04
  • connect via(ssh) and move copied file FROM /tmp directory TO /etc/corosync directory
#connect(ssh) to armadillium02 
ssh armadillium02@192.168.1.142 #02
sudo mv /tmp/authkey /etc/corosync
sudo chown root: /etc/corosync/authkey
sudo chmod 400 /etc/corosync/authkey

corosync setup

PCS: Pacemaker Configuration System

PCS simplifies the management of Pacemaker-based clusters, allowing users to easily view, modify, and create clusters. It also includes pcsd, which acts as both a graphical user interface (GUI) and [...]

Start the PCS Service

sudo service pcsd start

Set Password and Authenticate Localhost

Create Password for hacluster User

# On armadillium01
sudo passwd hacluster

Authenticate Localhost

sudo pcs client local-auth
# Username: hacluster
# Password:
# localhost: Authorized

Authorize/Authenticate Hosts

Authenticate Cluster Nodes

# On armadillium01
sudo pcs host auth armadillium01 armadillium02 armadillium03 armadillium04
# Username: hacluster
# Password:
# armadillium01: Authorized
# armadillium02: Authorized
# armadillium03: Authorized
# armadillium04: Authorized

Reference:
ClusterLabs: Enable pcs Daemon (3.3.2. Enable pcs Daemon)

PCS Cluster Configuration

Disable STONITH

sudo pcs property set stonith-enabled=false

Ignore Quorum Policy

sudo pcs property set no-quorum-policy=ignore

Create Resources

Install Required Resource Agents

sudo apt install resource-agents-extra

Create Web Server Resource

sudo pcs resource create webserver ocf:heartbeat:nginx configfile=/etc/nginx/nginx.conf op monitor timeout="5s" interval="5s"

Reference:
PCS Create Resources
ClusterLabs Resource Agents

Create Floating IP Resource

Add Floating IP

sudo pcs resource create virtual_ip ocf:heartbeat:IPaddr2 ip=192.168.1.140 cidr_netmask=32 op monitor interval=30s

Add Constraints

Colocation Constraint
sudo pcs constraint colocation add webserver with virtual_ip INFINITY
Order Constraint
sudo pcs constraint order webserver then virtual_ip
# Adding webserver virtual_ip (kind: Mandatory) (Options: first-action=start then-action=start)

Start and Enable the Cluster

sudo pcs cluster start --all
sudo pcs cluster enable --all
# armadillium01: Starting Cluster...
# armadillium02: Starting Cluster...
# armadillium03: Starting Cluster...
# armadillium04: Starting Cluster...
# armadillium01: Cluster Enabled
# armadillium02: Cluster Enabled
# armadillium03: Cluster Enabled
# armadillium04: Cluster Enabled

Note:

CRM

Consider this configuration tool as an alternative to PCS.

Setup

Pacemaker

Cluster Resource Manager:

-Description: Pacemaker is a distributed finite state machine capable of co-ordinating the startup and recovery of inter-related services across a set of machines. Pacemaker understands many different resource types (OCF, SYSV, systemd) and can accurately model the relationships between them (colocation, ordering).

Run Pacemaker after corosync service: TO each node
sudo update-rc.d pacemaker defaults 20 01

PCMK

Create the PCMK Configuration File

  1. Create the necessary directory and file:

    sudo mkdir /etc/corosync/service.d
sudo nano /etc/corosync/service.d/pcmk

  2. Add the following content to the file:

    service {
  name: pacemaker
  ver: 1
}

Webserver

Nginx as a Reverse Proxy

Install the necessary packages for setting up Nginx as a reverse proxy:

sudo apt install openssl nginx git -y

Reference:
OpenSSL WebServer

Self-Signed Certificate (HTTPS) with OpenSSL

Generate a self-signed certificate using OpenSSL:

git clone https://github.com/universalbit-dev/HArmadillium/
cd HArmadillium/ssl
sudo mkdir /etc/nginx/ssl
sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/nginx/ssl/host.key -out /etc/nginx/ssl/host.cert --config distinguished.cnf
sudo openssl dhparam -out /etc/nginx/ssl/dhparam.pem 2048

Nginx Configuration

Edit the default Nginx configuration file:

sudo rm /etc/nginx/sites-enabled/default
sudo nano /etc/nginx/sites-enabled/default

Webserver Nginx Node Configuration

Refer to the node-specific Nginx configuration files:

Start the Nginx service:

sudo service nginx start

Alternative Webserver: Apache High Availability

For an alternative to Nginx, you can use Apache to set up high availability. Start by installing Apache and the required packages:

sudo apt update
sudo apt install apache2 libapache2-mod-ssl ssl-cert -y

Self-Signed Certificate (HTTPS) with OpenSSL for Apache

Generate a self-signed certificate for Apache:

git clone https://github.com/universalbit-dev/HArmadillium/
cd HArmadillium/ssl
sudo mkdir /etc/apache2/ssl
sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/apache2/ssl/host.key -out /etc/apache2/ssl/host.cert --config distinguished.cnf
sudo openssl dhparam -out /etc/apache2/ssl/dhparam.pem 2048

Once installed, you can proceed to configure Apache for high availability.

Refer to the following resources for configuration files:

For more details, visit the Apache High Availability Documentation.

Start the Apache2 service:

sudo service apache2 start

Reference:

Virtual IP (VIP) Configuration for High Availability

To ensure optimal High Availability (HA) performance, it is highly recommended to use a single Virtual IP (VIP) for your web server configuration. A VIP simplifies failover management by directing[...]

For advanced configurations, including load balancing across multiple nodes, ensure proper health checks and synchronization of all nodes. Full details are available in the [VIP Configuration Guide](h[...]

Troubleshooter

Common Error:

**Error**
Warning: Unable to read the known-hosts file: No such file or directory: '/var/lib/pcsd/known-hosts'
armadillium03: Unable to authenticate to armadillium03 - (HTTP error: 401)...
armadillium01: Unable to authenticate to armadillium01 - (HTTP error: 401)...
armadillium04: Unable to authenticate to armadillium04 - (HTTP error: 401)...
armadillium02: Unable to authenticate to armadillium02 - (HTTP error: 401)...

Cause:

The PCSD service is not started.

Fix:

Start the PCSD service on the affected node(s):

# On armadillium02
ssh armadillium02@192.168.1.142
sudo service pcsd start
sudo service pcsd status

Check PCSD Cluster Status:

sudo pcs cluster status

Example Output:

  * armadillium03: Online
  * armadillium04: Online
  * armadillium02: Online
  * armadillium01: Online

View Cluster Property List for Each Node:

sudo pcs property list

Example Output:

Cluster Properties:
cluster-infrastructure: corosync
cluster-name: HArmadillium
dc-version: 2.0.5
have-watchdog: false
no-quorum-policy: ignore
stonith-enabled: false

Your HACluster is now configured and ready to host something amazing!

Resources: