POC_ESNET_MikroTik_Proxmox_HA

Description

A proof-of-concept for deploying a fully automated, High-Availability Proxmox + Ceph cluster over a MikroTik spine-leaf fabricrunning entirely inside GNS3 as a nested virtualization lab. Infrastructure is defined as code using Terraform and Ansible, making the entire environment reproducible from a single make all.

Tech stack

Architecture

Layer	Devices	Role
Spine	2× MikroTik CHR	L3 core, OSPFv2 area 0, ECMP
Leaf	2× MikroTik CHR	ToR switches, MLAG peer pair, VLAN trunks
Compute	3× Proxmox VE	Corosync HA cluster + Ceph storage
Monitoring	1× Zabbix LXC	SNMPv3 (MikroTik) + Agent 2 (PVE)

All nodes run as QEMU VMs inside GNS3. The routing protocol is OSPFv2 with point-to-point links between every leaf and both spines — no BGP, no ASNs. Each PVE node dual-homes to both leaves via an 802.3ad LACP bond; the leaves run MLAG so the bond appears as a single logical port.

Network segmentation

VLAN	Name	Subnet	Purpose
10	mgmt	10.0.10.0/24	PVE web UI, SSH, host management
20	pve-cluster	10.0.20.0/24	Corosync heartbeat, pve-cluster daemon
30	ceph-public	10.0.30.0/24	Ceph public network (client → OSD)
40	ceph-cluster	10.0.40.0/24	Ceph cluster network (OSD ↔ OSD replication)
50	production	10.0.50.0/24	VM and container workload traffic
99	monitoring	10.0.99.0/24	Zabbix polling, SNMP, metrics

Hardware requirements

Resource	Minimum	Recommended
CPU	8 cores with VT-x/AMD-V + nested virt	16+ cores
RAM	32 GB	64+ GB
Disk	150 GB SSD	300+ GB NVMe

Each Proxmox node runs as a QEMU VM (4 vCPU / 8 GB RAM / 32 GB disk). Ceph requires at least one additional OSD disk per node.

GNS3 server setup

This project uses the GNS3 REST API directly (Terraform provider + curl). The netopschic/gns3 Terraform provider does not support HTTP Basic Auth, so the GNS3 server must have authentication disabled.

Edit ~/.config/GNS3/2.2/gns3_server.conf:

[Server]
auth = False

Note: GNS3 2.2.x is required. Version 3.x changed the API from /v2/ to /v3/ and is not compatible with the netopschic/gns3 Terraform provider used here.

Software requirements

• GNS3 Server >= 2.2 with QEMU support & Nested Virtualization
• Terraform >= 1.5 (run via WSL on Windows. The netopschic/gns3 provider is Linux-only)
• Ansible >= 2.15
• Proxmox VE ISO >= 9.2.1 (for GNS3 QEMU template)
• MikroTik CHR image >= 7.21.4 (.img format, for GNS3 QEMU template)
• Packer >= 1.15

Documentation & resources

• GNS3
  • Docs: https://docs.gns3.com/
  • Downloads: https://www.gns3.com/software/download
• Proxmox VE
  • Docs: https://pve.proxmox.com/pve-docs/
  • ISO download: https://www.proxmox.com/en/downloads/proxmox-virtual-environment
• MikroTik RouterOS / CHR
  • Docs: https://help.mikrotik.com/docs/
  • CHR image download: https://mikrotik.com/download
• Terraform
  • Docs: https://developer.hashicorp.com/terraform/docs
  • Downloads: https://developer.hashicorp.com/terraform/install
• Packer
  • Docs: https://developer.hashicorp.com/packer/docs
  • Downloads: https://developer.hashicorp.com/packer/install
• Ansible
  • Docs: https://docs.ansible.com/
  • Downloads: https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html
• Zabbix
  • Docs: https://www.zabbix.com/documentation/current/manual
  • Downloads: https://www.zabbix.com/download
• D2 diagrams
  • Docs: https://d2lang.com/docs
  • Downloads: https://d2lang.com/download
• QEMU / KVM
  • Docs: https://www.qemu.org/documentation/
  • Downloads: https://www.qemu.org/download/
• Ceph
  • Docs: https://docs.ceph.com/en/latest/start/
  • Downloads: https://docs.ceph.com/en/latest/install/get-packages

Nested virtualization check

# Intel
cat /sys/module/kvm_intel/parameters/nested
# AMD
cat /sys/module/kvm_amd/parameters/nested

# Enable if output is 'N'
echo "options kvm_intel nested=1" | sudo tee /etc/modprobe.d/kvm-nested.conf
sudo modprobe -r kvm_intel && sudo modprobe kvm_intel

Requirements

• Host OS: Linux with KVM (recommended). On Windows, run Terraform via WSL2 — the GNS3 provider has no Windows binary.
• CPU: VT-x / AMD-V support and nested virtualization enabled.
• Tools: make, terraform (>= 1.5), ansible (>= 2.15), qemu/kvm, gns3 (>= 2.2), d2 (for diagrams).
• Images: Proxmox VE ISO (>= 8.x) and MikroTik CHR image (>= 7.x) available locally.

Quick start

# 1. Set GNS3 server (no hardcoded URLs)
export GNS3_HOST=http://<gns3-server-ip>   # GNS3 VM v2.2.8+ uses port 80

# 2. Copy and populate variable files
cp terraform/gns3/terraform.tfvars.example terraform/gns3/terraform.tfvars
# Fill in mikrotik_chr_template_id and proxmox_ve_template_id

# 3. Deploy topology, then configure in order
make topology          # GNS3 project + nodes + links
make mikrotik-spine    # OSPF on spines
make mikrotik-leaf     # OSPF + VLANs on leaves
make proxmox-bootstrap # PVE install and cluster join
make proxmox-config    # VMs, storage, HA groups
make zabbix            # Monitoring stack