Skip to content

Unknowlars/docker-swarm-proxmox

BranchesTags

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

3 Commits
ansible
ansible
 
 
scripts
scripts
 
 
terraform
terraform
 
 
.gitignore
.gitignore
 
 
Makefile
Makefile
 
 
README.md
README.md
 
 

Repository files navigation

Swarm on Proxmox (IaC)

Provision and configure a 3-node Docker Swarm homelab on Proxmox with Infrastructure as Code.

  • Provisioning: OpenTofu/Terraform + Telmate/proxmox
  • Configuration: Ansible
  • Topology: 1 manager + 2 workers, static IPs
  • Storage: GlusterFS replica-3 — each node contributes a dedicated local brick disk (provisioned by Terraform as a second SCSI disk). SQLite-safe, no single point of failure.
  • Swarm network: encrypted, attachable overlay
  • Monitoring: Grafana + Loki + Prometheus + Alertmanager, with Grafana Alloy as a systemd service on every node
  • Access model: two users per VM
    • personal bootstrap user (cloud_init_user + your SSH key)
    • dedicated automation user (ansible_account_username, default ansible, separate SSH key)

High-level flow

flowchart TD
  A[make apply] --> B[OpenTofu: clone VMs + attach brick disks]
  B --> C[Generate ansible/inventory/hosts.yml]
  C --> D[make configure]
  D --> E[bootstrap-access.yml - create ansible user]
  E --> F[configure-docker.yml - Docker daemon + DNS + disable IPv6]
  F --> G[install-docker.yml - Docker Engine]
  G --> H[bootstrap-swarm.yml - init swarm + join workers]
  H --> I[configure-glusterfs.yml - partition + XFS + GlusterFS replica-3]
  I --> J[configure-swarm-network.yml - encrypted overlay]
  J --> K[configure-alloy.yml - Grafana Alloy systemd service on all nodes]
  K --> L[make verify - node health + GlusterFS replication check]
  L --> M[make deploy-stack STACK=monitoring]
  M --> N[make deploy-stack STACK=glance]
Loading

Disk layout per VM

/dev/sda      → OS disk (scsi0, from cloud-init template)
/dev/sdb      → GlusterFS brick disk (scsi1, from Terraform glusterfs_disk_size)
  /dev/sdb1   → single XFS partition
  /gluster/brick      → XFS mount point  (glusterfs_brick_parent)
  /gluster/brick/vol  → actual GlusterFS brick (glusterfs_brick_dir)

/mnt/gluster  → GlusterFS FUSE mount — all stacks bind-mount from here

Repository layout

terraform/                    Proxmox VM provisioning (OS + brick disks, inventory generation)
ansible/
  playbooks/
    bootstrap-access.yml      Create automation user on all nodes
    configure-docker.yml      Docker daemon config, IPv6 disable, resolv.conf fix
    install-docker.yml        Docker Engine install
    bootstrap-swarm.yml       Swarm init + join
    configure-glusterfs.yml   Disk partition + XFS + GlusterFS cluster
    configure-swarm-network.yml   Overlay network
    configure-alloy.yml       Grafana Alloy install + config on all nodes
    verify.yml                Node health + GlusterFS replication test
    deploy-stack.yml          Generic stack deployer (all stacks)
    remove-stack.yml          Generic stack removal
    site.yml                  Full configuration pipeline
  stacks/
    README.md                 Stack convention guide
    monitoring/
      stack.yml.j2            Grafana + Loki + Prometheus + Alertmanager
      vars.yml                Non-secret stack defaults
      config/                 Static config files (prometheus.yaml, nginx.conf,
                              alertmanager.yml, grafana datasources, resolv.conf)
      templates/              Ansible-rendered templates (loki.yaml.j2)
    glance/
      stack.yml.j2            Glance dashboard Compose template
      vars.yml                Non-secret stack defaults
      config/                 Config files pre-populated on GlusterFS
  templates/
    alloy/
      config.alloy.j2         Grafana Alloy config template (per-node, Jinja2)
  inventory/
    group_vars/all.yml        Shared defaults (GlusterFS paths, overlay settings)
    hosts.yml                 Generated by Terraform (do not hand-edit)
scripts/
  preflight.sh                Pre-apply checks
  install-deps.sh             Local dependency installer
Makefile                      Workflow entry points

Prerequisites

  • OpenTofu or Terraform
  • Ansible + grafana.grafana collection (make install-deps)
  • Proxmox API token with VM clone/create/disk rights
  • Existing cloud-init template in Proxmox
  • Garage S3 (on TrueNAS) with two buckets: loki-data, loki-ruler

Required ports between swarm nodes: 2377/TCP, 7946/TCP+UDP, 4789/UDP, 24007/TCP (GlusterFS)

1) Create Proxmox API token

  1. Open https://<proxmox-ip>:8006
  2. Datacenter → Permissions → Users → Add — user: terraform, realm: pve
  3. Datacenter → Permissions → Add — path: /, user: terraform@pve, role: Administrator
  4. Datacenter → Permissions → API Tokens → Add — user: terraform@pve, token id: swarm, privilege separation: Off
  5. Save into terraform/terraform.tfvars

2) Configure terraform/terraform.tfvars

cp terraform/terraform.tfvars.example terraform/terraform.tfvars

Key additions vs the previous NFS version:

glusterfs_disk_size    = "50G"      # second disk per VM for GlusterFS bricks
# glusterfs_storage_pool = "fast-lvm" # optional: use a different pool for brick disks

Per-node override (if one node has a larger disk):

swarm_nodes = {
  swarm-mgr-1 = {
    vmid                = 9101
    ip                  = "192.168.0.241"
    role                = "manager"
    glusterfs_disk_size = "100G"   # manager gets extra brick space
  }
  ...
}

3) Deploy and configure

make install-deps    # installs Ansible + grafana.grafana collection
make preflight
make init-upgrade
make plan
make apply           # provisions VMs with OS + brick disks
make configure       # bootstrap → docker → swarm → GlusterFS → overlay → Alloy
make verify          # node health + GlusterFS replication check

4) Deploy stacks

# Monitoring stack (Grafana + Loki + Prometheus + Alertmanager)
LOKI_S3_ACCESS_KEY_ID=xx \
LOKI_S3_SECRET_ACCESS_KEY=yy \
GRAFANA_ADMIN_PASSWORD=zz \
make deploy-stack STACK=monitoring

# Glance dashboard
GLANCE_BESZEL_TOKEN=xxx make deploy-stack STACK=glance

# Any future stack
make deploy-stack STACK=myapp

# Remove a stack
make remove-stack STACK=glance

See ansible/stacks/README.md for how to add new stacks.

Monitoring stack

The monitoring stack runs as Docker Swarm services. Metrics and logs are collected by Grafana Alloy installed as a systemd service on each node — not as a container.

Architecture

Each node (systemd):
  Grafana Alloy
    ├── prometheus.exporter.unix     → host metrics (replaces node-exporter)
    ├── prometheus.exporter.cadvisor → container metrics (built-in cAdvisor)
    ├── loki.source.docker           → container stdout/stderr logs
    ├── loki.source.journal          → systemd journal logs
    └── loki.source.file             → /var/log/* files

Manager node Alloy only:
    ├── discovery.dockerswarm (global tasks)     → scrapes global swarm services
    └── discovery.dockerswarm (replicated tasks) → scrapes replicated swarm services

Docker Swarm services:
  Grafana        :3000   → dashboards
  Prometheus     :9090   → metrics storage (receives remote_write from Alloy)
  Alertmanager   :9093   → alert routing
  loki-gateway   :3100   → nginx reverse proxy → loki-read / loki-write
  loki-read      ×3      → Loki read path
  loki-write     ×3      → Loki write path (ingesters)
  loki-backend   ×3      → Loki backend (compactor, ruler, index gateway)

Storage:
  Chunks + indexes → Garage S3 on TrueNAS (loki-data, loki-ruler buckets)
  Grafana state    → GlusterFS
  Prometheus TSDB  → GlusterFS (7d retention)
  Alertmanager     → GlusterFS

Swarm service discovery

Any service you deploy to the swarm is automatically scraped by Prometheus (via Alloy on the manager) if it has these labels:

labels:
  prometheus_port: "8080"    # required — port that exposes /metrics
  prometheus_path: "/metrics" # optional — defaults to /metrics

No changes to prometheus.yaml needed. The Alloy config on the manager uses discovery.dockerswarm to find all running tasks with this label.

Alloy UI

Each node exposes the Alloy UI for live debugging:

http://192.168.0.241:12345   # manager
http://192.168.0.242:12345   # wrk-1
http://192.168.0.243:12345   # wrk-2

Reconfigure Alloy after config changes

make configure-alloy

GlusterFS paths

  • Brick disk mount: glusterfs_brick_parent = /gluster/brick
  • Brick subdir: glusterfs_brick_dir = /gluster/brick/vol
  • Volume name: glusterfs_volume_name = swarm-vol
  • FUSE mount: glusterfs_mount_point = /mnt/gluster
  • Stack data root: /mnt/gluster/docker-data/<stack-name>/

All configurable in ansible/inventory/group_vars/all.yml.

Edit terraform/terraform.tfvars, then:

make plan && make apply && make configure && make verify

Makefile targets

Target Description
make install-deps Install local tools + Ansible collections
make preflight Pre-apply environment checks
make init-upgrade Init/upgrade OpenTofu providers
make plan Terraform plan
make apply Provision VMs
make configure Full Ansible configuration pipeline
make configure-docker Docker daemon + IPv6 disable + resolv.conf fix
make configure-glusterfs GlusterFS setup only
make configure-network Overlay network only
make configure-alloy Deploy/update Grafana Alloy on all nodes
make verify Health checks
make deploy-stack STACK=x Deploy a stack
make remove-stack STACK=x Remove a stack
make clean-known-hosts Clear SSH known_hosts for all node IPs
make destroy Destroy all VMs

Troubleshooting

DNS failures inside containers ([::1]:53: connection refused)

Non-root container users (Loki UID 10001, Grafana UID 472, Prometheus UID 65534) can fail to read Docker's auto-generated /etc/resolv.conf. Go's DNS resolver falls back to [::1]:53 when it can't read the file — which doesn't exist.

Permanent fix (already applied by configure-docker.yml):

  • IPv6 disabled system-wide via sysctl
  • /etc/resolv.conf symlinked to the systemd-resolved full resolver (real upstream IPs) instead of the stub resolver (127.0.0.53)
  • Each monitoring service mounts an explicit resolv.conf via GlusterFS as a belt-and-suspenders fallback

Loki memberlist can't find peers

Loki uses loki-read:7946, loki-write:7946, loki-backend:7946 for gossip. If DNS resolution fails, members can't form a cluster. Root cause is always the resolv.conf issue above. Verify with:

docker run --rm --network monitoring_monitoring alpine sh -c \
  "nslookup loki-read && nslookup loki-write && nslookup loki-backend"

nginx gateway fails to start (host not found in upstream)

nginx resolves upstream hostnames once at startup and caches them. If Loki isn't ready yet, nginx fails. Fixed by using resolver 127.0.0.11 valid=30s ipv6=off and variable-based upstreams (set $loki_read "loki-read:3100") so nginx re-resolves on each request.

GlusterFS brick disk not detected

Verify the VM has two disks: lsblk on a node should show sda (OS) and sdb (brick). If Terraform didn't add the brick disk, the VM may need to be reprovisioned: make destroy && make apply

Volume not mounting after reboot

Check systemctl status glusterd — glusterd must be running before the FUSE mount. _netdev in fstab ensures mount waits for network; verify it is present: grep gluster /etc/fstab

Split-brain scenario

gluster volume heal swarm-vol info
gluster volume heal swarm-vol
gluster volume heal swarm-vol full   # force

REMOTE HOST IDENTIFICATION HAS CHANGED

Expected after VM recreation: make clean-known-hosts

Permission denied during bootstrap

Verify key paths in terraform.tfvars; run make preflight

Grafana can't connect to Prometheus or Loki

Check that the datasource URLs in config/grafana/datasources.yaml use the service names (http://prometheus:9090, http://loki-gateway). These resolve via Docker's overlay DNS — requires the services to be on the same overlay network and the resolv.conf fix to be applied.

Security notes

  • Do not commit terraform/terraform.tfvars (already in .gitignore)
  • Rotate your Proxmox API token if it was ever committed or shared
  • Set proxmox_tls_insecure = false once your Proxmox cert is trusted
  • All monitoring services run as non-root users (by design — this is correct)
  • Loki S3 credentials are passed as environment variables at deploy time, never stored in the repo

Official docs

About

Provision and configure a 3-node Docker Swarm homelab on Proxmox with Infrastructure as Code.

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors

Languages