Skip to content

MASTER_PLAN.md

Latest commit

 

History

History
827 lines (670 loc) · 22.2 KB

MASTER_PLAN.md

File metadata and controls

827 lines (670 loc) · 22.2 KB
Raw

🚀 HAM Homelab Installer v3.0 - MASTER PLAN

Single Source of Truth - All Planning & Decisions

Last Updated: November 20, 2025
Status: ✅ Phases 1 & 2 Complete - Phase 3 Ready to Begin
Progress: 50% complete (3/6 phases done)

📋 Table of Contents

  1. Executive Summary
  2. Current Status
  3. Final Decisions
  4. Service Architecture
  5. Guided Setup Flow
  6. Technical Architecture
  7. Quadlet Best Practices
  8. Implementation Phases
  9. Configuration Schema
  10. Testing Strategy
  11. Success Metrics

📊 Executive Summary

Project Goal

Transform HAM from manual docker-compose setup into a guided TUI installer that:

  • Walks users through account creation (free options!)
  • Builds configuration interactively
  • Generates production-grade Podman Quadlets
  • Deploys with systemd integration
  • Supports multiple providers (VPN, reverse proxy, password managers, cloud backup)

Key Innovation

Config-First Architecture: TUI → config.toml → Generate Quadlets → Deploy

Cost

$0/month possible using all free options:

  • Cloudflare (tunnel & DNS) - FREE
  • Tailscale (VPN, 100 devices) - FREE
  • Vaultwarden (password manager) - FREE self-hosted
  • Backblaze B2 (cloud backup, 10GB) - FREE
  • All services - FREE open source

Timeline

  • ✅ Phase 0: Planning (4 hours) - COMPLETE
  • ✅ Phase 1: Quadlet conversion (3 hours) - COMPLETE
  • ✅ Phase 2: Project structures (2 hours) - COMPLETE
  • 🔜 Phase 3: Implementation (7-10 days) - NEXT
  • ⏳ Phase 4: Testing (3 days)
  • ⏳ Phase 5: Evaluation & polish (3-5 days)
  • Completed: ~9 hours, 50% done
  • Remaining: ~2 weeks to v1.0

🎯 Current Status

What's Complete ✅

Phase 0: Planning (4 hours)

  • ✅ All documents consolidated into MASTER_PLAN.md
  • ✅ Service architecture defined
  • ✅ TUI flow designed
  • ✅ Technical approaches specified
  • ✅ Testing strategy documented

Phase 1: Quadlet Conversion (3 hours)

  • ✅ 22 quadlet files created
  • ✅ 12 core services (Caddy, Traefik, Pi-hole, HA stack, 3 VPNs)
  • ✅ 5 optional services (Node-RED, Samba, Web, Backup, Cloudflared)
  • ✅ 3 networks (frontend, backend, VPN)
  • ✅ 4 config templates (Caddyfile, Traefik configs)
  • ✅ All production-grade (health checks, auto-updates, resource limits)
  • ✅ See: PHASE1_COMPLETE.md

Phase 2: Project Structures (2 hours)

  • ✅ Go implementation working (5.9 MB binary)
    • Compiles and runs
    • CLI with all flags
    • 700 lines of code
    • Module structure complete
  • ✅ Rust implementation ready (needs Rust 1.56+)
    • 460 lines of code
    • Structure complete
    • Version requirement documented
  • ✅ 32 source files total
  • ✅ 1,160 lines of code total
  • ✅ See: PHASE2_SUCCESS.md

What's Next 🔜

Phase 3: Implementation (7-10 days)

Focus on Go implementation first (Rust later after system upgrade):

Priority 1: Template-First Approach (2-3 days)

  • Convert existing quadlets to Go templates
  • Implement template rendering engine
  • Test with various configurations

Priority 2: System Configuration (2-3 days)

  • Automate Podman setup
  • Firewall configuration
  • USB device access
  • VPN module loading

Priority 3: TUI Components (3-4 days)

  • Account wizard (Cloudflare, VPN, password manager)
  • Service selection
  • Configuration builder
  • Review screen

Priority 4: Other Approaches (as needed)

  • Dynamic generation (compose → podlet → config)
  • Hybrid approach (template compose → convert)

✅ Final Decisions

Core Services (Always Installed)

1. Reverse Proxy (User Choice)

  • Caddy (recommended) - Simple, auto-HTTPS, beginner-friendly
  • Traefik (alternative) - Advanced routing, label-based configuration
  • User selects during TUI

2. Pi-hole (Required)

  • Network-wide ad blocking
  • DNS server with web interface
  • Query logging and statistics
  • Local DNS for homelab services

3. Home Assistant (Required)

  • Home Assistant core
  • MQTT broker (Eclipse Mosquitto)
  • ESPHome (ESP device management)
  • Whisper (voice recognition)
  • Piper (text-to-speech)
  • OpenWakeWord (wake word detection)

4. VPN (Required - Multi-Provider Support)

  • Tailscale (recommended default) - FREE, zero-config mesh VPN, 100 devices
  • WireGuard (advanced) - FREE, self-hosted, maximum control
  • User selects with guided setup

Optional Services (User Selects)

5. Node-RED

  • Visual automation and flow programming
  • Complements Home Assistant
  • Lightweight, low resource usage

6. Samba

  • Network file sharing (SMB/CIFS)
  • Common homelab requirement
  • Configurable shares and permissions

7. Static Web Server

  • Host personal websites
  • Cloudflare Tunnel integration option
  • Simple static file serving

8. Cloud Backup (Rclone)

  • Generic cloud backup to any provider
  • Backblaze B2 (10GB FREE)
  • Scheduled with systemd timer
  • Configurable retention

Infrastructure

Networks (3 total - always created)

  1. homelab-frontend (10.88.0.0/24)
    • Reverse proxy
    • Services with web UI
  2. homelab-backend (10.88.1.0/24)
    • Databases
    • Internal services
  3. homelab-vpn (10.88.2.0/24)
    • VPN containers
    • Exit point for services needing VPN

Storage

  • Auto-detect drives and pools
  • User confirms or customizes
  • Separate pools for hot/cold data
  • Automatic mount point creation

🏗️ Service Architecture

Service Tiers

Tier 1: Core Infrastructure (Always Running)

  • Reverse Proxy (Caddy OR Traefik)
  • Pi-hole (DNS/Ad Blocking)
  • VPN (Tailscale OR WireGuard)

Tier 2: Primary Services (Always Running)

  • Home Assistant + Stack
    • Home Assistant Core
    • MQTT Broker
    • ESPHome
    • Voice Services (Whisper, Piper, OpenWakeWord)

Tier 3: Enhancement Services (User Selected)

  • Node-RED
  • Samba
  • Static Web Server
  • Cloud Backup

Dependencies

Reverse Proxy (Caddy/Traefik)
├── Home Assistant
│   ├── MQTT
│   ├── ESPHome
│   └── Voice Services
│       ├── Whisper
│       ├── Piper
│       └── OpenWakeWord
├── Pi-hole
├── Node-RED (optional)
└── Static Web (optional)

VPN (Independent)
├── Tailscale (recommended)
└── WireGuard (advanced)

Storage (Independent)
├── Samba (optional)
└── Rclone Backup (optional)

Service Groupings

DNS & Access

  • Pi-hole (DNS/Ad Block)
  • Reverse Proxy (Caddy/Traefik)
  • VPN (Tailscale/WireGuard/NordVPN)
  • Cloudflare Tunnel (optional per-service)

Home Automation

  • Home Assistant
  • MQTT
  • ESPHome
  • Node-RED (optional)

Voice Assistant

  • Whisper (recognition)
  • Piper (TTS)
  • OpenWakeWord (wake word)

Sharing & Storage

  • Samba (optional)
  • Rclone Backup (optional)
  • Static Web Server (optional)

🎨 Guided Setup Flow

TUI Philosophy

"Hold the user's hand through everything"

Flow Overview

┌─────────────────────────────────────────┐
│  1. Welcome                             │
│     - Show cost estimate                │
│     - Explain free options              │
│     - Prerequisites check               │
└──────────────┬──────────────────────────┘
               │
┌──────────────▼──────────────────────────┐
│  2. Account Wizard                      │
│     - Cloudflare (optional)             │
│     - VPN provider selection            │
│     - Password manager (optional)       │
│     - Cloud backup (optional)           │
│     - Domain setup (optional)           │
└──────────────┬──────────────────────────┘
               │
┌──────────────▼──────────────────────────┐
│  3. Service Configuration               │
│     - Reverse proxy choice              │
│     - Optional services                 │
│     - Domains & subdomains              │
│     - Storage detection                 │
└──────────────┬──────────────────────────┘
               │
┌──────────────▼──────────────────────────┐
│  4. Review & Edit                       │
│     - Show config.toml                  │
│     - Allow manual edits                │
│     - Validate settings                 │
└──────────────┬──────────────────────────┘
               │
┌──────────────▼──────────────────────────┐
│  5. System Configuration                │
│     - Enable Podman socket              │
│     - Configure firewall                │
│     - Set up USB (if HA)                │
│     - Load VPN modules                  │
└──────────────┬──────────────────────────┘
               │
┌──────────────▼──────────────────────────┐
│  6. Generate Quadlets                   │
│     - Build from config.toml            │
│     - Apply best practices              │
│     - Create networks                   │
└──────────────┬──────────────────────────┘
               │
┌──────────────▼──────────────────────────┐
│  7. Deploy                              │
│     - Write files                       │
│     - Reload systemd                    │
│     - Start services                    │
│     - Health checks                     │
└──────────────┬──────────────────────────┘
               │
┌──────────────▼──────────────────────────┐
│  8. Success Summary                     │
│     - Service URLs                      │
│     - Credentials                       │
│     - Next steps                        │
└─────────────────────────────────────────┘

🔧 Technical Architecture

Config-First Approach

Key Innovation: Generate quadlets FROM config, not config FROM quadlets

TUI → config.toml → QuadletGenerator → Quadlet Files → Deploy

Three Implementation Approaches

1. Template-First (Recommended)

Quadlet Templates → Render with config → Write files

Pros: Fast, predictable, easy to maintain
Cons: Less flexible, requires template per service

2. Dynamic Generation

Compose Files → Parse → Podlet Convert → Inject Config → Write

Pros: Flexible, handles new services easily
Cons: Complex, dependent on podlet tool

3. Hybrid

Compose Templates → Render → Podlet Convert → Post-process → Write

Pros: Best of both, flexible with validation
Cons: More complex, multi-stage process

Implementation Plan

Current: Focus on Go + Template-First

  • Go is working (5.9 MB binary)
  • Template-First is simplest
  • Rust after system Rust upgrade

Future: Evaluate other approaches as needed

Module Structure

ham-installer/
├── main.go                # Entry point & CLI
├── config/                # Config schema & loading
│   └── config.go
├── tui/                   # Terminal UI components
│   ├── account_wizard.go
│   ├── config_builder.go
│   ├── review.go
│   └── summary.go
├── approaches/            # Generation strategies
│   ├── template_first.go
│   ├── dynamic.go
│   └── hybrid.go
├── installer/             # Deployment logic
│   ├── deploy.go
│   ├── quadlet.go
│   └── verify.go
├── system/                # System configuration
│   ├── podman.go
│   ├── firewall.go
│   └── usb.go
├── template/              # Template rendering
│   └── render.go
└── password/              # Password managers
    ├── onepassword.go
    ├── vaultwarden.go
    └── vault.go

🎖️ Quadlet Best Practices

All quadlets include:

Health Checks

HealthCmd=/usr/bin/curl -f http://localhost:8123/ || exit 1
HealthInterval=30s
HealthRetries=3
HealthStartPeriod=90s
HealthOnFailure=restart

Auto-Updates

AutoUpdate=registry
Label=io.containers.autoupdate=registry

Service Dependencies

After=network-online.target
Wants=network-online.target
Requires=homelab-frontend.service

Resource Limits

MemoryHigh=1G
MemoryMax=1.5G
CPUQuota=150%

Restart Policies

Restart=always
RestartSec=30

Logging

SyslogIdentifier=homeassistant
StandardOutput=journal
StandardError=journal

Security

# Secrets
Secret=db-password,type=env,target=DB_PASSWORD

# Read-only root
ReadOnlyRootFilesystem=true

# Drop capabilities
DropCapability=ALL
AddCapability=NET_ADMIN

See QUADLET_BEST_PRACTICES.md for complete examples.

📅 Implementation Phases

Phase 0: Planning ✅ COMPLETE

Duration: 4 hours (November 20, 2025)

Tasks:

  • Analyze existing docker-compose files
  • Define service architecture
  • Design TUI flow
  • Create MASTER_PLAN
  • Document best practices
  • Consolidate all planning docs

Deliverables: ✅

  • Complete planning in MASTER_PLAN.md
  • 16 documentation files
  • Clear path forward

Phase 1: Quadlet Conversion ✅ COMPLETE

Duration: 3 hours (November 20, 2025)

Tasks:

  • Convert Caddy to quadlet
  • Convert Traefik to quadlet (alternative)
  • Convert Home Assistant stack (HA, MQTT, ESPHome, voice)
  • Create Tailscale quadlet
  • Create WireGuard quadlet (alternative)
  • Create NordVPN quadlet (alternative)
  • Convert Node-RED to quadlet
  • Convert Samba to quadlet
  • Convert Static Web Server to quadlet
  • Create Cloudflare Tunnel quadlet
  • Create Rclone backup quadlet + timer
  • Generate network definitions
  • Add Pi-hole quadlet
  • Validate all production-grade features

Deliverables: ✅

  • 22 quadlet files created
  • Both reverse proxy options (Caddy + Traefik)
  • All 3 VPN providers
  • 4 config templates (Caddyfile, Traefik configs)
  • Pi-hole with system setup notes
  • See: PHASE1_COMPLETE.md

Phase 2: Project Structures ✅ COMPLETE

Duration: 2 hours (November 20, 2025)

Tasks:

  • Initialize Rust project with Cargo
  • Initialize Go project with modules
  • Create directory structures
  • Add dependencies (CLI, config, logging)
  • Set up module structures
  • Create stub implementations
  • Add README documentation

Deliverables: ✅

  • Go: Working implementation (5.9 MB binary)
    • Compiles and runs successfully
    • Full CLI with all flags
    • 700 lines of code
    • See: implementations/go-installer/
  • Rust: Structure complete, needs Rust 1.56+
    • 460 lines of code
    • Version requirement documented
    • See: implementations/rust-installer/
  • See: PHASE2_SUCCESS.md

Key Learnings:

  • Go 1.25.4 works perfectly
  • Rust 1.41.1 too old (needs 1.56+ for Edition 2021)
  • Focus on Go first, Rust later
  • Template-First is simplest approach

Phase 3: Implementation 🔜 NEXT

Goal: Implement Template-First approach in Go

Duration: 7-10 days (estimated)

Focus: Go implementation first since it's working

Milestone 1: Template-First Approach (2-3 days)

  • Create templates/ directory
  • Convert Caddy quadlet to Go template
  • Implement template rendering engine
  • Test with various configurations
  • Expand to all services

Milestone 2: System Configuration (2-3 days)

  • Implement Podman socket enablement
  • Automate firewall configuration
  • Set up USB device access (for Home Assistant)
  • Load VPN kernel modules
  • Configure SELinux policies
  • Enable linger for rootless containers

Milestone 3: TUI Components (3-4 days)

  • Implement account wizard
    • Cloudflare setup guidance
    • VPN provider selection
    • Password manager integration
    • Cloud backup configuration
  • Implement service selection
    • Reverse proxy choice
    • Optional services
  • Implement configuration builder
    • Domains and subdomains
    • Storage auto-detection
    • Resource limits
  • Implement review screen
    • Show generated config.toml
    • Allow manual edits
    • Validate before deployment

Milestone 4: Deployment (1-2 days)

  • Implement file writing
  • Systemd reload/enable
  • Service startup with dependencies
  • Health check verification
  • Post-install summary

Optional: Additional Approaches

  • Dynamic generation (if needed)
  • Hybrid approach (if needed)
  • Rust implementation (after system upgrade)

Phase 4: Testing (3 days)

Goal: Validate implementation on test machines

Setup:

# Create test Podman machines
podman machine init ham-test-minimal --cpus 4 --memory 8192
podman machine init ham-test-full --cpus 4 --memory 8192
podman machine init ham-test-custom --cpus 4 --memory 8192

Test Scenarios:

  1. Minimal: Caddy + HA + Tailscale
  2. Full Free: All services with free options
  3. Custom: Traefik + WireGuard + custom paths

Metrics:

  • Deployment success rate (40%)
  • Configuration accuracy (30%)
  • Manual interventions (20%)
  • Performance (10%)

Phase 5: Evaluation & Polish (3-5 days)

Goal: Refine and prepare for release

Tasks:

  • Analyze test results
  • Fix critical bugs
  • Enhance error messages
  • Complete documentation
  • Create installation guide
  • User manual
  • Final QA
  • Release v1.0

⚙️ Configuration Schema

config.toml Format

[meta]
version = "3.0"
created = "2025-11-20"
installer_version = "0.1.0"

[global]
base_domain = "home.lan"
install_path = "/opt/homelab"
data_path = "/var/lib/homelab"
timezone = "America/New_York"
user_id = 1000
group_id = 1000

[network]
frontend_subnet = "10.88.0.0/24"
backend_subnet = "10.88.1.0/24"
vpn_subnet = "10.88.2.0/24"

[reverse_proxy]
provider = "caddy"  # or "traefik"
http_port = 80
https_port = 443
admin_email = "admin@example.com"
auto_https = true

[services]
enabled = ["pihole", "homeassistant", "node-red", "samba"]

[vpn]
provider = "tailscale"  # recommended: "tailscale" or "wireguard"
enable = true

[storage]
auto_detect = true
[[storage.pools]]
name = "main"
device = "/dev/sda1"
mount = "/mnt/main"
type = "ssd"

[cloudflare]
tunnel_token_source = "1password://vault/cloudflare/tunnel-token"
tunnel_name = "homelab"
zone_id = "abc123"

[domain]
provider = "porkbun"
domain = "yourdomain.com"
cloudflare_integration = "full"  # or "dns-only"

[password_manager]
provider = "1password"  # or "vaultwarden" or "vault"

[updates]
auto_update = true
schedule = "daily"
time = "02:00"
auto_rollback = true
notify_via = "email"

🧪 Testing Strategy

Test Categories

1. Unit Tests

  • Config validation
  • Template rendering
  • Quadlet parsing
  • Service dependencies

2. Integration Tests

  • Full workflow (mocked system)
  • Quadlet generation
  • Config file I/O

3. End-to-End Tests

  • Deploy to test Podman machine
  • Verify service health
  • Test rollback
  • Network validation

4. Evaluation Tests

  • Deployment success rate
  • Configuration accuracy
  • Manual intervention required
  • Performance metrics

Success Criteria

Metric Target
Deployment Success > 95%
Config Accuracy 100%
Manual Steps < 3
Startup Time < 5 min
Memory Usage < 100 MB

📈 Success Metrics

Phase Completion

  • ✅ Phase 0: Planning - COMPLETE
  • ✅ Phase 1: Quadlets - COMPLETE
  • ✅ Phase 2: Structures - COMPLETE
  • 🔜 Phase 3: Implementation - 50% ready
  • ⏳ Phase 4: Testing
  • ⏳ Phase 5: Polish

Code Metrics

  • Quadlets: 22 files ✅
  • Source Files: 32 ✅
  • Lines of Code: 1,160 ✅
  • Documentation: 16 files ✅

Quality Metrics

  • Production-Grade: 100% of quadlets ✅
  • Documentation Coverage: Comprehensive ✅
  • Test Coverage: TBD
  • Performance: TBD

📚 Related Documents

Core Documentation

  • PHASE1_COMPLETE.md - Phase 1 completion report
  • PHASE2_SUCCESS.md - Phase 2 completion report
  • PROGRESS_SUMMARY.md - Overall progress tracking
  • START_HERE.md - Quick start guide

Technical Documentation

  • QUADLET_BEST_PRACTICES.md - Production patterns
  • SYSTEM_SETUP_PLAN.md - System-level configuration
  • GUIDED_SETUP_DESIGN.md - TUI flow details

Implementation Documentation

  • implementations/STATUS.md - Implementation status
  • implementations/go-installer/README.md - Go docs
  • implementations/rust-installer/README.md - Rust docs

Reference Documentation

  • DECISIONS_NEEDED.md - Final decisions
  • EXTENDED_SERVICE_CATALOG.md - Future services
  • quadlets/PHASE1_SUMMARY.md - Quadlet summary

🎉 Current Achievements

Planning ✅

  • Single source of truth established
  • Comprehensive documentation
  • Clear architecture

Quadlets ✅

  • 22 production-grade files
  • All services converted
  • Best practices applied

Implementation ✅

  • Go installer working
  • Rust structure ready
  • 1,160 lines of code

Next ✨

  • Template-First in Go
  • System automation
  • TUI implementation
  • First deployments!

Ready to build the future of homelab deployments! 🚀

Last Updated: November 20, 2025