User Story: Migration to VM-Based Deployment With Automated CI/CD

[PhilipWoulfe]

Title:

As a developer and operator, I want my application to run inside a dedicated VM with an automated deployment pipeline so that updates are reliable, repeatable, and never impact the Proxmox host.
Problem / Pain Points

The application currently runs inside an unprivileged LXC, causing Docker to run in rootless mode and leak processes onto the Proxmox host.

This results in:

    host-level Docker daemons spawning

    veth storms and CNI bridges on the host

    kernel log spam

    instability when containers crash

Deployments require:

    manually editing .env files

    manually running docker compose up

    SSHing into the container

The process is error‑prone, slow, and not suitable for production.

Goal / Outcome

Move the application into a dedicated VM and implement a zero‑touch deployment pipeline so that:

Docker runs safely inside the VM, isolated from the host.

The Proxmox node remains stable regardless of app failures.

Deployments happen automatically on push or tag.

Environment variables are managed centrally and securely.

No manual SSH or docker compose up is required.

Acceptance Criteria
Infrastructure

A new VM is created (Ubuntu or Debian).

Docker and Docker Compose are installed inside the VM.

The VM is configured with:

    static IP

    firewall rules

    persistent storage

    automatic updates for security patches

Application Deployment

The application runs via Docker Compose inside the VM.

The VM is the only place where the app runs.

No Docker processes appear on the Proxmox host.

CI/CD Pipeline

A Git repository contains:

    Dockerfile

    docker-compose.yml

    environment template

A pipeline (GitHub Actions, GitLab CI, or similar):

    builds the image

    pushes to a registry

    triggers a deploy on the VM

The VM pulls the new image and restarts the stack automatically.

Environment Management

Environment variables are stored in:

    a .env file managed by the pipeline, or

    a secrets manager (Vault, Doppler, GitHub Secrets, etc.)

No manual editing of .env on the server.

Deployment Flow

Developer pushes to main or tags a release.

CI builds and publishes the image.

VM receives a webhook or SSH trigger.

VM runs docker compose pull && docker compose up -d.

Deployment completes without manual intervention.

Non‑Functional Requirements

Deployment must be repeatable and idempotent.

VM must remain isolated from the Proxmox host.

Logs must be accessible centrally (e.g., Loki, ELK, or simple journald + docker logs).

Rollbacks must be possible by redeploying a previous image tag.

Success Metrics

Zero Docker processes on the Proxmox host.

Zero manual SSH steps for deployment.

Deployment time reduced to under 1 minute.

No kernel log spam or veth churn during app crashes.

VM uptime remains stable even during app failures.

[PhilipWoulfe]

Plan: VM Isolated Zero-Touch Deployment
Recommended approach: keep your current image build/promotion pipeline, move runtime fully to a dedicated VM, and add one automated deploy stage that renders environment from GitHub Secrets and performs pull/restart/smoke checks over SSH. This gives isolation from Proxmox host issues while minimizing churn in your existing repo.

Decisions captured from you

Deploy trigger: CI SSH push from GitHub Actions.
Secrets source: GitHub Encrypted Secrets with pipeline-rendered .env on VM.
Rollback: manual redeploy of a previous image tag.
OS: open choice; recommendation is Ubuntu 24.04 LTS unless you prefer Debian 12.
Steps

Phase 1: Finalize target topology and cutover plan.

Confirm VM-only runtime boundary (no app containers on Proxmox host), static IP, and firewall allowlist.

Define persistent storage and log path policy (for example /mnt/f1-logs) plus retention/backup.

Define cutover checkpoints and rollback checkpoint (old LXC preserved but inactive during burn-in).

Phase 2: Provision and harden VM.

Provision VM using repeatable automation (template/cloud-init).

Install Docker Engine + Compose plugin with pinned setup.

Configure hardening: unattended security updates, SSH key-only auth, least-privileged deploy user, firewall defaults.

Create runtime dirs and permissions for app and logs.

Phase 3: Compose/runtime hardening in repo.

Ensure production compose path uses tagged images only and no dev bind-mount assumptions.

Add/verify API and Web healthchecks and health-aware startup dependencies.

Move hardcoded environment constants into .env-driven variables.

Add deploy preflight checks (required env vars, writable log path, minimum disk space).

Ensure rollback safety by retaining enough prior image tags in production path.

Phase 4: CI/CD deploy automation.

Reuse existing build/test/publish/promotion flow to keep sha-*, test, stable tags.

Add VM deploy workflow/job after production promotion approval:

SSH to VM.

Render/sync .env from GitHub Secrets.

Run compose pull + up.

Run smoke checks (API health + Web reachability).

Emit deployment metadata (tag/SHA/time/host) in workflow summary.

Add rollback workflow path with tag input to redeploy any previous image.

Phase 5: Validation and cutover.

Verify idempotent deploy (same tag deploy twice, no drift).

Verify host isolation (no Docker daemons/veth churn on Proxmox host).

Verify logs are accessible (baseline journald + docker logs).

Execute traffic cutover and burn-in monitoring window.

Finalize operator runbook for deploy, rollback, and incident checks.

Relevant files

docker-compose.yml — runtime service definitions, env wiring, restart/health behavior.
.env.example — canonical env template for pipeline rendering.
README.md — deployment/runbook docs and environment guidance.
docker-build.yaml — existing build/test/publish/promotion chain.
workflows — add deploy and rollback workflows.
scripts — add preflight/deploy helper logic.
Dockerfile — API healthcheck support.
Dockerfile — Web/nginx healthcheck support.
Verification

Push-to-deploy path works with zero manual SSH steps.
Deployment is idempotent (repeat same tag deploy safely).
Rollback by previous tag succeeds and restores service quickly.
Proxmox host remains clean during app crash/restart scenarios.
No plaintext secrets appear in repo or workflow logs.
Redeploy path (no rebuild) is under 1 minute.
Scope boundaries

Included: VM migration design, compose hardening, CI deploy automation, rollback workflow, ops runbook.
Excluded for this phase: full IaC platform overhaul (Terraform/Ansible end-to-end), HA/multi-region architecture, full observability platform migration.

[PhilipWoulfe]

Epic: Dedicated VM Deployment and Zero-Touch Release
Summary
Move F1Competition out of the current Proxmox LXC and into a dedicated VM, keep Docker isolated inside that VM, and complete the CI/CD path so deployments are automatic, repeatable, and do not require manual SSH or ad hoc docker compose up steps.

Problem Statement
The current LXC-based runtime causes Docker side effects to leak onto the Proxmox host:

host-level Docker daemons
veth/CNI churn
kernel log noise
instability during crashes
Operationally, deployment still depends on:

hand-edited .env files
manual compose commands
SSHing into the runtime target
That is not a production-grade deployment model.

Goal / Outcome
Docker runs only inside a dedicated VM.
The Proxmox host is unaffected by app/container failures.
CI builds and publishes images automatically.
CI deploys the app automatically to the VM.
Runtime env/secrets are centrally managed.
Rollback is possible by redeploying a prior tag.
Public ingress and private operations are separated cleanly:
Cloudflare for ingress, Tailscale for admin/deploy access.
Scope
In scope:

Dedicated Ubuntu 24.04 LTS VM runtime
Docker Engine + Compose plugin inside VM
Static VM IP
Firewall rules and security updates
Cloudflare for public ingress
Tailscale for private operator/deploy access
GitHub Actions deploy workflow
Pipeline-rendered .env from GitHub Secrets
Rollback-by-tag workflow
Compose/runtime hardening
Health checks, preflight checks, and operator runbook
Out of scope:

Full Terraform/Ansible automation for Proxmox estate
HA / blue-green / multi-node rollout
Full observability platform rollout beyond baseline logging
Secrets-manager migration beyond GitHub Secrets for this phase
Architecture Decision
Public traffic path:
Cloudflare -> VM-hosted app stack
Private operator/deploy path:
Tailscale -> VM
Deployment trigger:
GitHub Actions deploy job over Tailscale-routed SSH
Runtime source of truth:
docker-compose.yml + generated .env
Default schema migration owner:
API, not worker
Story Breakdown
Story 1: VM Provisioning Baseline
Objective:
Create a dedicated VM that can safely host Docker without affecting the Proxmox host.

Done when:

VM exists with static IP and persistent storage.
Docker Engine and Compose plugin are installed.
Automatic security updates are enabled.
Firewall rules are applied.
Public SSH is disabled or tightly blocked.
Tailscale admin path is available.
Story 2: Network and Access Topology
Objective:
Split public ingress from private admin traffic.

Done when:

Cloudflare handles public DNS/TLS/tunnel/edge access.
Tailscale provides private deploy/admin access to the VM.
Deployment does not require public SSH exposure.
Network diagram and firewall expectations are documented.
Story 3: Compose Runtime Hardening
Objective:
Make production compose deterministic and safe for VM runtime.

Done when:

docker-compose.yml reflects production-safe restart policies.
Worker/API migration ownership is unambiguous.
Health checks exist where required.
Runtime directories and log mounts are documented and validated.
Compose config can be rendered and validated non-interactively.
Story 4: Environment and Secret Management
Objective:
Remove manual server-side .env editing.

Done when:

.env.example is the canonical env contract.
GitHub Secrets hold deploy-time values.
Deploy pipeline renders or syncs .env to the VM automatically.
README.md documents env ownership and consumers.
Story 5: Automated VM Deployment Workflow
Objective:
Complete the final mile from built images to running containers on the VM.

Done when:

GitHub Actions deploy job connects over Tailscale/private SSH.
Workflow pulls new images and runs compose update automatically.
Smoke checks confirm API and Web health.
Workflow emits deploy metadata: tag, SHA, host, timestamp.
Story 6: Rollback Workflow
Objective:
Make rollback explicit and fast.

Done when:

A prior tag can be redeployed via workflow input.
Rollback uses the same deploy path as forward deploy.
Runbook documents rollback verification.
Image retention policy keeps rollback practical.
Story 7: Cutover from LXC Runtime
Objective:
Move the active workload off the LXC path and onto the VM safely.

Done when:

VM is the only active runtime target.
LXC runtime path is disabled or left off only as short rollback safety.
Proxmox host shows no app Docker daemons or veth churn after cutover.
End-to-end traffic is served from VM runtime.
Story 8: Operations Runbook and Verification
Objective:
Give operators a repeatable deploy/recovery process.

Done when:

README.md and HANDOVER.MD include deploy, rollback, smoke-check, and log-inspection steps.
Repeat deploy with same tag is proven idempotent.
Crash/restart behavior is tested and documented.
Success metrics can be measured post-cutover.
Suggested PR Sequence
PR1: Runtime and Network Hardening
Files:

docker-compose.yml
.env.example
README.md
scripts
Focus:

Cloudflare/Tailscale split
migration-owner defaults
runtime validation and preflight checks
health and restart policy alignment
PR2: CI/CD Deploy and Rollback
Files:

docker-build.yaml
new deploy workflow in workflows
new rollback workflow in workflows
Focus:

generated .env
Tailscale-routed deploy
smoke checks
deploy audit output
rollback-by-tag
PR3: Cutover and Runbook
Files:

README.md
HANDOVER.MD
optional helpers in scripts
Focus:

cutover checklist
rollback drill
host cleanliness verification
operator handoff
Acceptance Criteria
Infrastructure:

Dedicated VM created and configured.
Docker and Compose installed only inside VM.
Static IP, firewall rules, persistent storage, and automatic updates configured.
Application Deployment:

App runs via Compose inside VM.
VM is the only active runtime target.
No app Docker processes appear on Proxmox host.
CI/CD:

Images build and publish from CI.
CI triggers VM deployment automatically after approval/promotion.
VM pulls new image tags and restarts stack automatically.
Environment Management:

No manual editing of server .env required.
Runtime env values come from GitHub Secrets-rendered .env or equivalent pipeline-managed source.
Deployment Flow:

Push or release tag triggers build/publish.
Deploy job updates VM automatically.
Deployment completes without manual SSH steps.
Non-Functional:

Deploy is repeatable and idempotent.
VM remains isolated from Proxmox host.
Logs are accessible via documented baseline path.
Rollback by previous image tag is supported.
Success Metrics
Zero Docker processes on Proxmox host for app runtime.
Zero manual SSH steps during normal deployment.
Redeploy path under 1 minute.
No kernel log spam or veth churn on Proxmox during app failures.
VM remains stable during container crash loops.
Risks and Mitigations
Split ingress/admin topology adds complexity.
Mitigation: keep Cloudflare for public path and Tailscale only for admin/deploy path; document clearly.

Migration ownership ambiguity between API and worker.
Mitigation: keep API as sole default migration owner and document override behavior.

Rollback becomes impossible if images are cleaned too aggressively.
Mitigation: define retention policy and avoid destructive cleanup in production path.

Environment drift between repo template and deployed VM.
Mitigation: generate .env from pipeline using .env.example as the contract.

Recommended Starting Point
Start with PR1. It has the least deployment risk and creates the contract the later workflows depend on.

Immediate first tasks for PR1:

finalize env contract in .env.example
externalize any remaining hardcoded deploy/runtime values in docker-compose.yml
document Cloudflare vs Tailscale responsibilities in README.md
add a deploy preflight script in scripts