docs: add runbooks, troubleshooting, onboarding, ADRs, and Mermaid diagrams

Add operational runbooks (restart, add app, rotate credentials, restore
backup, debug deploy), symptom-based troubleshooting guide, onboarding
checklist, and 5 architecture decision records. Replace ASCII art in
architecture.md with Mermaid diagrams for network topology, CI/CD flow,
backup/restore, and preview environments. Update CONTRIBUTING.md with
validator and infrastructure sections. Add new pages to mkdocs nav.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Author: baijum

CONTRIBUTING.md

@@ -38,6 +38,37 @@ mkdocs serve
 
 Then open [http://127.0.0.1:8000](http://127.0.0.1:8000) in your browser. The site auto-reloads on file changes.
 
+## Spec Validator
+
+The platform includes a validator that checks app repositories against the [application specification](docs/spec.md). Run it against any app directory:
+
+```bash
+python validator/validate.py /path/to/your-app
+```
+
+The validator checks three tiers: file structure, configuration, and runtime compliance. All tiers should pass before deploying.
+
+## Infrastructure Scripts
+
+Scripts in the `infrastructure/` directory follow these conventions:
+
+- **ShellCheck clean** — all scripts pass `shellcheck` with no warnings
+- **Idempotent** — every section is guarded with existence checks; safe to re-run
+- **No interactive prompts** in automated scripts (cron jobs, CI steps)
+- **Documented** — see `docs/server-contract.md` for the full scripts reference table
+
+When modifying infrastructure scripts, test on a fresh Debian 12 server or verify with `infrastructure/verify-server.sh`.
+
+## Operational Tasks
+
+For day-to-day server operations, see the [runbooks](docs/runbooks/):
+
+- [Restart an app](docs/runbooks/restart-app.md)
+- [Add a new app](docs/runbooks/add-new-app.md)
+- [Rotate credentials](docs/runbooks/rotate-credentials.md)
+- [Restore a backup](docs/runbooks/restore-backup.md)
+- [Debug a failed deploy](docs/runbooks/debug-failed-deploy.md)
+
 ## Code of Conduct
 
 This project follows the [Contributor Covenant v2.1](CODE_OF_CONDUCT.md). By participating, you agree to uphold its standards.

README.md

@@ -80,6 +80,10 @@ Full documentation is available at **[towlion.github.io/platform](https://towlio
 - [Architecture](https://towlion.github.io/platform/architecture/) — platform design and diagrams
 - [App Specification](https://towlion.github.io/platform/spec/) — application contract (ports, endpoints, env vars)
 - [Self-Hosting](https://towlion.github.io/platform/self-hosting/) — fork model, server requirements, bootstrap
+- [Troubleshooting](https://towlion.github.io/platform/troubleshooting/) — symptom-based debugging guide
+- [Onboarding](https://towlion.github.io/platform/onboarding/) — new contributor checklist
+- [Runbooks](https://towlion.github.io/platform/runbooks/restart-app/) — operational procedures
+- [Architecture Decisions](https://towlion.github.io/platform/decisions/001-github-as-control-plane/) — ADRs
 
 ## License
 

docs/architecture.md

@@ -4,26 +4,36 @@
 
 The Towlion platform runs on a single Debian server. Applications run as Docker containers and share a set of core infrastructure services.
 
+```mermaid
+graph TB
+    User[User Browser] -->|HTTPS| Caddy
+
+    subgraph Server["Debian 12 Server"]
+        subgraph Docker["Docker / towlion network"]
+            Caddy["Caddy :80/:443"]
+
+            Caddy --> App1["App 1 :8000"]
+            Caddy --> App2["App 2 :8000"]
+            Caddy --> App3["App 3 :8000"]
+            Caddy --> Grafana["Grafana :3000"]
+
+            App1 --> Postgres[("PostgreSQL :5432")]
+            App2 --> Postgres
+            App3 --> Postgres
+            App1 --> Redis[("Redis :6379")]
+            App1 --> MinIO["MinIO :9000"]
+
+            Promtail["Promtail"] --> Loki["Loki :3100"]
+            Loki --> Grafana
+
+            Prometheus["Prometheus :9090"] -.->|optional| Grafana
+            cAdvisor["cAdvisor :8080"] -.->|optional| Prometheus
+            NodeExp["Node Exporter :9100"] -.->|optional| Prometheus
+        end
+    end
 ```
-                Internet
-                    │
-                    ▼
-                  DNS
-                    │
-                    ▼
-              Reverse Proxy
-                 (Caddy)
-            /       |       \
-           ▼        ▼        ▼
-       App 1     App 2     App 3
-           │
-           ▼
-      Shared Services
-           │
-  ┌────────┼─────────┬─────────┐
-  ▼        ▼         ▼         ▼
-PostgreSQL Redis   MinIO    Workers
-```
+
+Dashed lines indicate optional metrics services (enabled via `COMPOSE_PROFILES=metrics`).
 
 ## Technology Stack
 
@@ -214,6 +224,71 @@ pg_dump → /data/backups
 
 Backups can be synced to remote storage using `rclone`.
 
+## CI/CD Flow
+
+```mermaid
+graph LR
+    Push["git push to main"] --> Actions["GitHub Actions"]
+
+    subgraph Actions["GitHub Actions"]
+        Test["Test Job"] --> Deploy["Deploy Job"]
+    end
+
+    Deploy -->|SSH| Server["Server"]
+
+    subgraph Server["Server Operations"]
+        Pull["git pull"] --> Build["docker compose up -d --build"]
+        Build --> Trivy["Trivy image scan"]
+        Build --> Migrate["Alembic migrate"]
+        Migrate --> CaddyWrite["Write Caddyfile"]
+        CaddyWrite --> Reload["Caddy reload"]
+    end
+```
+
+## Backup and Restore Flow
+
+```mermaid
+graph LR
+    subgraph Backup["Daily Backup (cron 02:00)"]
+        Cron["cron"] --> Script["backup-postgres.sh"]
+        Script --> PgDump["pg_dump per database"]
+        PgDump --> Compress["gzip"]
+        Compress --> Store["/data/backups/postgres/"]
+        Store --> Prune["Prune backups > 7 days"]
+    end
+
+    subgraph Restore["Manual Restore"]
+        List["List backups"] --> Choose["Choose backup file"]
+        Choose --> RestoreScript["restore-postgres.sh"]
+        RestoreScript --> Drop["Drop + recreate DB"]
+        Drop --> Import["Import from backup"]
+        Import --> Verify["Verify data"]
+    end
+```
+
+## Preview Environment Flow
+
+```mermaid
+graph TB
+    subgraph Create["PR Opened / Updated"]
+        PR["Pull Request"] --> Actions["GitHub Actions"]
+        Actions -->|SSH| Clone["Clone to /opt/apps/app-pr-N/"]
+        Clone --> Schema["Create PR-specific DB schema"]
+        Schema --> BuildPR["docker compose up -d --build"]
+        BuildPR --> CaddyPR["Write pr-N.preview.domain.caddy"]
+        CaddyPR --> ReloadPR["Caddy reload"]
+        ReloadPR --> Comment["Comment preview URL on PR"]
+    end
+
+    subgraph Cleanup["PR Closed / Merged"]
+        Closed["PR closed"] --> StopContainers["Stop + remove containers"]
+        StopContainers --> DropSchema["Drop PR schema"]
+        DropSchema --> RemoveCaddy["Remove .caddy file"]
+        RemoveCaddy --> ReloadClean["Caddy reload"]
+        ReloadClean --> RemoveDir["Remove /opt/apps/app-pr-N/"]
+    end
+```
+
 ## Docker Compose Services
 
 Each application lives in its own GitHub repository under the `towlion` organization. The server runs two layers of Compose services:
@@ -249,9 +324,8 @@ services:
     env_file: .env
   frontend:
     build: ./frontend
-  celery-worker:
-    build: ./app
-    command: celery -A app.tasks worker
 ```
 
+Celery workers are opt-in. To add background task processing, add a `celery-worker` service to your compose file. See the [app-template README](https://github.com/towlion/app-template#background-tasks) for instructions.
+
 For single-app self-hosting (fork scenario), a repository may bundle platform services in its own Compose file so it can run standalone.

docs/decisions/001-github-as-control-plane.md

@@ -0,0 +1,43 @@
+# ADR 001: GitHub as the Control Plane
+
+**Date:** 2025-12-01
+**Status:** Accepted
+
+## Context
+
+Traditional PaaS platforms (Heroku, Render, Fly.io) require a dedicated control plane — a custom API, dashboard, or CLI that manages deployments, configuration, and access control. Building and maintaining a control plane is a significant engineering effort and introduces another system to secure and operate.
+
+Towlion targets indie developers and small projects. The overhead of a custom control plane would outweigh its benefits for this audience.
+
+## Decision
+
+Use GitHub as the control plane for all platform operations:
+
+- **CI/CD**: GitHub Actions workflows handle building, testing, and deploying
+- **Configuration**: GitHub Secrets store deployment credentials
+- **Access control**: GitHub repository permissions manage who can deploy
+- **Workflow orchestration**: Actions workflows coordinate multi-step deployments
+- **Issue tracking**: GitHub Issues for alerts (created by `check-alerts.sh`)
+
+No custom dashboard, API server, or CLI is needed. The deployment flow is:
+
+```
+GitHub repository -> GitHub Actions -> SSH -> Docker runtime
+```
+
+## Consequences
+
+**Benefits:**
+
+- Zero infrastructure to build or maintain for the control plane
+- Familiar interface — developers already use GitHub daily
+- Built-in audit trail via commit history and Actions logs
+- Free for public repos; generous free tier for private repos
+- Access control, 2FA, and SSO handled by GitHub
+
+**Trade-offs:**
+
+- Vendor dependency on GitHub (Actions, Secrets, API)
+- Limited to what GitHub Actions can express (no custom UI for deployments)
+- Secrets management is per-repository, not centralized
+- No real-time deployment dashboard — must check Actions tab for status

docs/decisions/002-caddy-over-nginx.md

@@ -0,0 +1,29 @@
+# ADR 002: Caddy Over Nginx for Reverse Proxy
+
+**Date:** 2025-12-01
+**Status:** Accepted
+
+## Context
+
+The platform needs a reverse proxy to route traffic to application containers and handle TLS certificate provisioning. The two main contenders are Nginx (with certbot/Let's Encrypt) and Caddy.
+
+## Decision
+
+Use Caddy as the reverse proxy.
+
+## Consequences
+
+**Benefits:**
+
+- **Automatic TLS** — Caddy provisions and renews Let's Encrypt certificates with zero configuration. No certbot cron jobs, no manual renewal scripts.
+- **Simple configuration** — A Caddyfile is shorter and more readable than equivalent Nginx config. Adding a new app route is a 3-line file.
+- **Per-app config fragments** — The `import /etc/caddy/apps/*.caddy` pattern allows deploy workflows to write individual `.caddy` files per app without editing a monolithic config.
+- **Hot reload** — `caddy reload` applies config changes without dropping connections. No `nginx -s reload` dance.
+- **Single binary** — No module system, no package dependencies. The official Docker image works out of the box.
+
+**Trade-offs:**
+
+- Nginx has broader community knowledge and more Stack Overflow answers
+- Nginx supports more advanced configurations (TCP/UDP proxying, complex rewrites)
+- Caddy is less battle-tested at high scale (not a concern for single-server deployments)
+- Some hosting tutorials assume Nginx, requiring translation to Caddy equivalents

docs/decisions/003-single-server-architecture.md

@@ -0,0 +1,34 @@
+# ADR 003: Single-Server Architecture
+
+**Date:** 2025-12-01
+**Status:** Accepted
+
+## Context
+
+Modern deployment platforms typically use container orchestration (Kubernetes, Docker Swarm, Nomad) to distribute workloads across multiple servers. This provides high availability, auto-scaling, and fault tolerance — but at significant operational complexity.
+
+Towlion targets indie developers, small SaaS products, and hobby projects. These applications typically serve a handful of users and run comfortably on a single server.
+
+## Decision
+
+Limit the platform to a single Debian server. All applications, databases, and infrastructure services run on one machine using Docker Compose.
+
+## Consequences
+
+**Benefits:**
+
+- **Simplicity** — no cluster management, no service mesh, no distributed consensus
+- **Low cost** — a single $12-24/month VPS runs the entire platform
+- **Easy debugging** — everything is on one machine; `docker logs` and `docker exec` are sufficient
+- **Fast deploys** — no image registry push/pull; images build locally from source
+- **Predictable** — no network partitions, no node failures, no split-brain scenarios
+
+**Trade-offs:**
+
+- Single point of failure — if the server goes down, all apps go down
+- No auto-scaling — resource limits are static
+- Brief downtime during deploys (container rebuild takes a few seconds)
+- Vertical scaling only — must move to a bigger server when capacity is reached
+- Not suitable for high-traffic applications or strict uptime requirements
+
+These are intentional design boundaries, not missing features. For workloads that need HA or auto-scaling, use Kubernetes, Fly.io, or a cloud-native platform instead.

docs/decisions/004-fork-based-self-hosting.md

@@ -0,0 +1,44 @@
+# ADR 004: Fork-Based Self-Hosting
+
+**Date:** 2025-12-01
+**Status:** Accepted
+
+## Context
+
+Self-hosted software typically uses one of these distribution models:
+
+1. **Installer script** — download and run a setup script (e.g., `curl | bash`)
+2. **Docker image** — pull a pre-built image from a registry
+3. **Helm chart / Terraform module** — declarative infrastructure definition
+4. **Fork** — fork the repository and deploy from your own copy
+
+Each model has different trade-offs for isolation, customization, and update management.
+
+## Decision
+
+Use the fork model for self-hosting. Each operator forks the application repository, configures their own GitHub Secrets, and deploys from their fork.
+
+```
+Original repo -> fork by Alice -> alice.example.com
+                -> fork by Bob   -> bob.example.com
+                -> fork by Carol -> carol.example.com
+```
+
+## Consequences
+
+**Benefits:**
+
+- **Strong isolation** — each fork is completely independent; no shared tenancy
+- **Full customization** — operators can modify any code, workflow, or configuration
+- **No installer to maintain** — the repository is the installer
+- **GitHub handles updates** — operators can sync upstream changes via GitHub's fork sync
+- **Built-in CI/CD** — GitHub Actions workflows come with the fork
+- **Transparent** — all deployment logic is visible in the repository
+
+**Trade-offs:**
+
+- Operators must manage their own GitHub Secrets and server
+- Upstream updates require manual sync (GitHub's "Sync fork" button)
+- No centralized management across forks — each is fully independent
+- Requires GitHub account and understanding of fork workflows
+- Private forks require a GitHub paid plan

docs/decisions/005-apparmor-over-selinux.md

@@ -0,0 +1,35 @@
+# ADR 005: AppArmor Over SELinux
+
+**Date:** 2026-03-01
+**Status:** Accepted
+
+## Context
+
+Mandatory Access Control (MAC) adds a security layer beyond standard Unix permissions. The two main MAC systems on Linux are AppArmor and SELinux. The platform runs on Debian 12, which ships with AppArmor enabled by default.
+
+## Decision
+
+Use AppArmor (Debian's native MAC system) instead of SELinux.
+
+Docker automatically applies the `docker-default` AppArmor profile to all containers, which restricts capabilities like writing to `/proc` and `/sys`, mounting filesystems, and accessing raw sockets. No additional configuration is needed.
+
+## Consequences
+
+**Benefits:**
+
+- **Zero configuration** — AppArmor is already active on Debian 12 out of the box
+- **Docker integration** — Docker applies the `docker-default` profile automatically to every container
+- **Debian-native** — maintained by the Debian security team, well-tested with Debian packages
+- **Simple profile model** — AppArmor profiles are path-based and easier to read than SELinux policies
+
+**Trade-offs:**
+
+- AppArmor is less granular than SELinux (path-based vs. label-based)
+- SELinux is the standard on RHEL/Fedora ecosystems; cross-distro documentation often assumes SELinux
+
+**Why not SELinux on Debian:**
+
+- SELinux policies on Debian are incomplete — the `selinux-policy-default` package lags far behind RHEL equivalents
+- Docker + SELinux on Debian causes bind-mount labeling issues (`:z`/`:Z` volume flags) with no community support
+- Enabling SELinux requires disabling AppArmor, losing Docker's automatic profile enforcement
+- The Debian security team does not maintain SELinux policies to the same standard as AppArmor