homelab-cloud

Personal experimental cloud that grew into a reusable onboarding path for services running on a shared k3s environment.

At A Glance

• Runs a personal k3s cloud for onboarding experimental services with Helm, Terraform, SSM-backed secrets, and Grafana-backed observability.
• Hosts analytics, load-test, AI workflow, and remote MCP services behind repeatable deployment and provisioning paths.
• Uses Tailscale for private admin and cluster paths, while Cloudflare Tunnel exposes only selected public HTTPS entrypoints.
• Keeps the control plane stable and attaches disposable AWS Spot workers for low-cost burst capacity, load tests, and stateless workloads.
• Treats each service as an onboarding contract: chart, cloud resources, secrets, deploy workflow, runtime checks, and feedback loop.

Why This Exists

I use AI to generate and iterate on a lot of service code, but the hard part is rarely just writing the code.

• local success does not prove k3s deploy success
• prototypes still need stable runtime paths when they interact with external apps such as Discord and ChatGPT
• infra and secrets provisioning should not be rebuilt manually per prototype
• post-deploy quality lanes need real runtime capacity, not only local mocks
• public projects need tighter secret, network, and deployment boundaries because AI makes code and config easier to inspect at scale

This repository is the shared runtime layer behind those experiments. App repos can stay focused on service behavior, while homelab-cloud owns the repeatable parts of getting a service live:

• expose the service through the right public or private entrypoint
• deploy the Helm release into the correct k3s namespace
• provision the cloud resources the service needs
• distribute runtime secrets without committing them to chart values
• attach worker capacity and observability before post-deploy checks run
• feed logs, metrics, workflow results, and incident notes back into the repo

Platform Overview

This diagram is intentionally control-plane-first: it shows homelab-cloud as the orchestration layer that owns deploy execution, runtime resource allocation, and perf-test environment provisioning. The source lives in docs/infra-overview.d2.

What It Proves

The repo is intentionally a personal lab, but the interesting part is that new services no longer start from a blank page. A workload can be onboarded by adding a chart, a Terraform entrypoint, secret mappings, deploy workflow, and observability surface, then reusing the same cluster and operator conventions.

Capability	How it shows up in this repo
Public service exposure	Cloudflare Tunnel is used for the remote MCP path, so an in-cluster service can be reached through public HTTPS without opening the cluster directly.
Private cluster path	Tailscale is used for k3s API access, CI-to-cluster reachability, operator devices, and spot-worker node registration across mixed local/cloud networks.
Elastic worker capacity	Terraform provisions the AWS VPC, launch template, and Spot Auto Scaling Group used to attach low-cost k3s worker capacity.
Cloud resource provisioning	Workload-specific Terraform stacks provision backing resources such as Neon, Upstash Redis, Kafka, Grafana dashboards, and per-service AWS SSM parameters.
Managed secret handoff	AWS SSM Parameter Store is the handoff point between Terraform and Kubernetes; External Secrets materializes only the runtime values each namespace needs.
Observability as code	Terraform provisions Grafana folders and dashboards for flashsale, strategy-tester, and cluster monitoring instead of treating dashboards as manual UI state.

Security And Network Overview

• Tailscale is the private path for node-to-node communication, k3s API access, GitHub Actions deploy jobs, and operator devices such as laptop and phone.
• Cloudflare Tunnel is the public edge for selected services only; it is not the cluster network or the admin path.
• Centralized secrets are provisioned into AWS SSM Parameter Store so service-level credentials can be rotated without committing secret material to the repo.
• External Secrets turns SSM parameters into namespace-local Kubernetes secrets, keeping chart values and runtime secrets separate.
• The public ingress surface is intentionally narrow, while the control plane and workers stay reachable through the private tailnet.

Budget Control

• One always-on control node stays small and stable because it is the cluster brain.
• AWS Spot workers are treated as disposable capacity for day-to-day experiments, load tests, and stateless workers.
• GitHub-hosted runners can be used for CI/CD after joining Tailscale, which avoids maintaining a dedicated public runner just to reach the private cluster.
• Managed pay-as-you-go services such as Neon, Upstash, Aiven Kafka, AWS SSM, and Grafana keep durable state outside nodes that can disappear.

Operating Model

The architecture is intentionally split between stable coordination, disposable compute, and externally managed state.

Layer	Role in this repo
Stable control plane	The k3s control node is the cluster brain. It owns scheduling, deployment coordination, and private cluster access, but is kept away from disposable or heavy batch work where possible.
Disposable workers	AWS Spot workers join the cluster through Terraform and Tailscale, then run stateless or bursty workloads such as load tests, background workers, and experimental services.
External state	Durable state is pushed out to managed services such as Neon, S3-backed Terraform state, AWS SSM Parameter Store, Grafana Cloud, Upstash Redis, and Aiven Kafka instead of being tied to a Spot node.
Private network	Tailscale connects the control node, AWS workers, GitHub Actions runners, and operator devices such as laptop or phone without exposing the k3s API as a public control path.
Public edge	Cloudflare Tunnel is used only for selected public service entrypoints, such as the remote MCP server. It is not the cluster network or the admin path.

This gives the lab a practical cost boundary: keep the always-on core small, scale cheap interruptible workers when experiments need capacity, and preserve durable state outside the nodes that can disappear.

Onboarded Projects

The platform currently runs a mix of application workloads, public/private control-plane services, and operator utilities. Each onboarded project follows the same rough contract: a deployable chart or external target, a secrets/provisioning path when needed, and a workflow entrypoint that can reconcile it from GitHub Actions.

Project	Runtime shape	Onboarding surface	What it proves
flashsales	k3s namespace with user, product, order, order-worker, and optional database proxy services	charts/flashsales, terraform/flashsale, deploy workflow, manual perf workflow	Experimental load-test playground only. It exists to exercise service onboarding, autoscaling, async terminalization, Redis admission control, Kafka terminalization, and Grafana-backed runtime analysis, not to serve a real business use case.
strategy-tester	k3s scheduled ingestion workload driven by CronJobs	charts/strategy-tester, terraform/strategy-tester, deploy workflow	Options-data ingestion can reuse the same deploy, External Secrets, worker placement, and dashboard provisioning path as larger services.
leetcode-intelligence	k3s intelligence API and background workers, plus Vercel client deployment	charts/leetcode-intelligence, terraform/leetcode-intelligence, service deploy workflow, client deploy workflow	A stateful personal AI workflow can split core services onto the private cluster while keeping the client on Vercel with SSM-synced runtime config.
seer-stone	public remote MCP service exposed through Cloudflare Tunnel	apps/seer-stone, charts/seer-stone, deploy workflow, infra workflow	Selected in-cluster MCP tools can be exposed through public HTTPS without opening the k3s control plane or worker nodes directly.
control-plane-triage-agent	long-running k3s control-plane agent for GitHub Actions failure diagnosis	apps/control-plane-triage-agent, charts/control-plane-triage-agent, terraform/control-plane-triage-agent, deploy workflow	Failed workflows can feed logs, Kubernetes diagnostics, service health, OpenHands analysis, and Discord summaries back into the operator loop.
node-disk-janitor	privileged k3s DaemonSet on cluster nodes	apps/node-disk-janitor, charts/node-disk-janitor, deploy workflow	Node disk pressure, stale jobs, empty namespaces, and Tailscale/k3s health can be handled as a reusable cluster utility instead of one-off shell maintenance.
control-plane-status	Vercel-only read-only status UI	apps/client, Vercel deploy workflow, boundary guard	The cluster can publish a lightweight operational view while keeping the UI outside the k3s runtime boundary.

Two shared infrastructure lanes support those projects:

• terraform/k3s-spot-network and terraform/k3s-spot-node provision the AWS Spot worker path that automatically joins the Tailscale-reachable k3s cluster.
• terraform/cluster-observability, deploy/monitoring, and the workload-specific dashboard stacks keep monitoring configuration versioned alongside deployment code.

Per-Project Provisioning Flow

This D2 diagram focuses on the onboarding contract: how one project-specific Terraform entrypoint provisions cloud resources, persists state, and then syncs runtime secrets into AWS SSM for Kubernetes consumption.

Read it left to right:

• each workload gets a project-owned Terraform entrypoint, such as terraform/flashsale/
• Terraform provisions only the cloud resources that belong to that project
• Terraform keeps desired state in the shared S3 backend, but writes runtime credentials into AWS SSM Parameter Store
• Kubernetes does not read Terraform state directly; External Secrets Operator reads the project SSM path and materializes a namespaced Secret
• the Helm release consumes that Secret at deploy time, so apps get DATABASE_URL, REDIS_URL, tokens, and other sensitive values without committing them to values.yaml

For flashsales, the current aggregate stack is:

• terraform/flashsale provisions Neon, Upstash Redis, Aiven Kafka, Grafana dashboards, and SSM parameters
• the default SSM path is /flashsales/prod/*
• the charts/flashsales chart can read those values through ExternalSecret

What This Repo Schedules

Control-plane responsibility	What it means here
Service onboarding	Charts, Terraform stacks, SSM key maps, and deploy workflows define how a workload joins the shared runtime
Deploy orchestration	GitHub Actions drives Helm-based rollout into the shared k3s cluster
Resource allocation	Terraform and cluster config provision databases, queues, secrets, dashboards, and spot-backed worker capacity
Runtime quality provisioning	Manual perf workflows prepare the live runtime needed by flashsale app-owned quality lanes
Feedback loop	Workflow logs, Discord notifications, Grafana, and docs feed runtime behavior back into the repo

Start Here

Page	What it covers
Repository overview	High-level architecture, layout, and shared conventions
Wiki	Mirrored investigations, ADRs, and workload notes with stable root-level paths
Onboarded projects	Current workload catalog, runtime shape, and deploy/provisioning entrypoints
Flashsales chart	Platform-side Kubernetes release for the flashsales workload
Flashsales Terraform	Neon, Redis, Kafka, Grafana, and SSM provisioning for flashsales
Flashsales deploy	Platform-side deploy executor for the shared k3s runtime
Flashsales perf test	Manual platform-side executor for the app-owned flashsale perf cadence
Strategy tester workload	Scheduled ingestion app, cron jobs, and secret wiring
LeetCode intelligence chart	Continuous intelligence API, background workers, and optional submission service
Remote MCP service	Public MCP endpoint, stable bearer auth, and Cloudflare Tunnel wiring
Control-plane triage agent	GitHub Actions failure watcher with Kubernetes diagnostics and Discord summaries
Node disk janitor	Privileged DaemonSet for bounded node cleanup and Tailscale/k3s self-healing
Control-plane status UI	Vercel-only read-only status dashboard
Infrastructure	Terraform-backed resource provisioning for Neon, SSM, networking, and worker capacity
Operations and tooling	CI/CD, runtime gates, perf workflows, and operator commands

If you only need one place to orient yourself, start with Repository overview.

Quick Commands

make deploy KUBECONFIG_PATH=$HOME/.kube/config
make status KUBECONFIG_PATH=$HOME/.kube/config
make e2e KUBECONFIG_PATH=$HOME/.kube/config
make concurrency-baseline KUBECONFIG_PATH=$HOME/.kube/config
make k3s-spot-plan

These commands are operator entrypoints into the same control plane:

• make deploy: reconcile a workload release into k3s
• make status: inspect live runtime state
• make e2e: exercise the deployed path
• make concurrency-baseline: provision and run a baseline perf lane
• make k3s-spot-plan: inspect worker-capacity allocation changes

Repository Layout

.
├── .github/workflows/      # Deploy, post-deploy runtime gates, and infra automation entrypoints
├── .github/scripts/        # Workflow-side orchestration helpers
├── apps/                   # In-repo services and operator utilities
├── charts/                 # Helm release definitions for platform-managed workloads
├── application/            # App submodules plus workload-owned quality contracts
├── wiki/                   # Root entrypoint to mirrored wiki content
├── terraform/              # Resource allocation for Neon, SSM, networking, and spot-backed capacity
├── docs/                   # Platform docs and control-plane diagrams
└── Makefile                # Operator-facing deploy and diagnostics commands

For workflow-specific guidance, see Operations and tooling.