Skip to content

README.md

Latest commit

 

History

History
437 lines (305 loc) · 12.7 KB

README.md

File metadata and controls

437 lines (305 loc) · 12.7 KB
Raw

Infrastructure & Deployment Guide

This directory contains Terraform configuration for provisioning Hetzner Cloud infrastructure and the CD pipeline setup for automated deployments.

Folder Structure

infra/
├── modules/                       # Reusable Terraform modules
├── environments/                  # Environment-specific configurations
└── scripts/                       # Deployment scripts
    └── deploy-to-server.sh        # Server deployment script (used by CD pipeline)

Table of Contents

Quick Start

  1. Configure GitHub secrets (see CD Pipeline Setup)

  2. Push to main branch: The CD pipeline will automatically:

    • Provision infrastructure (if needed)
    • Build and push Docker image
    • Deploy to all servers

That's it! Everything is fully automated.

Prerequisites

  1. Hetzner Cloud Account: Sign up at https://www.hetzner.com/cloud
  2. Hetzner API Token: Create a token at https://console.hetzner.com/projects/{project-id}/security/tokens
  3. Terraform: Install Terraform >= 1.0 from https://www.terraform.io/downloads
  4. GitHub Repository: Your code should be in a GitHub repository
  5. SSH Key Pair: Generate if you don't have one: 
    ssh-keygen -t ed25519 -C "your_email@example.com"

Infrastructure Setup

Infrastructure provisioning is fully automated by the CD pipeline. When you push to main or master, the pipeline will:

  1. Initialize Terraform
  2. Plan infrastructure changes
  3. Apply infrastructure (create/update servers)
  4. Get server IPs from outputs
  5. Deploy application to all servers

No manual Terraform commands needed! Just configure GitHub secrets and push.

Manual Terraform Operations (Optional)

If you need to run Terraform manually (e.g., for troubleshooting):

cd infra/environments/production
terraform init
terraform plan
terraform apply

Note: The CD pipeline uses GitHub secrets (HETZNER_TOKEN, HETZNER_SSH_PUBLIC_KEY) instead of terraform.tfvars. For manual runs, you'll need to set these as environment variables or use -var flags.

Configuration

Terraform Variables

The CD pipeline automatically provisions infrastructure using GitHub secrets. Configuration is hardcoded in environments/production/main.tf.

Required GitHub Secrets (for CD pipeline)

  • HETZNER_TOKEN: Hetzner Cloud API token
  • HETZNER_SSH_PUBLIC_KEY: SSH public key for server access

Configuration (hardcoded in main.tf)

  • project_name: "quantarded"
  • server_type: "cx23"
  • server_image: "ubuntu-22.04"
  • server_location: "nbg1"
  • instance_count: 1

To change configuration, edit environments/production/main.tf directly.

Manual Terraform (if needed)

If running Terraform manually, set environment variables:

export TF_VAR_hetzner_token="your-hetzner-cloud-api-token"
export TF_VAR_ssh_public_key="ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAI..."
export AWS_ACCESS_KEY_ID="your-aws-key"  # If using S3 backend
export AWS_SECRET_ACCESS_KEY="your-aws-secret"  # If using S3 backend

Server Setup

Server setup is fully automated by the CD pipeline. The cloud-init script creates the necessary directory structure and installs Docker, and the deployment pipeline automatically:

  1. Creates docker-compose.yml if it doesn't exist
  2. Updates the Docker image reference with the latest build
  3. Creates .env file from GitHub Actions secrets automatically
  4. Stops and removes old containers to prevent stale resources
  5. Cleans up old Docker images
  6. Pulls the latest image and starts the service

Configuration

All configuration is done via GitHub Actions secrets. The deployment pipeline automatically creates the .env file on each server using the secrets you configure in GitHub Actions (see CD Pipeline Setup).

No manual server setup is required - just configure the GitHub secrets and push to trigger deployment.

Automated Cleanup

Each deployment automatically:

  • Stops and removes old containers (docker-compose down --remove-orphans)
  • Removes old images with the same repository name
  • Cleans up unused Docker resources (docker system prune -f --volumes)

This ensures no stale resources accumulate on the server.

CD Pipeline Setup

The CD pipeline automatically builds Docker images and deploys to Hetzner servers.

Step 1: Configure GitHub Secrets

Go to: Repository → Settings → Secrets and variables → Actions

Add these secrets:

Infrastructure Secrets

HETZNER_TOKEN Your Hetzner Cloud API token

HETZNER_SSH_PRIVATE_KEY Your SSH private key (contents of ~/.ssh/id_ed25519):

cat ~/.ssh/id_ed25519
# Copy the entire output

HETZNER_SSH_PUBLIC_KEY Your SSH public key (contents of ~/.ssh/id_ed25519.pub):

cat ~/.ssh/id_ed25519.pub
# Copy the entire output

AWS_ACCESS_KEY_ID (if using S3 backend) AWS access key ID for S3 backend access

AWS_SECRET_ACCESS_KEY (if using S3 backend) AWS secret access key for S3 backend access

Note:

  • Infrastructure is automatically provisioned/updated on each push
  • Server IPs are automatically discovered from Terraform outputs
  • When you scale up or down, the pipeline automatically detects and applies changes

Application Secrets (Required)

GITHUB_TOKEN (automatically available) GitHub token for pulling private Docker images from GHCR. The default GITHUB_TOKEN is automatically available in GitHub Actions workflows. If your repository/package is private and you encounter authentication issues, you may need to create a Personal Access Token with read:packages permission and add it as a secret.

LLM_API_KEY OpenAI API key for Reddit scraper (required if Reddit scraper is enabled)

TINYBIRD_TOKEN Tinybird API token (required)

QUIVER_API_KEY Quiver Quant API key (required if Quiver scraper is enabled)

Application Configuration

Configuration values are hardcoded in docker-compose.yml and can be modified by editing the deployment workflow (.github/workflows/cd.yml). Current defaults:

  • REDDIT_SCRAPER_ENABLED: true
  • TIME_WINDOW_MINUTES: 15
  • SCRAPER_INTERVAL_MINUTES: 5
  • CLASSIFY_CONCURRENCY: 3
  • QUIVER_SCRAPER_ENABLED: false
  • QUIVER_SCRAPER_CRON: 0 */6 * * *

To change these values, edit the docker-compose.yml section in .github/workflows/cd.yml.

Step 2: Verify GitHub Container Registry

The CD pipeline automatically:

  • Builds Docker images on push to main/master
  • Pushes to GitHub Container Registry (GHCR)
  • Deploys to configured Hetzner servers

Make sure:

  • Package visibility is set correctly: Repository → Packages → quantarded → Package settings
  • The image name in docker-compose.yml matches your GitHub repository

Deployment

Automatic Deployment

The CD pipeline automatically deploys on push to main or master branch:

  1. Push your code:

    git push origin main

  2. Monitor deployment:

    • Go to: Repository → Actions
    • Watch the "CD Pipeline" workflow

Verify Deployment

ssh root@<server-ip>
cd /opt/quantarded

# Check running containers
docker-compose ps

# View logs
docker-compose logs -f

# Check specific container logs
docker logs quantarded-scraper -f

Scaling

Add More Servers

  1. Update Terraform configuration:
    • Edit environments/production/main.tf: Change instance_count = 2
    • Push to main branch
    • CD pipeline will automatically provision the new server and deploy to it

Change Server Configuration

To change server type, image, or location:

  1. Edit environments/production/main.tf:

    server_type = "cx31"  # Upgrade to larger instance
server_location = "fsn1"  # Change location

  2. Push to main branch: CD pipeline will apply changes automatically

Note: Changing server type or location will recreate the server. Make sure to backup data first.

Updating Configuration

Update Application Configuration

Application configuration is hardcoded in infra/scripts/deploy-to-server.sh. To change:

  1. Edit the script file
  2. Push to main branch
  3. CD pipeline will automatically deploy with new configuration

Update Infrastructure Configuration

Infrastructure configuration is in environments/production/main.tf. To change:

  1. Edit the file (e.g., change instance_count, server_type, etc.)
  2. Push to main branch
  3. CD pipeline will automatically apply changes

Monitoring

Check application health:

# View logs
docker logs quantarded-scraper -f

# Check container status
docker ps

# View resource usage
docker stats

# Check docker-compose status
docker-compose ps

Rollback

To rollback to a previous version:

ssh root@<server-ip>
cd /opt/quantarded
export IMAGE_TAG=<previous-sha-or-tag>
docker-compose pull
docker-compose up -d

To rollback via CD pipeline, push a commit that references the previous image tag, or manually trigger the workflow with a specific commit SHA.

Troubleshooting

Image Pull Fails

If you get authentication errors when pulling from GHCR:

  1. Login to GHCR on the server:

    echo $GITHUB_TOKEN | docker login ghcr.io -u USERNAME --password-stdin

    Or create a GitHub Personal Access Token with read:packages permission.

  2. Check package visibility: Repository → Packages → quantarded → Package settings

Deployment Fails

  1. Check GitHub Actions logs: Repository → Actions → CD Pipeline
  2. Verify SSH key: Test connection manually: ssh root@<server-ip>
  3. Check server setup: Ensure docker-compose.yml and .env exist

Container Not Starting

  1. Check logs:

    docker logs quantarded-scraper

  2. Verify .env file: Ensure all required variables are set

  3. Check docker-compose.yml syntax:

    docker-compose config

  4. Check disk space:

    df -h
docker system df

SSH Connection Issues

  1. Check SSH key:

    ssh-add -l  # Should show your key

  2. Test connection:

    ssh root@<server-ip>

  3. Verify key in GitHub secrets: Check that HETZNER_SSH_PUBLIC_KEY matches your public key

Terraform Issues

  1. Invalid credentials: Verify HETZNER_TOKEN in GitHub secrets
  2. Resource conflicts: Run terraform plan to see what will change
  3. State lock: If terraform is stuck, check for .terraform.tfstate.lock.info and remove if safe
  4. Module not found: Make sure you're running terraform from the environment directory (environments/production)

Destroying Infrastructure

To destroy all resources:

cd infra/environments/production
terraform destroy

Warning: This will delete all servers and data. Make sure you have backups!

Module Documentation

See modules/server/README.md for detailed module documentation.

Server Module

The modules/server module creates Hetzner Cloud servers with firewall rules and SSH key management.

Module Usage

module "servers" {
  source = "../../modules/server"

  project_name    = "quantarded"
  server_type     = "cx23"
  server_image    = "ubuntu-22.04"
  server_location = "nbg1"
  instance_count  = 1
  ssh_public_key  = var.ssh_public_key
}

Module Inputs

Name Description Type Required
project_name Project name (used for resource naming) string yes
server_type Hetzner server type string yes
server_image Server image (OS) string yes
server_location Server location string yes
instance_count Number of server instances to create number yes
ssh_public_key SSH public key for server access string yes

Module Outputs

Name Description
server_ips Public IP addresses of the servers
server_names Names of the servers
ssh_command SSH command to connect to the first server
deployment_info Deployment information with server details
firewall_id Firewall ID
ssh_key_id SSH key ID

Module Resources

  • hcloud_ssh_key: SSH key for server access
  • hcloud_firewall: Firewall with rules for SSH (22), HTTP (80), HTTPS (443)
  • hcloud_server: Server instances with cloud-init configuration

See Also