Kirby - Digital Ocean Deployment Guide

This guide will walk you through deploying Kirby to Digital Ocean using Docker Compose.

Prerequisites
1. Create Digital Ocean Droplet
2. Initial Server Setup
3. Install Docker
4. Deploy Kirby
5. Configure and Start Services
6. Verify Deployment
7. Monitoring and Maintenance
8. Security Hardening
9. Troubleshooting

Prerequisites

Digital Ocean account
SSH key pair (or ability to create one)
Domain name (optional, for production use)
Basic command line knowledge

1. Create Digital Ocean Droplet

Step 1.1: Create Droplet

Log into Digital Ocean: https://cloud.digitalocean.com
Click Create → Droplets
Choose configuration:
- Image: Ubuntu 24.04 (LTS) x64
- Droplet Size:
  - Minimum: Basic - $12/mo (2 GB RAM, 1 vCPU, 50 GB SSD)
  - Recommended: Basic - $18/mo (2 GB RAM, 2 vCPU, 60 GB SSD)
  - Production: Basic - $24/mo (4 GB RAM, 2 vCPU, 80 GB SSD)
- Datacenter: Choose closest to your target market (e.g., New York for US markets)
- Authentication: SSH keys (recommended) or Password
- Hostname: kirby-collector (or your choice)
- Tags: kirby, production, crypto
Click Create Droplet

Step 1.2: Note Your Droplet IP

Once created, note your droplet's public IP address:

Droplet IP: xxx.xxx.xxx.xxx

2. Initial Server Setup

Step 2.1: Connect via SSH

ssh root@YOUR_DROPLET_IP

Step 2.2: Update System

apt update && apt upgrade -y

Step 2.3: Set Timezone (Optional)

timedatectl set-timezone America/New_York  # Or your preferred timezone
timedatectl

Step 2.4: Create Non-Root User (Recommended)

# Create user
adduser kirby

# Add to sudo group
usermod -aG sudo kirby

# Copy SSH keys (if using SSH authentication)
rsync --archive --chown=kirby:kirby ~/.ssh /home/kirby

Step 2.5: Switch to New User

su - kirby

Or reconnect:

ssh kirby@YOUR_DROPLET_IP

3. Install Docker

Step 3.1: Install Docker

# Install prerequisites
sudo apt install -y apt-transport-https ca-certificates curl software-properties-common

# Add Docker's official GPG key
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

# Add Docker repository
echo "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

# Update package index
sudo apt update

# Install Docker
sudo apt install -y docker-ce docker-ce-cli containerd.io

Step 3.2: Install Docker Compose

# Install Docker Compose plugin
sudo apt install -y docker-compose-plugin

# Verify installation
docker compose version

Step 3.3: Add User to Docker Group

sudo usermod -aG docker $USER

# Log out and back in for group changes to take effect
exit
# SSH back in
ssh kirby@YOUR_DROPLET_IP

Step 3.4: Verify Docker Installation

docker run hello-world

You should see a success message.

4. Deploy Kirby

Step 4.1: Install Git

sudo apt install -y git

Step 4.2: Clone Repository

cd ~
git clone https://github.com/oakwoodgates/kirby.git
cd kirby

Note: Replace with your actual repository URL

Step 4.3: Set Up Directory Structure

# Create logs directory
mkdir -p logs

# Set permissions
chmod 755 logs

5. Configure and Start Services

Automated Deployment (Recommended)

The easiest way to deploy Kirby is using the automated deployment script. This sets up both production and training databases automatically.

Step 5.1: Make Deploy Script Executable

chmod +x deploy.sh

Step 5.2: Run Automated Deployment

./deploy.sh

The script will automatically:

✅ Create .env file with generated password (if doesn't exist)
✅ Check Docker and Docker Compose are installed
✅ Create necessary directories
✅ Build Docker images
✅ Start all services
✅ Create production database (kirby) with 8 starlistings
✅ Create training database (kirby_training) with 24 starlistings
✅ Run migrations on both databases
✅ Sync configurations for both databases
✅ Verify everything is working correctly

Expected Output:

========================================
  Kirby Deployment Script
========================================

[✓] .env file created with generated password
[!] IMPORTANT: Your database password is: <generated_password>
[!] Save this password securely!

[✓] Docker is installed
[✓] Docker Compose is installed
[✓] Directories created
[✓] Docker images built
[✓] Services started
[✓] Database is ready

========================================
  Setting up Production Database
========================================
[✓] Production migrations completed
[✓] Production configuration synced
[✓] Production starlistings: 8

========================================
  Setting up Training Database
========================================
[✓] Training database created
[✓] TimescaleDB extension enabled
[✓] Training migrations completed
[✓] Training configuration synced
[✓] Training starlistings: 24

========================================
  Final Verification
========================================
Running comprehensive verification...
[✓] Production database verified (8 starlistings)
[✓] Training database verified (24 starlistings)

========================================
  Deployment Complete!
========================================

Step 5.3: Save Your Database Password

If this is a fresh deployment, the script generated a random password. Save it securely!

You can find it in the .env file:

cat .env | grep POSTGRES_PASSWORD

Step 5.4: View Logs

Check that everything is working:

docker compose logs -f collector

You should see:

✅ "Connected to Hyperliquid WebSocket"
✅ "Subscribed to candles" (8 times for production trading pairs)
✅ "Collector connected and running"

Press Ctrl+C to exit logs

Manual Deployment (Advanced)

If you prefer manual control over each step, follow these instructions instead of using ./deploy.sh:

Click to expand manual deployment steps

Manual Step 1: Create Environment File

cp .env.example .env
nano .env  # or use vim

Edit the following values:

# REQUIRED: Change this password!
POSTGRES_PASSWORD=YOUR_SECURE_PASSWORD_HERE

# Update DATABASE_URL with your password
DATABASE_URL=postgresql+asyncpg://kirby:YOUR_SECURE_PASSWORD_HERE@timescaledb:5432/kirby

# Update TRAINING_DATABASE_URL with your password
TRAINING_DATABASE_URL=postgresql+asyncpg://kirby:YOUR_SECURE_PASSWORD_HERE@timescaledb:5432/kirby_training

# Set to production
ENVIRONMENT=production

# Optional: Adjust log level
LOG_LEVEL=info

# Optional: Adjust collector settings
COLLECTOR_MAX_RETRIES=5
COLLECTOR_RESTART_DELAY=30

Generate a secure password:

openssl rand -base64 32

Save and exit (Ctrl+X, then Y, then Enter in nano).

Manual Step 2: Review Configuration

Check your starlisting configuration:

cat config/starlistings.yaml      # Production config (8 starlistings)
cat config/training_stars.yaml    # Training config (24 starlistings)

Manual Step 3: Build Docker Images

docker compose build

This will take a few minutes on first build.

Manual Step 4: Start Services

docker compose up -d

Manual Step 5: Wait for Database

# Wait 10 seconds for database to be ready
sleep 10

Manual Step 6: Setup Production Database

# Run migrations
docker compose exec collector alembic upgrade head

# Sync configuration
docker compose exec collector python -m scripts.sync_config

# Verify
docker compose exec timescaledb psql -U kirby -d kirby -c "SELECT COUNT(*) FROM starlistings;"
# Expected: 8

Manual Step 7: Setup Training Database

# Create database
docker compose exec timescaledb psql -U kirby -c "CREATE DATABASE kirby_training;"

# Enable TimescaleDB extension
docker compose exec timescaledb psql -U kirby -d kirby_training -c "CREATE EXTENSION IF NOT EXISTS timescaledb;"

# Run migrations
docker compose exec collector python -m scripts.migrate_training_db

# Sync configuration
docker compose exec collector python -m scripts.sync_training_config

# Verify
docker compose exec timescaledb psql -U kirby -d kirby_training -c "SELECT COUNT(*) FROM starlistings;"
# Expected: 24

Manual Step 8: Restart Services

docker compose restart collector api

Manual Step 9: View Logs

docker compose logs -f collector

6. Verify Deployment

Step 6.1: Check API Health

curl http://localhost:8000/health

Expected output:

{
  "status": "healthy",
  "timestamp": "2025-10-26T...",
  "database": "connected",
  "version": "0.1.0"
}

Step 6.2: Check Starlistings

curl http://localhost:8000/starlistings | jq

You should see your configured trading pairs.

Step 6.3: Wait for Data Collection

Wait 1-2 minutes for the collector to gather some candles, then check:

curl "http://localhost:8000/candles/hyperliquid/BTC/USD/perps/1m?limit=5" | jq

Step 6.4: Check Database Directly

docker compose exec timescaledb psql -U kirby -d kirby

# Inside psql:
SELECT COUNT(*) FROM candles;
SELECT * FROM candles LIMIT 5;

# Exit psql
\q

Step 6.5: Verify Collector is Running

# Check collector logs
docker compose logs --tail=50 collector

# Look for messages like:
# "Connected to Hyperliquid WebSocket"
# "Subscribed to candles"

Step 6.6: Backfill Historical Data (Optional)

After your collector is running and collecting real-time data, you may want to backfill historical candle data. This is useful for:

Building historical datasets for analysis
Filling gaps if the collector was offline
Getting data from before your deployment

Important Notes:

⚠️ All commands below use docker compose exec collector - this runs the script inside the Docker container
The backfill script uses CCXT which maps Hyperliquid's USD quotes to USDC internally (this is how CCXT represents Hyperliquid markets)
Your data will be stored as USD in the database - the USDC mapping is automatic

Backfill 1 Day of Data (Quick Test)

# Backfill 1 day of BTC data across all intervals
docker compose exec collector python -m scripts.backfill --exchange=hyperliquid --coin=BTC --days=1

# Expected output:
# - 1m: ~1,440 candles
# - 15m: ~96 candles
# - 4h: ~6 candles
# - 1d: ~1 candle

Backfill 30 Days for a Specific Coin

# Backfill 30 days of SOL data
docker compose exec collector python -m scripts.backfill --exchange=hyperliquid --coin=SOL --days=30

Backfill All Active Starlistings

# Backfill 90 days for all configured trading pairs
docker compose exec collector python -m scripts.backfill --all --days=90

# For 1 year of data (this will take longer)
docker compose exec collector python -m scripts.backfill --all --days=365

Monitor Backfill Progress

# Watch the backfill logs in real-time
docker compose logs -f collector

# Check how many candles were backfilled
docker compose exec timescaledb psql -U kirby -d kirby -c "SELECT COUNT(*) FROM candles;"

# Check candles by interval
docker compose exec timescaledb psql -U kirby -d kirby -c "
SELECT i.name as interval, COUNT(*) as candle_count
FROM candles c
JOIN starlistings s ON c.starlisting_id = s.id
JOIN intervals i ON s.interval_id = i.id
GROUP BY i.name
ORDER BY i.name;
"

Backfill Best Practices

Start Small: Test with 1 day first to verify everything works

docker compose exec collector python -m scripts.backfill --coin=BTC --days=1

Rate Limiting: The backfill script automatically handles rate limits, but large backfills will take time

1 day: ~1-2 minutes per coin
30 days: ~5-10 minutes per coin
365 days: ~30-60 minutes per coin

Run During Low Traffic: For large backfills (>30 days), run during off-peak hours to avoid impacting real-time collection

Check for Duplicates: The backfill uses UPSERT logic, so running it multiple times won't create duplicates - it will update existing candles

Verify Backfill Results

# Check date range of backfilled data
docker compose exec timescaledb psql -U kirby -d kirby -c "
SELECT
  MIN(time) as earliest_candle,
  MAX(time) as latest_candle,
  COUNT(*) as total_candles
FROM candles;
"

# Check backfill completeness for BTC 1m
docker compose exec timescaledb psql -U kirby -d kirby -c "
SELECT
  c.symbol as coin,
  i.name as interval,
  COUNT(*) as candle_count,
  MIN(ca.time) as earliest,
  MAX(ca.time) as latest
FROM candles ca
JOIN starlistings s ON ca.starlisting_id = s.id
JOIN coins c ON s.coin_id = c.id
JOIN intervals i ON s.interval_id = i.id
WHERE c.symbol = 'BTC' AND i.name = '1m'
GROUP BY c.symbol, i.name;
"

Step 6.7: Backfill Funding Rates (Optional)

After your collector is running and collecting real-time funding/OI data, you may want to backfill historical funding rate data. This provides historical context for funding rate trends.

Important Notes:

⚠️ All commands use docker compose exec collector - runs inside the Docker container
Funding rates update hourly on Hyperliquid (24 records/day)
API Limitation: Historical endpoint ONLY provides funding_rate and premium
- ❌ NO historical data for: mark_price, oracle_price, mid_price, open_interest, next_funding_time
- ✅ Real-time collector captures ALL fields going forward
- Safe to run - won't overwrite real-time data (uses COALESCE)
Uses minute-precision timestamps aligned with candle data
Safe to run multiple times (UPSERT with COALESCE preserves existing data)

Backfill 7 Days (Recommended Starting Point)

# Backfill 7 days for BTC
docker compose exec collector python -m scripts.backfill_funding --coin=BTC --days=7

# Expected output:
# - Funding rates: ~168 records (24 per day × 7 days)
# - Only funding_rate and premium populated (API limitation)
# - Other fields (mark_price, OI, etc.) remain NULL or preserve existing real-time data

Backfill Specific Coins

# Backfill 30 days for SOL
docker compose exec collector python -m scripts.backfill_funding --coin=SOL --days=30

# Backfill 90 days for both BTC and SOL (run separately)
docker compose exec collector python -m scripts.backfill_funding --coin=BTC --days=90
docker compose exec collector python -m scripts.backfill_funding --coin=SOL --days=90

Backfill All Configured Coins

# Backfill 30 days for all coins in starlistings.yaml
docker compose exec collector python -m scripts.backfill_funding --all --days=30

# For maximum historical data (365 days)
docker compose exec collector python -m scripts.backfill_funding --all --days=365

Monitor Backfill Progress

# Watch backfill logs
docker compose logs -f collector

# Check funding rates count
docker compose exec timescaledb psql -U kirby -d kirby -c "
SELECT COUNT(*) as total_funding_records FROM funding_rates;
"

# Check open interest count
docker compose exec timescaledb psql -U kirby -d kirby -c "
SELECT COUNT(*) as total_oi_records FROM open_interest;
"

# View recent funding/OI data by coin
docker compose exec timescaledb psql -U kirby -d kirby -c "
SELECT
  co.symbol as coin,
  COUNT(DISTINCT f.time) as funding_records,
  COUNT(DISTINCT o.time) as oi_records,
  MIN(f.time) as earliest,
  MAX(f.time) as latest
FROM funding_rates f
JOIN starlistings s ON f.starlisting_id = s.id
JOIN coins co ON s.coin_id = co.id
LEFT JOIN open_interest o ON f.time = o.time AND f.starlisting_id = o.starlisting_id
GROUP BY co.symbol
ORDER BY co.symbol;
"

Verify Backfill Data Quality

# Check for gaps in funding data (should update hourly)
docker compose exec timescaledb psql -U kirby -d kirby -c "
SELECT
  time,
  funding_rate,
  mark_price,
  open_interest
FROM funding_rates f
JOIN starlistings s ON f.starlisting_id = s.id
JOIN coins c ON s.coin_id = c.id
WHERE c.symbol = 'BTC'
ORDER BY time DESC
LIMIT 10;
"

# Verify timestamp alignment with candles
docker compose exec timescaledb psql -U kirby -d kirby -c "
SELECT
  'candles' as source,
  time,
  close as value
FROM candles
WHERE starlisting_id = 1
ORDER BY time DESC
LIMIT 3
UNION ALL
SELECT
  'funding' as source,
  time,
  funding_rate as value
FROM funding_rates
WHERE starlisting_id = 1
ORDER BY time DESC
LIMIT 3;
"

Backfill Best Practices

1. Start Small: Always test with 7 days first

docker compose exec collector python -m scripts.backfill_funding --coin=BTC --days=7

2. Data Retention: Consider your storage needs

7 days: ~168 records per coin (24 per day)
30 days: ~720 records per coin
365 days: ~8,760 records per coin
Note: Backfill only populates funding_rate and premium (API limitation)

3. Rate Limiting: The backfill script respects Hyperliquid API limits

7 days: <1 minute per coin
30 days: 1-2 minutes per coin
365 days: 5-10 minutes per coin

4. Concurrent Backfills: Safe to backfill candles and funding simultaneously

# Terminal 1: Backfill candles
docker compose exec collector python -m scripts.backfill --coin=BTC --days=30

# Terminal 2: Backfill funding (simultaneously)
docker compose exec collector python -m scripts.backfill_funding --coin=BTC --days=30

5. Data Consistency: Funding backfills use minute-precision timestamps

Aligns with candle data: 2025-11-02 20:00:00+00
Easy to JOIN: ON f.time = c.time
Consistent with 1-minute buffering strategy
Real-time collector provides complete data (all price fields + OI)

Common Issues

Issue: "No funding history found"

# Verify coin symbol is correct (case-sensitive)
docker compose exec collector python -m scripts.backfill_funding --coin=BTC --days=1

# Check if coin exists in starlistings
docker compose exec timescaledb psql -U kirby -d kirby -c "SELECT * FROM coins;"

Issue: Backfill is slow

# This is normal - funding history API has rate limits
# For 365 days, expect 5-10 minutes per coin
# Monitor progress with: docker compose logs -f collector

Issue: Duplicate prevention

# Safe to re-run - uses UPSERT (ON CONFLICT DO UPDATE)
# Will update existing records, not create duplicates
docker compose exec collector python -m scripts.backfill_funding --coin=BTC --days=7

7. Monitoring and Maintenance

Step 7.1: Set Up Automatic Restarts

Services are already configured with restart: unless-stopped in docker-compose.yml.

To verify:

docker compose ps

Step 7.2: Monitor Resource Usage

# Docker stats
docker stats

# System resources
htop  # Install: sudo apt install htop

Step 7.3: Set Up Log Rotation

Create log rotation config:

sudo nano /etc/logrotate.d/docker-containers

Add:

/var/lib/docker/containers/*/*.log {
  rotate 7
  daily
  compress
  missingok
  delaycompress
  copytruncate
}

Step 7.4: Database Backups

Create backup script:

nano ~/backup-kirby.sh

Add:

#!/bin/bash
BACKUP_DIR="/home/kirby/backups"
DATE=$(date +%Y%m%d_%H%M%S)
mkdir -p $BACKUP_DIR

# Backup database
docker compose exec -T timescaledb pg_dump -U kirby kirby | gzip > "$BACKUP_DIR/kirby_$DATE.sql.gz"

# Keep only last 7 days
find $BACKUP_DIR -name "kirby_*.sql.gz" -mtime +7 -delete

echo "Backup completed: kirby_$DATE.sql.gz"

Make executable and test:

chmod +x ~/backup-kirby.sh
./backup-kirby.sh

Set up daily cron job:

crontab -e

# Add this line (runs daily at 2 AM):
0 2 * * * /home/kirby/backup-kirby.sh >> /home/kirby/logs/backup.log 2>&1

Step 7.5: Monitor Disk Space

# Check disk usage
df -h

# Check Docker disk usage
docker system df

# Clean up old images (careful!)
docker image prune -a

Step 7.6: Update or Deploy Application

The automated deployment script (deploy.sh) works for both fresh deployments AND updates. It's idempotent, meaning it's safe to run multiple times.

Automated Update (Recommended)

cd ~/kirby

# Pull latest changes
git pull origin main

# Run automated deployment
chmod +x deploy.sh
./deploy.sh

The script will:

✅ Detect existing .env file (won't overwrite)
✅ Rebuild Docker images
✅ Start all services
✅ Run migrations on both databases (production + training)
✅ Sync configurations
✅ Verify everything is working

Expected Output:

[✓] .env file already exists
[✓] Docker is installed
[✓] Docker Compose is installed
[✓] Docker images built
[✓] Services started
[✓] Database is ready
[✓] Production migrations completed
[✓] Production starlistings: 8
[✓] Training database already exists
[✓] Training migrations completed
[✓] Training starlistings: 24
[✓] Deployment Complete!

Verify Update

# Check services are running
docker compose ps

# Check API health
curl http://localhost:8000/health

# Check production database
docker compose exec timescaledb psql -U kirby -d kirby -c "SELECT COUNT(*) FROM starlistings;"
# Expected: 8

# Check training database
docker compose exec timescaledb psql -U kirby -d kirby_training -c "SELECT COUNT(*) FROM starlistings;"
# Expected: 24

# Check collector logs
docker compose logs -f collector

Quick Update (Code Changes Only)

If you only changed application code (no database/config changes):

cd ~/kirby
git pull origin main
docker compose down
docker compose up -d --build
docker compose logs -f

Rollback on Failure

If an update fails:

# Rollback to previous commit
cd ~/kirby
git log --oneline -5  # Find the commit hash
git checkout <previous-commit-hash>

# Redeploy
./deploy.sh

# Or manually rebuild
docker compose down
docker compose up -d --build

Manual Update (Advanced)

Click to expand manual update steps

cd ~/kirby

# Pull latest changes
git pull origin main

# Stop services
docker compose stop

# Rebuild images
docker compose build --no-cache

# Start database only
docker compose up -d timescaledb
sleep 10

# Run migrations on both databases
docker compose exec collector alembic upgrade head
docker compose exec collector python -m scripts.migrate_training_db

# Sync configurations
docker compose exec collector python -m scripts.sync_config
docker compose exec collector python -m scripts.sync_training_config

# Start all services
docker compose up -d

# Verify
docker compose ps
docker compose logs -f collector

Important Notes:

Use ./deploy.sh for all updates - it's safe and idempotent
Check logs after update to verify services started correctly
If collector or API shows errors, check database migrations ran successfully
Training database is now included in all deployments automatically

Step 7.7: Fast Updates and Downtime Recovery (NEW!)

For production environments, full deployments via deploy.sh can be slow (5-10 minutes). Use these optimized scripts for faster updates and automatic data recovery.

update.sh - Fast Updates (30-60 seconds)

The update.sh script provides intelligent, fast updates by only rebuilding/migrating what changed.

When to Use:

✅ Code changes (Python files, dependencies)
✅ Configuration updates (YAML files)
✅ Database schema changes (new migrations)
✅ Quick restarts after fixes

When NOT to Use:

❌ Initial deployment (use deploy.sh instead)
❌ Major infrastructure changes (use deploy.sh instead)

Speed Comparison:

Scenario	deploy.sh	update.sh	Improvement
Code-only update	5-10 min	30-60 sec	90% faster
With new migrations	5-10 min	1-2 min	70% faster
Simple restart	5-10 min	10-20 sec	95% faster

Basic Usage:

cd ~/kirby

# Make executable (first time only)
chmod +x update.sh

# Auto-detect changes and update (prompts for backfill if gaps detected)
./update.sh

# Automatically backfill detected gaps without prompting
./update.sh --auto-backfill

# Skip downtime detection and backfill entirely
./update.sh --skip-backfill

# Force rebuild even if no code changes detected
./update.sh --force-build

# Force migration check
./update.sh --force-migrate

# Skip build (only config/docs changed)
./update.sh --skip-build

Example Output:

╔════════════════════════════════════════════════════════════╗
║         Kirby Fast Update Script                          ║
╚════════════════════════════════════════════════════════════╝

✓ Docker is running
✓ Services are running

[1/6] Checking for changes...
✓ Git pull successful
⚠ Code changes detected

[2/6] Building Docker images (code changed)...
✓ Docker images built (45s)

[3/6] Checking for database migrations...
✓ Database is up to date (revision: abc123)

[4/6] Restarting services...
✓ Services restarted (8s)

[5/6] Verifying services...
✓ Database is healthy
✓ API is healthy
✓ Collector is running

[6/7] Update summary...

╔════════════════════════════════════════════════════════════╗
║             Update Completed Successfully                 ║
╚════════════════════════════════════════════════════════════╝

  Total time: 58s

  Changes applied:
    ✓ Docker images rebuilt
    ○ No new migrations
    ✓ Services restarted

Next Steps:
  • Check logs: docker compose logs -f collector api
  • Check health: curl http://localhost:8000/health

[7/7] Checking for data gaps...

⚠ Data gap detected: 0.3 hours (18 minutes)

  Data recovery available:
    ✓ Candles (OHLCV) - 100% recoverable
    ⚠ Funding rates - 40% recoverable (rate + premium only)
    ✗ Open interest - NOT recoverable (permanently lost)

Run backfill now? [y/N]: y

[Running backfill_downtime.sh...]
✓ Data backfill completed

What the Script Does:

Smart Change Detection: Uses git diff to detect what changed
Conditional Building: Only rebuilds Docker images if code changed
Migration Check: Only runs migrations if new ones exist
Quick Restart: Uses docker compose restart (not down/up)
Health Verification: Confirms services started correctly
Downtime Detection: Automatically checks for data gaps after restart
Optional Auto-Backfill: Prompts to backfill or auto-backfills with --auto-backfill
Minimal Downtime: Typically 10-20 seconds of collector downtime

backfill_downtime.sh - Automatic Data Recovery

When the collector is down (during updates or outages), some data is missed. This script automatically detects gaps and recovers what's possible.

Data Recovery Levels:

Data Type	Recovery	Tool Used	Time Range
Candles (OHLCV)	✅ 100%	CCXT API	~2 years back
Funding Rates	⚠️ 40%	Hyperliquid API	~1 year back
Funding Prices	❌ LOST	N/A	Not available
Open Interest	❌ LOST	N/A	Not available

Important: Open Interest (OI) data has NO historical API and is permanently lost during downtime. This makes minimizing collector downtime critical!

Basic Usage:

cd ~/kirby

# Make executable (first time only)
chmod +x backfill_downtime.sh

# Auto-detect gaps and backfill
./backfill_downtime.sh

# Dry run (show what would be backfilled)
./backfill_downtime.sh --dry-run

# Manual: backfill last 7 days
./backfill_downtime.sh --days 7

# Manual: backfill specific coin only
./backfill_downtime.sh --coin BTC --days 1

Example Output:

╔════════════════════════════════════════════════════════════╗
║         Kirby Downtime Backfill Script                    ║
╚════════════════════════════════════════════════════════════╝

✓ Docker is running
✓ Services are running

[1/4] Detecting data gaps...

  Current time: 2025-11-18 15:30:00 UTC
  Active starlistings: 8
  Maximum gap: 2.5 hours (150 minutes)

Data Gaps by Starlisting:
  BTC/USD 1m:
    Candles: 2.50h, Funding: 2.50h, OI: 2.50h
  BTC/USD 15m:
    Candles: 2.50h, Funding: 2.50h, OI: 2.50h

Auto-detected gap: 2.5 hours
Will backfill last 1 days to ensure complete recovery

[2/4] Backfilling candle data (100% recoverable)...
  Backfilling all active starlistings...
  Processing BTC/USD perps 1m...
  Backfilled 150 candles
  Processing BTC/USD perps 15m...
  Backfilled 10 candles
✓ Candles backfilled (45s)

[3/4] Backfilling funding rates (partially recoverable)...
⚠ Note: Historical funding API only provides:
    ✓ funding_rate
    ✓ premium
    ✗ mark_price (LOST)
    ✗ index_price (LOST)
    ✗ oracle_price (LOST)
    ✗ mid_price (LOST)
    ✗ next_funding_time (LOST)

  Backfilling all active coins...
✓ Funding rates backfilled (12s)

[4/4] Generating data loss report...

╔════════════════════════════════════════════════════════════╗
║              Data Recovery Summary                        ║
╚════════════════════════════════════════════════════════════╝

✓ FULLY RECOVERED:
  • Candles (OHLCV data for all intervals)
    - 100% recovery via CCXT API
    - All intervals: 1m, 15m, 4h, 1d

⚠ PARTIALLY RECOVERED:
  • Funding Rates (basic fields only)
    - ✓ funding_rate
    - ✓ premium
    - ✗ mark_price (NOT available historically)
    - ✗ index_price (NOT available historically)
    - Recovery: ~40% (2 of 7 fields)

✗ PERMANENTLY LOST:
  • Open Interest (no historical API available)
    - ✗ open_interest
    - ✗ notional_value
    - ✗ day_base_volume
    - ✗ day_notional_volume
    - Recovery: 0% (no historical data)

Affected Time Range:
  Start: 2025-11-18 13:00:00 UTC
  End:   2025-11-18 15:30:00 UTC
  Duration: 2.5 hours (1 days backfilled)

╔════════════════════════════════════════════════════════════╗
║          Backfill Completed Successfully                  ║
╚════════════════════════════════════════════════════════════╝

  Total time: 62s

What the Script Does:

Detect Gaps: Queries database for last timestamp per table
Calculate Downtime: Compares to current time to find gaps
Backfill Candles: 100% recovery via CCXT (all intervals)
Backfill Funding: Partial recovery via Hyperliquid API (rate + premium only)
Report Losses: Shows what data was lost permanently (OI, funding prices)
Verify Results: Confirms recovered data counts

Best Practices for Production Updates

Recommended Update Workflow:

Option 1: Interactive (Default) - Prompts for backfill

cd ~/kirby
git pull origin main
./update.sh
# Script will detect gaps and prompt: "Run backfill now? [y/N]"
# Answer 'y' to backfill or 'N' to skip

Option 2: Fully Automatic - No prompts

cd ~/kirby
git pull origin main
./update.sh --auto-backfill
# Automatically detects and backfills gaps without prompting

Option 3: Manual Control - Skip auto-detection

# 1. Pull latest code
cd ~/kirby
git pull origin main

# 2. Fast update (skip automatic backfill detection)
./update.sh --skip-backfill

# 3. Monitor logs for 1-2 minutes
docker compose logs -f collector

# 4. Manually backfill if needed
./backfill_downtime.sh

# 5. Verify everything is healthy
curl http://localhost:8000/health
docker compose ps

Minimize Data Loss:

Use update.sh instead of deploy.sh for routine updates (faster = less downtime)
Run backfill_downtime.sh immediately after any restart
Monitor collector uptime - OI data is NOT recoverable
Schedule updates during low-volatility periods if possible
Consider alerts for collector downtime (future enhancement)

When to Use Each Script:

Scenario	Script to Use	Downtime	Notes
Initial deployment	`deploy.sh`	30-60s	First time setup
Code updates	`update.sh`	10-20s	Fastest option
Config changes	`update.sh --skip-build`	10s	No rebuild needed
Schema changes	`update.sh`	1-2min	Runs migrations
Major changes	`deploy.sh`	30-60s	Full verification
After downtime	`backfill_downtime.sh`	0s	Runs alongside live collector

Troubleshooting:

Issue: update.sh says "Services are not running"

# Solution: Use deploy.sh for initial deployment
./deploy.sh

Issue: Backfill reports no gaps but data is missing

# Check if collector is actually running
docker compose logs collector | tail -50

# Manually run backfill for last 24 hours
./backfill_downtime.sh --days 1

Issue: Update failed mid-way

# Rollback to previous commit
git log --oneline -5
git checkout <previous-commit-hash>
./deploy.sh  # Full redeploy

# Then return to latest
git checkout main
./update.sh

8. NordVPN Setup for Training Data Backfills (Optional)

If you need to backfill training data from geo-restricted exchanges like Binance, you can use the built-in NordVPN integration.

Note: The deploy.sh script automatically detects and configures the TimescaleDB IP for VPN networking. You don't need to configure this manually.

Step 8.1: Add NordVPN Token to .env

# Get your NordVPN token from: https://my.nordaccount.com
# Navigate to: Services → NordVPN → Access Token → Generate new token

nano .env

# Add these lines:
NORDVPN_TOKEN=your_actual_token_here
NORDVPN_COUNTRY=Chile
NORDVPN_TECHNOLOGY=NordLynx

Save and exit (Ctrl+X, Y, Enter)

Step 8.2: Start VPN (Only When Needed)

The VPN does NOT auto-start. You start it manually when backfilling, then stop it to save resources.

# Start VPN
docker compose --profile vpn up -d vpn

# Wait for connection (30 seconds)
sleep 30

# Verify connection
docker compose logs vpn | tail -20
# Look for: "You are connected to Chile"

# Test Binance access
docker compose exec vpn curl -s https://api.binance.com/api/v3/ping
# Should return: {}

Step 8.3: Run Training Backfills

# Backfill BTC from Binance (through Chile VPN)
docker compose --profile vpn --profile tools run --rm collector-training python -m scripts.backfill_training --coin=BTC --days=30

# Backfill other coins
docker compose --profile vpn --profile tools run --rm collector-training python -m scripts.backfill_training --coin=ETH --days=30
docker compose --profile vpn --profile tools run --rm collector-training python -m scripts.backfill_training --coin=SOL --days=30

# Or backfill all configured coins
docker compose --profile vpn --profile tools run --rm collector-training python -m scripts.backfill_training --days=30

Note: Both --profile vpn and --profile tools are required because the VPN and collector-training services use different profiles.

Step 8.4: Verify IP Routing (Optional)

To confirm that VPN routing is working correctly:

# Check production collector's IP (should show your US IP)
docker compose exec collector curl -s https://ipinfo.io/json | grep -E '"ip"|"country"'

# Check VPN container's IP (should show Chile)
docker compose exec vpn curl -s https://ipinfo.io/json | grep -E '"ip"|"country"'

# Check collector-training's IP (should show Chile - routes through VPN)
docker compose --profile vpn --profile tools run --rm collector-training curl -s https://ipinfo.io/json | grep -E '"ip"|"country"'

Expected Results:

Production collector: US IP
VPN container: Chile IP
Collector-training: Chile IP (same as VPN)

Step 8.5: Stop VPN When Done

# Stop VPN to save resources
docker compose stop vpn

Step 8.6: Verify Training Data

# Check training database
docker compose exec timescaledb psql -U kirby -d kirby_training -c "SELECT COUNT(*) FROM candles;"

# Check data by exchange
docker compose exec timescaledb psql -U kirby -d kirby_training -c "
SELECT e.name, COUNT(*) as candle_count
FROM candles c
JOIN starlistings s ON c.starlisting_id = s.id
JOIN exchanges e ON s.exchange_id = e.id
GROUP BY e.name;"

Important Notes:

VPN only affects collector-training service, not production
Production data collection (Hyperliquid) continues normally on US IP
Stop VPN when not backfilling to save CPU/memory
For complete VPN setup guide, see docs/NORDVPN_SETUP.md

9. Security Hardening

Step 9.1: Configure Firewall (UFW)

# Enable UFW
sudo ufw enable

# Allow SSH (IMPORTANT: Do this first!)
sudo ufw allow 22/tcp

# Allow API (if exposing to public)
sudo ufw allow 8000/tcp

# Deny PostgreSQL from external access (keep it internal)
sudo ufw deny 5432/tcp

# Check status
sudo ufw status

Step 8.2: Change SSH Port (Optional but Recommended)

sudo nano /etc/ssh/sshd_config

# Change Port 22 to a different port (e.g., 2222)
Port 2222

# Restart SSH
sudo systemctl restart sshd

# Update firewall
sudo ufw allow 2222/tcp
sudo ufw delete allow 22/tcp

Step 8.3: Disable Password Authentication (If Using SSH Keys)

sudo nano /etc/ssh/sshd_config

# Set these values:
PasswordAuthentication no
PubkeyAuthentication yes

# Restart SSH
sudo systemctl restart sshd

Step 8.4: Set Up Fail2Ban

# Install fail2ban
sudo apt install -y fail2ban

# Copy default config
sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local

# Edit config
sudo nano /etc/fail2ban/jail.local

# Enable SSH protection (find [sshd] section):
[sshd]
enabled = true
port = 2222  # Change if you changed SSH port
maxretry = 3

# Restart fail2ban
sudo systemctl restart fail2ban
sudo systemctl enable fail2ban

# Check status
sudo fail2ban-client status

Step 8.5: Secure Docker

The Dockerfile already runs the application as a non-root user (kirby), which is a security best practice.

Step 8.6: Regular Updates

Set up automatic security updates:

sudo apt install -y unattended-upgrades
sudo dpkg-reconfigure --priority=low unattended-upgrades

9. Troubleshooting

Issue: Services Won't Start

# Check logs
docker compose logs

# Check disk space
df -h

# Check Docker daemon
sudo systemctl status docker

# Restart Docker
sudo systemctl restart docker
docker compose up -d

Issue: Can't Connect to Database

# Check if TimescaleDB is running
docker compose ps timescaledb

# Check database logs
docker compose logs timescaledb

# Test connection
docker compose exec timescaledb psql -U kirby -d kirby -c "SELECT 1;"

# Verify password in .env matches
cat .env | grep POSTGRES_PASSWORD

Issue: Collector Not Collecting Data

# Check collector logs
docker compose logs collector | tail -100

# Common issues:
# 1. No starlistings configured - run sync_config.py
# 2. WebSocket connection failed - check internet connectivity
# 3. Database connection failed - check DATABASE_URL

# Restart collector
docker compose restart collector
docker compose logs -f collector

Issue: API Not Responding

# Check if API is running
docker compose ps api

# Check API logs
docker compose logs api

# Test locally
curl http://localhost:8000/health

# Restart API
docker compose restart api

Issue: Out of Disk Space

# Check disk usage
df -h

# Check Docker disk usage
docker system df

# Clean up Docker
docker system prune -a --volumes  # CAREFUL: This removes unused data

# Check database size
docker compose exec timescaledb psql -U kirby -d kirby -c "SELECT pg_size_pretty(pg_database_size('kirby'));"

Issue: High Memory Usage

# Check container stats
docker stats

# If database is using too much memory, adjust shared_buffers:
docker compose exec timescaledb psql -U kirby -d kirby -c "SHOW shared_buffers;"

# Restart with limited memory (edit docker-compose.yml):
# Add under timescaledb service:
#   deploy:
#     resources:
#       limits:
#         memory: 1G

Issue: "python: command not found" When Running Backfill

Symptom: You get python: command not found or bash: python: command not found when trying to run the backfill script.

Cause: You're trying to run the script directly on the Ubuntu host system, which doesn't have a python command (only python3).

Solution: Use docker compose exec collector to run the script inside the Docker container:

# ❌ Wrong - tries to run on host system
python -m scripts.backfill --coin=BTC --days=1

# ❌ Also wrong - still on host
python3 -m scripts.backfill --coin=BTC --days=1

# ✅ Correct - runs inside Docker container
docker compose exec collector python -m scripts.backfill --coin=BTC --days=1

Why: The Docker container (based on python:3.13-slim) has Python installed, but your Ubuntu host may not have python symlinked to python3.

Alternative for Local Development: If you're running locally without Docker:

# On Ubuntu/Debian host (outside Docker)
python3 -m scripts.backfill --coin=BTC --days=1

Reset Everything (Nuclear Option)

# Stop and remove all containers, volumes, and images
docker compose down -v
docker system prune -a

# Start fresh
docker compose up -d
docker compose exec collector alembic upgrade head
docker compose exec collector python -m scripts.sync_config

Useful Commands

# View all logs
docker compose logs -f

# Restart specific service
docker compose restart collector

# Stop all services
docker compose stop

# Start all services
docker compose start

# Rebuild after code changes
docker compose build && docker compose up -d

# Execute command in container
docker compose exec collector python scripts/test_full_system.py

# Access database shell
docker compose exec timescaledb psql -U kirby -d kirby

# Check service health
docker compose ps

# View resource usage
docker stats

Production Checklist

Before going live, ensure:

Next Steps

Add More Trading Pairs: Edit config/starlistings.yaml and re-run sync script
Set Up Monitoring: Consider Grafana + Prometheus for metrics
Domain & SSL: Use Nginx as reverse proxy with Let's Encrypt SSL
Horizontal Scaling: Add more collector instances for different exchanges
Data Analysis: Build dashboards using the collected data

Support

For issues and questions:

GitHub Issues: https://github.com/oakwoodgates/kirby/issues
Check logs first: docker compose logs
Review TROUBLESHOOTING.md for common issues

License

See LICENSE file for details.

Last Updated: October 26, 2025 Version: 1.0.0

FilesExpand file tree

DEPLOYMENT.md

Latest commit

History

DEPLOYMENT.md

File metadata and controls

Kirby - Digital Ocean Deployment Guide

Table of Contents

Prerequisites

1. Create Digital Ocean Droplet

Step 1.1: Create Droplet

Step 1.2: Note Your Droplet IP

2. Initial Server Setup

Step 2.1: Connect via SSH

Step 2.2: Update System

Step 2.3: Set Timezone (Optional)

Step 2.4: Create Non-Root User (Recommended)

Step 2.5: Switch to New User

3. Install Docker

Step 3.1: Install Docker

Step 3.2: Install Docker Compose

Step 3.3: Add User to Docker Group

Step 3.4: Verify Docker Installation

4. Deploy Kirby

Step 4.1: Install Git

Step 4.2: Clone Repository

Step 4.3: Set Up Directory Structure

5. Configure and Start Services

Automated Deployment (Recommended)

Step 5.1: Make Deploy Script Executable

Step 5.2: Run Automated Deployment

Step 5.3: Save Your Database Password

Step 5.4: View Logs

Manual Deployment (Advanced)

Manual Step 1: Create Environment File

Manual Step 2: Review Configuration

Manual Step 3: Build Docker Images

Manual Step 4: Start Services

Manual Step 5: Wait for Database

Manual Step 6: Setup Production Database

Manual Step 7: Setup Training Database

Manual Step 8: Restart Services

Manual Step 9: View Logs

6. Verify Deployment

Step 6.1: Check API Health

Step 6.2: Check Starlistings

Step 6.3: Wait for Data Collection

Step 6.4: Check Database Directly

Step 6.5: Verify Collector is Running

Step 6.6: Backfill Historical Data (Optional)

Backfill 1 Day of Data (Quick Test)

Backfill 30 Days for a Specific Coin

Backfill All Active Starlistings

Monitor Backfill Progress

Backfill Best Practices

Verify Backfill Results

Step 6.7: Backfill Funding Rates (Optional)

Backfill 7 Days (Recommended Starting Point)

Backfill Specific Coins

Backfill All Configured Coins

Monitor Backfill Progress

Verify Backfill Data Quality

Backfill Best Practices

Common Issues

7. Monitoring and Maintenance

Step 7.1: Set Up Automatic Restarts

Step 7.2: Monitor Resource Usage

Step 7.3: Set Up Log Rotation

Step 7.4: Database Backups

Step 7.5: Monitor Disk Space

Step 7.6: Update or Deploy Application

Automated Update (Recommended)

Verify Update

Quick Update (Code Changes Only)

Rollback on Failure

Manual Update (Advanced)

Step 7.7: Fast Updates and Downtime Recovery (NEW!)

update.sh - Fast Updates (30-60 seconds)

backfill_downtime.sh - Automatic Data Recovery

Best Practices for Production Updates