Skip to content

PRODUCTION_INSTALL.md

Latest commit

 

History

History
600 lines (435 loc) · 14.8 KB

PRODUCTION_INSTALL.md

File metadata and controls

600 lines (435 loc) · 14.8 KB

AllSkyHyde Production Installation Guide

This guide explains how to install AllSkyHyde as a production service using Gunicorn and systemd.

Overview

The production installation uses:

  • Gunicorn - Production WSGI HTTP server for Flask applications
  • systemd - Linux service manager for auto-start and management
  • Virtual Environment - Isolated Python environment for dependencies

Prerequisites

  • Linux system with systemd (Raspberry Pi OS, Ubuntu, Debian, etc.)
  • Python 3.7 or higher
  • User account with sudo privileges
  • Internet connection for package installation

Quick Installation

  1. Navigate to the AllSkyHyde directory:

    cd /home/yourusername/AllSkyHyde

  2. Run the installation script:

    ./install_production.sh

  3. Follow the interactive prompts:

    • Confirm removal of any previous installations
    • Verify the installation directory
    • Choose whether to enable shutdown/restart from web interface

  4. Access the web interface:

    http://your-pi-ip-address:5000

What the Installation Script Does

1. Pre-Installation Checks

  • Verifies not running as root
  • Detects and removes previous installations
  • Confirms installation directory
  • Ensures installation is in user's home directory

2. Dependency Installation

  • Checks for Python3 and pip
  • Creates Python virtual environment (if not exists)
  • Installs all required packages from requirements.txt
  • Verifies Gunicorn installation

3. Directory Setup

  • Creates image directory at ~/allsky_images
  • Verifies all required files are present

4. Service Configuration

  • Creates systemd service file: /etc/systemd/system/allskyhyde.service
  • Configures Gunicorn with:
    • 2 worker processes
    • 4 threads per worker
    • Automatic restart on failure
    • Logging to application directory

5. Optional Sudo Configuration

  • Allows web interface to restart/shutdown system
  • Grants sudo access for specific commands only
  • Completely optional and can be skipped

6. Service Activation

  • Enables service to start on boot
  • Starts the service immediately
  • Verifies service is running

Installation Location

The script is designed to work with installations in your home directory (e.g., /home/pi/AllSkyHyde).

Service Management

View Service Status

sudo systemctl status allskyhyde

Stop Service

sudo systemctl stop allskyhyde

Start Service

sudo systemctl start allskyhyde

Restart Service

sudo systemctl restart allskyhyde

View Live Logs

sudo journalctl -u allskyhyde -f

Disable Auto-Start

sudo systemctl disable allskyhyde

Re-Enable Auto-Start

sudo systemctl enable allskyhyde

Automatic Service Restart Configuration

The AllSkyHyde service is configured to automatically restart if it crashes or stops unexpectedly. This section explains how to set up and configure this behaviour.

Understanding the Restart Settings

The systemd service file includes these restart-related settings:

Setting Value What it Does
Restart=always always Restarts the service whenever it stops (crash or clean exit)
RestartSec=10 10 seconds Waits 10 seconds before attempting to restart
StartLimitBurst=5 5 attempts Maximum number of restart attempts allowed
StartLimitIntervalSec=300 5 minutes Time window for the restart limit

How It Works

  1. If the service crashes: systemd waits 10 seconds, then restarts it automatically
  2. If it crashes repeatedly: After 5 restarts within 5 minutes, systemd stops trying (prevents infinite loops)

Setting Up Automatic Restart (Step-by-Step)

If your service file doesn't have these settings, follow these steps:

Step 1: Open the Service File for Editing

sudo nano /etc/systemd/system/allskyhyde.service

You will be prompted for your password. Type it and press Enter.

Step 2: Edit the Service File

Replace the entire contents with this configuration (adjust paths if your username is different):

[Unit]
Description=AllSkyHyde Web Application
After=network.target
StartLimitIntervalSec=300
StartLimitBurst=5

[Service]
Type=simple
User=kickpi
Group=kickpi
WorkingDirectory=/home/kickpi/AllSkyCode
Environment=PATH=/home/kickpi/AllSkyCode/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

# Gunicorn configuration
ExecStart=/home/kickpi/AllSkyCode/venv/bin/gunicorn --bind 0.0.0.0:5000 --workers 1 --threads 4 --timeout 120 --access-logfile /home/kickpi/AllSkyCode/gunicorn-access.log --error-logfile /home/kickpi/AllSkyCode/gunicorn-error.log --log-level info flask_app:app

# Restart configuration
Restart=always
RestartSec=10

# Security settings
PrivateTmp=true

[Install]
WantedBy=multi-user.target

Important: Change kickpi to your actual username and /home/kickpi/AllSkyCode to your actual installation path.

Step 3: Save and Exit

  • Press Ctrl + O (letter O) to save
  • Press Enter to confirm the filename
  • Press Ctrl + X to exit the editor

Step 4: Reload systemd Configuration

After editing the service file, tell systemd to reload its configuration:

sudo systemctl daemon-reload

Step 5: Restart the Service

sudo systemctl restart allskyhyde

Step 6: Verify the Settings

Check that the restart settings are applied:

systemctl show allskyhyde.service | grep -E "Restart=|RestartUSec|StartLimit"

You should see output like:

Restart=always
RestartUSec=10s
StartLimitIntervalSec=5min
StartLimitBurst=5

Testing Automatic Restart

To test that automatic restart is working:

# Find the process ID
sudo systemctl status allskyhyde | grep "Main PID"

# Kill the process (simulates a crash)
sudo kill -9 <PID>

# Wait 15 seconds, then check status
sleep 15
sudo systemctl status allskyhyde

The service should show as "active (running)" with a new PID.

Viewing Restart History

To see if the service has restarted recently:

sudo journalctl -u allskyhyde --since "1 hour ago" | grep -E "Started|Stopped|Starting"

Network Stay-Alive Monitor

AllSkyHyde includes a built-in network monitoring feature that automatically detects network disconnections and attempts to recover connectivity.

How It Works

  1. Regular Connectivity Checks: Every 60 seconds (configurable), the system pings Google's DNS server (8.8.8.8)
  2. Connection Loss Detection: If the ping fails, the system knows the network is down
  3. Reconnection Attempts: The system tries to restart network interfaces (up to 3 attempts)
  4. Automatic Reboot: If reconnection fails, the system reboots to restore connectivity
  5. Rate Limiting: Maximum 5 reboots per hour to prevent endless reboot loops
  6. Automatic Recovery: When connection is restored, all counters reset

Configuration Constants

These values are defined at the top of flask_app.py:

STAY_ALIVE_PING_HOST = "8.8.8.8"              # Host to ping (Google DNS)
STAY_ALIVE_PING_PORT = 53                      # Port to connect to
STAY_ALIVE_CHECK_INTERVAL_SECONDS = 60         # How often to check (seconds)
STAY_ALIVE_MAX_REBOOT_ATTEMPTS = 5             # Max reboots per tracking period
STAY_ALIVE_TRACKING_PERIOD_SECONDS = 3600      # Tracking period (1 hour)
STAY_ALIVE_CONNECTION_TIMEOUT_SECONDS = 10     # Connection timeout
STAY_ALIVE_RETRY_DELAY_SECONDS = 30            # Delay between reconnection attempts
STAY_ALIVE_MAX_RECONNECT_ATTEMPTS = 3          # Reconnection attempts before reboot

Viewing Network Status

Via Web Interface

Navigate to System Status page - the Network Monitor card shows:

  • Current connection status (Connected/Degraded/Disconnected)
  • Last successful ping time
  • Number of reboot attempts in the current hour
  • Recent activity log

Via API

curl http://localhost:5000/api/stay_alive/status

Returns JSON with all status information.

Via Command Line (Remote)

ssh kickpi@192.168.0.102 "curl -s http://localhost:5000/api/stay_alive/status | python3 -m json.tool"

API Endpoints

Endpoint Method Description
/api/stay_alive/status GET Get current monitor status and statistics
/api/stay_alive/start POST Start the network monitor
/api/stay_alive/stop POST Stop the network monitor
/api/stay_alive/test_connection GET Test network connectivity now
/api/stay_alive/logs GET Get all stay-alive log entries

Customising the Configuration

To change the stay-alive settings, edit flask_app.py:

nano ~/AllSkyCode/flask_app.py

Find the section starting with # ==================== STAY-ALIVE CONSTANTS ==================== and modify the values.

After making changes, restart the service:

sudo systemctl restart allskyhyde

Disabling the Stay-Alive Monitor

To disable the network monitor, you can either:

  1. Via API (temporary - until next restart):

    curl -X POST http://localhost:5000/api/stay_alive/stop

  2. Permanently: Edit flask_app.py and change:

    stay_alive_enabled = False  # Change from True to False

Log Files

The application creates two log files in the installation directory:

  • gunicorn-access.log - HTTP access logs (all requests)
  • gunicorn-error.log - Application errors and output

View logs:

tail -f ~/AllSkyHyde/gunicorn-error.log
tail -f ~/AllSkyHyde/gunicorn-access.log

Configuration Files

  • app_config.json - Application settings (location, API keys, capture settings)
  • requirements.txt - Python dependencies

Port Configuration

By default, the application runs on port 5000.

To change the port:

  1. Edit the service file: sudo nano /etc/systemd/system/allskyhyde.service
  2. Change the --bind 0.0.0.0:5000 line to your desired port
  3. Reload and restart: 
    sudo systemctl daemon-reload
sudo systemctl restart allskyhyde

Firewall Configuration

If you have a firewall enabled, allow port 5000:

# For UFW (Ubuntu/Debian)
sudo ufw allow 5000/tcp

# For firewalld (CentOS/RHEL)
sudo firewall-cmd --permanent --add-port=5000/tcp
sudo firewall-cmd --reload

Uninstallation

To remove the service:

cd /home/yourusername/AllSkyHyde
./uninstall_production.sh

The uninstall script will:

  • Stop and disable the service
  • Remove the systemd service file
  • Optionally remove sudo configuration
  • Optionally remove logs and config files
  • Optionally remove Python virtual environment

Note: Image files in ~/allsky_images are NOT removed automatically.

Helper Scripts

The installation includes several helper scripts:

  • install_production.sh - Main installation script
  • uninstall_production.sh - Complete removal script
  • fix_venv.sh - Fix virtual environment issues (WSL/Windows compatibility)
  • fix_paths.sh - Fix hardcoded paths in configuration file

Troubleshooting

Hardcoded Paths

If you moved the installation directory or see path-related errors, run:

cd ~/AllSkyHyde
./fix_paths.sh

This will update app_config.json with the correct paths based on your current directory.

Virtual Environment Issues (WSL/Windows)

If you see an error like /home/user/AllSkyHyde/venv/bin/pip: No such file or directory:

This happens when you have a Windows-style virtual environment on WSL. The installation script will automatically detect and fix this, or you can run:

cd ~/AllSkyHyde
./fix_venv.sh

This will recreate the virtual environment for Linux.

Service Won't Start

Check the service status:

sudo systemctl status allskyhyde

View recent errors:

sudo journalctl -u allskyhyde -n 50 --no-pager

Check error log:

tail -n 50 ~/AllSkyHyde/gunicorn-error.log

Permission Errors

Ensure the installation directory is owned by your user:

ls -la ~/AllSkyHyde

Fix permissions if needed:

sudo chown -R $USER:$USER ~/AllSkyHyde

Dependencies Missing

Reinstall dependencies:

cd ~/AllSkyHyde
source venv/bin/activate
pip install -r requirements.txt

Port Already in Use

Check what's using port 5000:

sudo lsof -i :5000

Kill the process or change the port in the service file.

Can't Access Web Interface

  1. Verify service is running:

    sudo systemctl status allskyhyde

  2. Check if port is listening:

    sudo netstat -tlnp | grep 5000

  3. Test locally:

    curl http://localhost:5000

  4. Check firewall settings

  5. Verify IP address: hostname -I

Upgrading the Application

To upgrade after pulling new code:

cd ~/AllSkyHyde
git pull  # if using git
sudo systemctl restart allskyhyde

If new dependencies were added:

source venv/bin/activate
pip install -r requirements.txt
sudo systemctl restart allskyhyde

Performance Tuning

Adjusting Workers and Threads

Edit the service file to tune Gunicorn:

sudo nano /etc/systemd/system/allskyhyde.service

Guidelines:

  • Workers: (2 x CPU cores) + 1
  • Threads: 2-4 per worker
  • Timeout: Increase if captures take longer

Example for Raspberry Pi 4 (4 cores):

--workers 4
--threads 4
--timeout 180

After changes:

sudo systemctl daemon-reload
sudo systemctl restart allskyhyde

Security Notes

  1. Sudo Configuration: Only enable if you trust all users who can access the web interface
  2. Firewall: Consider restricting access to port 5000 to your local network
  3. HTTPS: For remote access, consider using a reverse proxy (nginx) with HTTPS
  4. Updates: Keep your system and Python packages updated

Using with Nginx (Advanced)

For production deployments, consider using Nginx as a reverse proxy:

  1. Install Nginx:

    sudo apt-get install nginx

  2. Create Nginx configuration:

    sudo nano /etc/nginx/sites-available/allskyhyde

  3. Add configuration:

    server {
    listen 80;
    server_name your-domain.com;

    location / {
        proxy_pass http://127.0.0.1:5000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}

  4. Enable and restart:

    sudo ln -s /etc/nginx/sites-available/allskyhyde /etc/nginx/sites-enabled/
sudo nginx -t
sudo systemctl restart nginx

Support

If you encounter issues:

  1. Check the troubleshooting section above
  2. Review log files for error messages
  3. Ensure all prerequisites are met
  4. Verify file permissions

Additional Resources