Skip to content

installation.md

Latest commit

 

History

History
357 lines (270 loc) · 10.2 KB

installation.md

File metadata and controls

357 lines (270 loc) · 10.2 KB

Installation Guide

Complete setup instructions for the SWF Testbed.

Prerequisites

Before starting, ensure you have:

  • Python 3.9+ with pip
  • Docker Desktop (for infrastructure services)
  • Git for repository management
  • Administrative access (for some package installations)

Step 1: Clone the Repositories

Clone all three SWF repositories as siblings in the same parent directory:

# Create a directory for the SWF project
mkdir swf-project && cd swf-project

# Clone all repositories
git clone https://github.com/BNLNPPS/swf-testbed.git
git clone https://github.com/BNLNPPS/swf-monitor.git
git clone https://github.com/BNLNPPS/swf-common-lib.git

# Your directory structure should now look like:
# swf-project/
# ├── swf-testbed/
# ├── swf-monitor/
# └── swf-common-lib/

Step 2: Environment Configuration

IMPORTANT FOR BNL/pandaserver02 DEPLOYMENT: For deployments on systems with corporate proxies (like BNL pandaserver02), add this to your ~/.env file:

NO_PROXY=localhost,127.0.0.1,0.0.0.0

This prevents proxy interference with local service communications (Django, ActiveMQ, PostgreSQL, WebSocket connections). Other deployment environments may have different proxy requirements.

Step 3: Set Up Infrastructure Services

Choose one of the following approaches:

Option A: Using Docker (Recommended)

  1. Install Docker Desktop and ensure it's running
  2. Navigate to swf-testbed: 
    cd swf-testbed
  3. Start services: 
    docker-compose up -d

Option B: Local Installation (macOS with Homebrew)

  1. Install PostgreSQL and ActiveMQ:

    brew install postgresql@14 activemq

  2. Start services:

    brew services start postgresql@14
brew services start activemq

  3. Create database:

    createdb swfdb

Step 4: Configure Environment Variables

Core Infrastructure Configuration

The testbed requires environment variables for service coordination. Create/edit ~/.env:

# Export all variables to make them available to subprocesses
export NO_PROXY=localhost,127.0.0.1,0.0.0.0
export no_proxy=localhost,127.0.0.1,0.0.0.0

# Database configuration (PostgreSQL)
export DB_NAME='swfdb'
export DB_USER='your_db_user'
export DB_PASSWORD='your_db_password'
export DB_HOST='localhost'
export DB_PORT='5432'

# ActiveMQ configuration
export ACTIVEMQ_HOST='your_activemq_host'
export ACTIVEMQ_PORT=61612
export ACTIVEMQ_USER='your_activemq_user'
export ACTIVEMQ_PASSWORD='your_activemq_password'
export ACTIVEMQ_HEARTBEAT_TOPIC='epictopic'
export ACTIVEMQ_USE_SSL=True
export ACTIVEMQ_SSL_CA_CERTS='/path/to/full-chain.pem'

# Monitor URLs
export SWF_MONITOR_URL=https://localhost:8443      # Authenticated API calls
export SWF_MONITOR_HTTP_URL=http://localhost:8002  # REST logging (no auth)

# API Authentication
export SWF_API_TOKEN=your_token_here

# Django configuration
export SECRET_KEY='django-insecure-dev-key-for-testing-only-change-for-production'
export DEBUG=True

# Unset proxy variables for localhost connections
unset http_proxy https_proxy HTTP_PROXY HTTPS_PROXY

swf-monitor Configuration

  1. Configure swf-monitor secrets:

    cd ../swf-monitor
cp .env.example .env

    Edit the .env file with your database credentials:

    export DB_PASSWORD='admin'  # Match your Docker/local PostgreSQL password
export SECRET_KEY='django-insecure-dev-key-for-testing-only-change-for-production-12345678901234567890'

swf-testbed MCP Server Authentication

The .mcp.json at the swf-testbed repo root configures Claude Code to use the swf-monitor MCP server at https://pandaserver02.sdcc.bnl.gov/swf-monitor/mcp/, which requires a bearer token (env var SWF_MONITOR_MCP_TOKEN).

On the swf-testbed host (pandaserver02), a shared token file is provided for logged-in users. Add this line to your ~/.bashrc:

[ -r /data/wenauseic/swf-environment ] && source /data/wenauseic/swf-environment

Step 5: Set Up Python Environment and Install Dependencies

You can either use uv or plain pip to set up the Python environment.

Using uv

The following commands will create a virtual environment in .venv (if needed) and install swf-testbed and all of its dependencies:

cd ../swf-testbed
uv sync
source .venv/bin/activate

Using pip

  1. Navigate to swf-testbed and create virtual environment:

    cd ../swf-testbed
python -m venv .venv
source .venv/bin/activate

  2. Install all packages in development mode:

    # Install swf-common-lib (shared utilities)
pip install -e ../swf-common-lib

# Install swf-monitor (Django web application)
pip install -e ../swf-monitor

# Install swf-testbed CLI
pip install -e .

Changing dependencies (reqmts ⇒ dev-update ⇒ prod)

The shared .venv is the source of truth for what runs in production: the swf-monitor deploy copies this venv verbatim — it does not pip install from requirements.txt. A dependency change therefore reaches production only if it is applied to this venv:

  1. reqmts — edit the declarations (swf-monitor/requirements.txt and/or swf-monitor/pyproject.toml). Use >= floors, never == hard pins.
  2. dev-update — re-run the editable install so the venv, and the editable package metadata that pip check reads, reflect the change: 
    pip install -e ../swf-common-lib ../swf-monitor .
    Removing a dependency? pip install never uninstalls — pip uninstall the orphan explicitly.
  3. prod — deploy swf-monitor, which copies the now-synced venv: 
    sudo deploy-swf-monitor.sh branch infra/baseline-vNN

The deploy refuses to ship a venv that has drifted from the declared requirements (a pip install -r requirements.txt --dry-run + pip check gate), so a skipped dev-update fails loudly instead of shipping stale code.

Step 6: Initialize and Run the Testbed

  1. Initialize the testbed:

    swf-testbed init

  2. Set up Django database:

    # Run Django migrations and create superuser
python ../swf-monitor/src/manage.py migrate
python ../swf-monitor/src/manage.py createsuperuser

  3. Load sample data (optional):

    # Load fake log data to populate the monitoring interface
python ../swf-monitor/scripts/load_fake_logs.py

  4. Start the monitor web interface:

    # Start Django monitor with dual HTTP/HTTPS support
../swf-monitor/start_django_dual.sh

    This starts the monitor on:

    • HTTP (port 8002): For REST logging from agents (no authentication)
    • HTTPS (port 8443): For authenticated API calls (STF files, workflows, runs)

  5. Start the testbed agents:

    # If using Docker:
swf-testbed start

# If using local services:
swf-testbed start-local

  6. Check status:

    swf-testbed status  # or status-local

Step 7: Verify Setup

  1. Check web interfaces:

  2. Run tests:

    # Run all tests across all repositories
./run_all_tests.sh

# Or run tests for individual components
cd swf-monitor && ./run_tests.sh

Secrets and Configuration Management

The testbed uses a hierarchical configuration approach:

  • Core infrastructure: Environment variables in ~/.env (required - database, messaging, proxy settings)
  • swf-monitor: Django application secrets in swf-monitor/.env (required - core infrastructure)
  • swf-data-agent: Agent configuration in swf-data-agent/.env (if present)
  • swf-processing-agent: PanDA credentials in swf-processing-agent/.env (if present)

Setting Up swf-monitor Environment Variables

The Django monitoring application requires database and messaging credentials. To configure:

  1. Copy the environment template:

    cp swf-monitor/.env.example swf-monitor/.env

  2. Edit the .env file with your actual credentials:

    • Set DB_PASSWORD to match your PostgreSQL setup
    • Generate a unique SECRET_KEY for Django
    • Configure messaging credentials if using external ActiveMQ

  3. Load environment:

    source swf-monitor/.env

The Django application will automatically read these values during startup.

Security Considerations

For Development:

  • Use the provided development secrets and database passwords
  • The .env.example files contain safe defaults for local testing

For Production:

  • Generate secure, unique passwords for all services
  • Use proper SSL certificates (not self-signed)
  • Set DEBUG=False in Django configuration
  • Configure proper ALLOWED_HOSTS for Django
  • Use environment-specific configuration management

Secret Storage:

  • Never commit actual secrets to version control
  • Use .env files (which are .gitignored) for local development
  • Consider using dedicated secret management for production deployments

Troubleshooting Installation

Common Issues

  1. Docker services not starting:

    docker-compose ps
docker-compose logs postgresql
docker-compose logs activemq

  2. Virtual environment issues:

    # Ensure you're in the right directory and venv is activated
cd swf-testbed
source .venv/bin/activate
which python  # Should show .venv/bin/python

  3. Proxy connection issues:

    • Ensure NO_PROXY is set in ~/.env
    • Check that proxy variables are unset for localhost

  4. Database connection errors:

    • Check database credentials in .env files

  5. Import errors:

    # Reinstall packages in correct order
pip install -e ../swf-common-lib ../swf-monitor .

Next Steps

After successful installation: