Skip to content

installation.mdx

Latest commit

 

History

History
398 lines (310 loc) · 9.98 KB

installation.mdx

File metadata and controls

398 lines (310 loc) · 9.98 KB
Raw
title Installation
description Install scrapai CLI on Linux, macOS, or Windows

System Requirements

Version **3.9** or higher For cloning the repository ~500 MB (dependencies + browser) scrapai uses SQLite by default (no database installation required). For production scale, PostgreSQL is recommended.

Supported Platforms

  • Linux (Ubuntu, Debian, CentOS, Fedora, Arch)
  • macOS (Intel and Apple Silicon)
  • Windows (10/11 via WSL only)

Installation Steps

```bash sudo apt update sudo apt install python3.9 python3.9-venv python3-pip git ``` 
    <Accordion title="CentOS/Fedora">
      ```bash
      sudo dnf install python39 python39-pip git
      ```
    </Accordion>
    
    <Accordion title="Arch Linux">
      ```bash
      sudo pacman -S python python-pip git
      ```
    </Accordion>
    
    Verify installation:
    ```bash
    python3 --version
    ```
  </Tab>
  
  <Tab title="macOS">
    Install via Homebrew:
    ```bash
    brew install python@3.9 git
    ```
    
    Or download from [python.org](https://www.python.org/downloads/macos/)
    
    Verify installation:
    ```bash
    python3 --version
    ```
  </Tab>
  
  <Tab title="Windows">
    **Windows users must use WSL (Windows Subsystem for Linux).**

    [Install WSL](https://learn.microsoft.com/en-us/windows/wsl/install), then follow the Linux instructions above.
  </Tab>
</Tabs>
```bash git clone https://github.com/discourselab/scrapai-cli.git cd scrapai-cli ``` 
<Tip>
Clone to a location with write permissions. Avoid system directories like `/usr/local/`.
</Tip>
```bash ./scrapai setup ``` 
The setup process will:

<Steps>
  <Step title="Create virtual environment">
    Creates a `.venv` directory with isolated Python packages
  </Step>
  
  <Step title="Install dependencies">
    Installs Scrapy, SQLAlchemy, Alembic, newspaper4k, trafilatura, Playwright, and more
  </Step>
  
  <Step title="Install Playwright Chromium">
    Downloads Chromium browser for JavaScript rendering and Cloudflare bypass
    
    <Warning>
    **Linux users**: If Chromium fails to launch later, you may need to install system dependencies:
    ```bash
    sudo .venv/bin/python -m playwright install-deps chromium
    ```
    This requires sudo because it installs system packages (fonts, libraries, etc.).
    </Warning>
  </Step>
  
  <Step title="Create .env file">
    Copies `.env.example` to `.env` with default SQLite configuration
  </Step>
  
  <Step title="Initialize database">
    Runs Alembic migrations to create the database schema
  </Step>
  
  <Step title="Configure Claude Code permissions">
    If using AI agents, sets up permission rules in `.claude/settings.local.json`
  </Step>
</Steps>

<Accordion title="Expected output">
  ```
  🚀 Setting up scrapai environment...
  📦 Creating virtual environment...
  ✅ Virtual environment created
  📋 Installing requirements...
  ✅ Requirements installed
  🌐 Installing Playwright Chromium browser...
  ✅ Playwright Chromium installed
  📝 Creating .env from .env.example...
  ✅ .env file created (using SQLite by default)
  📁 Checking data directory permissions...
  ✅ Have permission to write to data directory: ./data
  🗄️  Initializing database...
  ✅ Database initialized with migrations
  🔧 Configuring Claude Code permissions...
  ✅ Claude Code permissions configured
  🎉 scrapai setup complete!
  ```
</Accordion>
```bash ./scrapai verify ``` 
You should see:
```
🔍 Verifying scrapai environment...

✅ Virtual environment exists
✅ Core dependencies installed
✅ Database initialized

🎉 Environment is ready!
```

<Warning>
If any checks fail, re-run `./scrapai setup`. If issues persist, see [Troubleshooting](#troubleshooting) below.
</Warning>

Configuration

Database Configuration

scrapai uses SQLite by default (no setup required). For production, you can transfer your existing data to PostgreSQL:

Use a managed service (AWS RDS, DigitalOcean, Supabase) or install locally: 
```bash
# Linux: sudo apt install postgresql postgresql-contrib
# macOS: brew install postgresql
```
```bash DATABASE_URL=postgresql://user:password@host:5432/scrapai ``` ```bash ./scrapai db migrate ./scrapai db transfer sqlite:///scrapai.db ```

Proxy Configuration

scrapai supports smart proxy escalation. Configure proxies in .env:

# Datacenter Proxy (recommended - faster, cheaper)
DATACENTER_PROXY_USERNAME=your_username
DATACENTER_PROXY_PASSWORD=your_password
DATACENTER_PROXY_HOST=your-datacenter-proxy.com
DATACENTER_PROXY_PORT=10000  # Port 10000 = rotating IPs

# Residential Proxy (for sites that block datacenter IPs)
RESIDENTIAL_PROXY_USERNAME=your_username
RESIDENTIAL_PROXY_PASSWORD=your_password
RESIDENTIAL_PROXY_HOST=your-residential-proxy.com
RESIDENTIAL_PROXY_PORT=7000  # Port 7000 = rotating residential IPs
Proxies are optional. scrapai starts with direct connections and only uses proxies when needed (403/429 errors). It learns which domains require proxies and remembers for future crawls.

S3 Storage Configuration

For automatic uploads to S3-compatible storage (Hetzner, DigitalOcean Spaces, Wasabi, Backblaze, etc.):

S3_ACCESS_KEY=your_access_key_here
S3_SECRET_KEY=your_secret_key_here
S3_ENDPOINT=https://your-s3-endpoint.com
S3_BUCKET=your-bucket-name

Troubleshooting

**Error**: `The virtual environment was not created successfully` 
**Solution**:
```bash
# Install venv module
sudo apt install python3.9-venv  # Ubuntu/Debian

# Or use a different Python version
python3.10 -m venv .venv
```
**Error**: `Error: browserType.launch: Host system is missing dependencies` 
**Solution**: Install system dependencies:
```bash
sudo .venv/bin/python -m playwright install-deps chromium
```

This installs required system packages (fonts, libraries, etc.).
**Error**: `PermissionError: [Errno 13] Permission denied: './data'` 
**Solution**: Change the data directory in `.env`:
```bash
DATA_DIR=~/scrapai-data
```

Or fix permissions:
```bash
sudo chown -R $USER:$USER ./data
chmod -R 755 ./data
```
**Error**: `sqlalchemy.exc.OperationalError: could not connect to server` 
**Solutions**:
1. Verify PostgreSQL is running:
   ```bash
   sudo systemctl status postgresql
   ```

2. Check your `DATABASE_URL` in `.env`

3. Test connection:
   ```bash
   psql -U scrapai_user -d scrapai -h localhost
   ```

4. Check PostgreSQL logs:
   ```bash
   sudo tail -f /var/log/postgresql/postgresql-*.log
   ```
**Error**: `bash: ./scrapai: No such file or directory` 
**Solution**: Make the script executable:
```bash
chmod +x scrapai
./scrapai verify
```
**Error**: `ERROR: This package requires Python 3.9 or higher` 
**Solution**: Install a newer Python version:
```bash
# Ubuntu/Debian
sudo apt install python3.10 python3.10-venv

# macOS
brew install python@3.10

# Then recreate the venv
rm -rf .venv
python3.10 -m venv .venv
./scrapai setup
```

Upgrading

To upgrade to the latest version:

git pull origin main
./scrapai setup  # Re-run setup to install new dependencies
./scrapai db migrate  # Apply any new database migrations
Always backup your database before upgrading: ```bash # SQLite cp scrapai.db scrapai.db.backup

PostgreSQL

pg_dump -U scrapai_user scrapai > scrapai_backup.sql

</Warning>

## Uninstallation

To completely remove scrapai:

```bash
cd scrapai-cli

# Remove virtual environment
rm -rf .venv

# Remove database (if using SQLite)
rm scrapai.db

# Remove data directory
rm -rf data/

# Remove the repository
cd ..
rm -rf scrapai-cli/

Next Steps

Build your first scraper in 5 minutes Complete command reference Advanced configuration options View source code and report issues