rpscrape-pro

Professional-grade fork of joenano/rpscrape with enhanced reliability, logging, and anti-ban protection.

Key Improvements

Bug Fixes

• SSL Errors: Proper retry mechanism with exponential backoff
• HTTP 406: Browser-like headers with User-Agent rotation
• Index Out of Range: Safe data extraction with safe_get() pattern
• Race Conditions: Proper error handling for void/cancelled races

New Features

• Professional Logging: Structured logging with structlog - JSON output, file rotation, metrics collection
• Anti-Ban Protection: User-Agent rotation, request delays with jitter, ban detection
• Resilient HTTP Client: Automatic retries with tenacity, circuit breaker pattern
• Data Validation: Pydantic models for type-safe data handling
• Comprehensive Exception Hierarchy: Specific exceptions for different error types

Installation

# Clone the repository
git clone https://github.com/yourusername/bf_rpscrape.git
cd bf_rpscrape

# Install with pip (development mode)
pip install -e ".[dev]"

Quick Start

from rpscrape_pro.scrapers.race_scraper import RaceScraper
from rpscrape_pro.core.logging import setup_logging

# Setup logging
setup_logging(level="INFO", json_format=False)

# Create scraper
scraper = RaceScraper(
    base_delay=2.0,      # Base delay between requests
    max_retries=3,       # Maximum retry attempts
)

# Scrape races for a date
results = scraper.scrape(
    region="gb",
    date="2024-01-15",
    race_type="flat",
)

# Export to CSV
for result in results:
    print(result.to_csv_rows())

Configuration

Configuration via environment variables or .env file:

# Scraping settings
RPSCRAPE_BASE_DELAY=2.0
RPSCRAPE_MAX_RETRIES=5
RPSCRAPE_TIMEOUT=30

# Anti-ban settings
RPSCRAPE_SESSION_ROTATION_INTERVAL=100
RPSCRAPE_BACKOFF_MULTIPLIER=2.0
RPSCRAPE_MAX_BACKOFF=300.0

# Logging settings
RPSCRAPE_LOG_LEVEL=INFO
RPSCRAPE_LOG_FILE=logs/rpscrape.log
RPSCRAPE_LOG_JSON_FORMAT=false

Project Structure

src/rpscrape_pro/
├── core/
│   ├── logging.py       # Structured logging with metrics
│   ├── anti_ban.py      # Anti-ban protection system
│   ├── http_client.py   # Resilient HTTP client
│   └── exceptions.py    # Exception hierarchy
├── config/
│   └── settings.py      # Pydantic settings
├── models/
│   ├── race.py          # Race/Runner data models
│   └── validators.py    # Data parsing functions
├── scrapers/
│   ├── base.py          # Base scraper class
│   └── race_scraper.py  # Main race scraper
└── utils/
    ├── cleaning.py      # String cleaning
    ├── course.py        # Course utilities
    ├── date.py          # Date handling
    ├── going.py         # Going descriptions
    ├── lxml_funcs.py    # XPath helpers
    └── region.py        # Region codes

Exception Hierarchy

RpScrapeError (base)
├── NetworkError
│   ├── ConnectionError
│   ├── TimeoutError
│   └── SSLError
├── ParsingError
│   ├── XPathError
│   ├── DataExtractionError
│   └── ValidationError
├── RateLimitError
├── BanDetectedError
│   ├── IP403Error
│   ├── CaptchaError
│   └── HTTP406Error
└── VoidRaceError

Anti-Ban Features

The AntiBanManager provides:

• User-Agent Rotation: 25+ modern browser User-Agents with weighted selection
• Header Profiles: Complete browser fingerprint including sec-ch-ua, Accept-Language, etc.
• Request Delays: Configurable base delay with random jitter (0.5x - 1.5x)
• Exponential Backoff: Automatic increase on errors with configurable multiplier
• Ban Detection: Identifies 403, 429, 406 responses and CAPTCHA pages
• Session Rotation: Periodic profile changes to avoid fingerprinting

Logging

Three output modes:

# Console (development)
setup_logging(level="DEBUG", json_format=False)

# JSON (production/analysis)
setup_logging(level="INFO", json_format=True, log_file="logs/rpscrape.log")

# Metrics collection
from rpscrape_pro.core.logging import metrics_collector
print(f"Total requests: {metrics_collector.get_summary()}")

Running Tests

# All tests
pytest tests/ -v

# With coverage
pytest tests/ -v --cov=rpscrape_pro --cov-report=html

# Specific module
pytest tests/unit/test_anti_ban.py -v

Original Issues Fixed

Issue	Problem	Solution
#186	SSL errors	Retry with tenacity, proper SSL context
#185	Timing issues	Configurable delays with jitter
#182	HTTP 406	Browser-like headers with rotation
#179	Data parsing	Pydantic validation
#177	Index out of range	Safe extraction with safe_get()

License

MIT License - See original rpscrape license.

Acknowledgments

• Original rpscrape by joenano
• Built with: structlog, tenacity, pydantic, aiohttp, requests