Copilot Instructions for Scout Requirements Archive

Repository Overview

This is a Python web crawling project that archives Scouts BSA merit badge requirements from scouting.org. The project uses Scrapy for crawling, uv for Python package management, and GitHub Actions for automated archiving. The repository generates both JSON and Markdown files for each merit badge and deploys a static website to GitHub Pages.

Repository Stats:

Language: Python 3.12
Framework: Scrapy 2.11.2+
Package Manager: uv (NOT pip/pipenv/poetry)
Size: ~150 merit badges archived
CI/CD: GitHub Actions with weekly automated runs

Build and Development Commands

CRITICAL: Always use uv run prefix for Python commands. Never use bare python or pip.

Environment Setup

# Install uv first (if not available)
# Check version: uv --version

# Install dependencies (uv handles virtual environment automatically)
uv sync

Core Development Commands

# Clean all build artifacts (always run first when debugging)
make clean

# Run all code quality checks (REQUIRED before commits)
make check

# Individual quality checks
make lint      # Fix linting issues
make format    # Format code with ruff
make pre-commit # Run all pre-commit hooks

Scraping Commands

# Create required directories
make dirs

# Run full archive process (takes 10-15 minutes)
make archive-merit-badges

# Test single merit badge (for development)
make archive-merit-badges-url URL="https://www.scouting.org/merit-badges/camping/"

# Generate index file
make index-merit-badges

# Validate archive results
make validate-merit-badges

# Generate change report
make report-merit-badges

# Generate change report for cub adventures
make report-cub-adventures

# Run complete pipeline
make all

# Clean all output
make clean

# Clean only cub adventures output
make clean-cub-adventures

Browser-Based Spider Development

For debugging crawlers, use Playwright MCP tools to validate selectors against live websites:

# Test CSS selectors in browser
browser_navigate("https://www.scouting.org/skills/merit-badges/all/")
browser_evaluate(() => {
  return {
    linkCount: document.querySelectorAll("h2 a[href*='/merit-badges/']").length,
    badgeNames: Array.from(document.querySelectorAll("h2 a")).slice(0,3).map(a => a.textContent)
  };
})

# Test individual badge page structure
browser_navigate("https://www.scouting.org/merit-badges/camping/")
browser_evaluate(() => {
  return {
    badgeName: document.querySelector("h1.elementor-heading-title")?.textContent,
    requirementItems: document.querySelectorAll("div.mb-requirement-item").length,
    eagleRequired: !!document.querySelector("h2:contains('Eagle Required')")
  };
})

Browser validation workflow:

Navigate to target pages with browser_navigate
Test selectors with browser_evaluate
Validate structure across multiple badge types
Document findings before updating spider code

Available browser tools:

browser_navigate - Visit target URLs
browser_evaluate - Test JavaScript/selectors in browser context
browser_snapshot - Capture page structure for analysis
browser_click, browser_type, browser_wait_for - Interact with pages

Key benefits:

Real-time validation against live site structure
Detect when websites update their HTML structure
Identify overly broad or broken selectors
Test consistency across different page types

Common Issues and Solutions

Problem: Error: build/merit-badges is not a valid directory

Solution: Run make dirs first, then make archive

Problem: Scrapy command not found

Solution: Always use uv run prefix: uv run python -m scrapy

Problem: Pre-commit hooks fail

Solution: Run make check to fix all formatting/linting issues

Problem: uv.lock out of date

Solution: Run uv lock to update lock file

GitHub Actions Build Failure Investigation

List recent runs for the workflow

gh run list -w archive.yml -L 5 --json databaseId,conclusion,createdAt,event,status,url,displayTitle | jq -r '.[] | {id: .databaseId, conclusion, status, event, createdAt, url, displayTitle}'

Inspect the failed run (fast path)

gh run view <RUN_ID> --log-failed

If you need full logs

gh run view <RUN_ID> --log | rg -n "error|failed|exception|traceback|validation|requirements|http|timeout|blocked"

Identify the failure class

Validation errors: look for “Only 0 requirements found” or missing URLs.
Crawl failures: look for HTTP 4xx/5xx, timeouts, or blocked/robot text.

Reproduce locally (single badge)

make archive-merit-badges-url URL="https://www.scouting.org/merit-badges/<slug>/"

Then inspect run-test.log and build/merit-badges/<slug>-merit-badge.json.

Interpretation

Local success, CI failure → likely temporary site response.
Local failure → likely parsing change; verify selectors with browser tools.

Project Architecture

Architecture Highlights

Merit Badge pipeline: HTML → Raw node tree → Semantic tree → Markdown (src/scout_archive/requirements_pipeline.py).
Test Lab badges: Different layout; parsed via the Test Lab extractor and tagged with is_lab. Test Lab note is rendered in the Overview section.
Outputs: JSON + Markdown under build/merit-badges/ and build/cub-scout-adventures/.
For full details, see docs/architecture-overview.md. For JSON schema details, see docs/json-format.md.

Directory Structure

├── src/                              # Source code
│   ├── scout_archive/                # Scrapy project
│   │   ├── spiders/                  # Web scrapers
│   │   │   └── merit_badges.py       # Main merit badge spider
│   │   ├── items.py                  # Data models
│   │   ├── pipelines.py              # Data processing
│   │   └── settings.py               # Scrapy configuration
│   ├── scripts/                      # Utility scripts
│   │   ├── validate-merit-badges-archive.py       # Archive validation
│   │   ├── make-merit-badges-index-file.py        # Index generation
│   │   └── generate-merit-badges-change-report.py # Change detection
│   └── scrapy.cfg                    # Scrapy project config
├── build/                            # Generated output (gitignored)
│   └── merit-badges/                 # JSON/MD files + assets
├── .github/workflows/                # CI/CD automation
│   └── archive.yml                   # Weekly archiving workflow
├── Makefile                          # Build automation
├── pyproject.toml                    # Python project config
├── uv.lock                           # Dependency lock file
└── .pre-commit-config.yaml           # Code quality hooks

Key Files to Understand

src/scout_archive/spiders/merit_badges.py - Main spider that:

Discovers merit badge URLs from scouting.org/skills/merit-badges/all/
Discovers Test Lab badge URLs from scouting.org/skills/merit-badges/test-lab/
Extracts badge name, overview, requirements, PDFs, images
Uses CSS selectors: h2 a[href*='/merit-badges/'] for discovery
Parses standard merit badge requirements and Test Lab requirements via the pipeline

Makefile - All build commands use uv run prefix. Key targets:

make all - Complete pipeline (archive → index → validate → report)
make check - Code quality (lint → format → pre-commit)
make clean - Remove all build artifacts

.github/workflows/archive.yml - Automated weekly archiving:

Runs every Sunday at 9 AM UTC
Creates GitHub releases with archived data
Deploys website to GitHub Pages
Commits changes only when merit badge requirements change

Configuration Files

pyproject.toml - Python project configuration:

Requires Python >=3.12
Dependencies: scrapy, jinja2, markdownify, pillow
Dev dependencies: pre-commit

.pre-commit-config.yaml - Code quality enforcement:

prettier (YAML/JSON formatting)
ruff (Python linting and formatting)
uv lock check (dependency validation)

CI/CD Pipeline

Pre-commit Checks (runs on every commit):

prettier formatting
ruff format (code formatting)
ruff check (linting with auto-fix)
uv lock validation

GitHub Actions Workflow:

Archive merit badges (10-15 min)
Generate index and validate
Create release if changes detected
Deploy website to GitHub Pages
Upload artifacts (JSON, Markdown, images)

Data Flow

Scraping: merit_badges.py spider crawls scouting.org
Processing: Pipelines generate JSON + Markdown files
Validation: validate-merit-badges-archive.py checks data integrity
Change Detection: generate-merit-badges-change-report.py compares versions
Deployment: GitHub Pages serves static site

Development Guidelines

Always run before making changes:

make clean && make check

Test changes with single badge:

make archive-merit-badges-url URL="https://www.scouting.org/merit-badges/camping/"

Validate changes:

make validate-merit-badges

For scraper modifications:

Use Playwright MCP tools to test selectors in browser first
Test CSS selectors with browser_evaluate before updating spider
Navigate to live pages with browser_navigate to validate structure
Update both JSON and Markdown output
Validate against multiple badge types (Eagle-required vs regular)

For Cub Scout adventure debugging:

ALWAYS use makefile commands for testing: make archive-cub-adventures-url URL="https://www.scouting.org/cub-scout-adventures/adventure-name/"
Never use bare scrapy commands - use make targets which handle proper directory setup and parameters
Use correct URL format: https://www.scouting.org/cub-scout-adventures/adventure-name/
Test with: make archive-cub-adventures-url URL="https://www.scouting.org/cub-scout-adventures/running-with-the-pack/"
Key selectors: rank from a[href*="/adventures/wolf/"], category/type from span elements
Adventure overview: Use XPath //h2[contains(text(), 'Snapshot of adventure')]/following::p[1]//text() to extract text from first paragraph after "Snapshot of adventure" heading
Bobcat adventures: Special handling required - found on rank pages with links like "View Lion Bobcat" going to /cub-scout-adventures/bobcat-{rank}/.
Debug approach: Add self.logger.info() statements to trace selector results, try multiple XPath/CSS approaches (direct siblings, descendants, following elements), check log files in project root
Avoid capturing entire HTML page content in category field

Diagnosing Validation Warnings

When you see warnings like:

Warnings in wood-carving-merit-badge: Missing image URL, Missing PDF URL, Missing shop URL

Diagnosis steps:

Test individual badge scraping:

make archive-merit-badges-url URL="https://www.scouting.org/merit-badges/wood-carving/"

Check the generated JSON:

cat build/merit-badges/wood-carving-merit-badge.json | jq '.pdf_url, .shop_url, .image_url'

Use browser tools to verify page structure:

# Navigate to the page and check for elements
browser_navigate("https://www.scouting.org/merit-badges/wood-carving/")
browser_evaluate(() => {
  return {
    pdfLinks: Array.from(document.querySelectorAll('a[href*=".pdf"]')).map(a => a.href),
    shopLinks: Array.from(document.querySelectorAll('a[href*="scoutshop.org"]')).map(a => a.href),
    badgeImage: document.querySelector('img[alt*="WoodCarving"], img[src*="WoodCarving"]')?.src
  };
})

Common causes:

Transient issues: Network timeouts, rate limiting, temporary site issues
Site structure changes: Scouting.org updated their HTML structure
Special badges: Some newer badges (like Citizenship in Society) legitimately lack standard sections
Parsing issues: CSS selectors need updating for changed page layouts

Resolution:

For transient issues: Re-run the scraper
For legitimate missing content: Add badge-specific exceptions in validate-merit-badges-archive.py
For parsing issues: Update selectors in merit_badges.py spider after browser verification

Trust these instructions - they are validated and current. Only search for additional information if these instructions are incomplete or incorrect.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Copilot Instructions for Scout Requirements Archive

Repository Overview

Build and Development Commands

Environment Setup

Core Development Commands

Scraping Commands

Browser-Based Spider Development

Common Issues and Solutions

GitHub Actions Build Failure Investigation

Project Architecture

Architecture Highlights

Directory Structure

Key Files to Understand

Configuration Files

CI/CD Pipeline

Data Flow

Development Guidelines

Diagnosing Validation Warnings

FilesExpand file tree

AGENTS.md

Latest commit

History

AGENTS.md

File metadata and controls

Copilot Instructions for Scout Requirements Archive

Repository Overview

Build and Development Commands

Environment Setup

Core Development Commands

Scraping Commands

Browser-Based Spider Development

Common Issues and Solutions

GitHub Actions Build Failure Investigation

Project Architecture

Architecture Highlights

Directory Structure

Key Files to Understand

Configuration Files

CI/CD Pipeline

Data Flow

Development Guidelines

Diagnosing Validation Warnings