Home

Testing scrapers

I test the scrapers using a test.py script like the following:

import uoftscrapers
import logging
import sys

# Set up logging so it prints to standard output
logger = logging.getLogger("uoftscrapers")
logger.setLevel(logging.INFO)
ch = logging.StreamHandler(sys.stdout)
ch.setLevel(logging.INFO)
logger.addHandler(ch)

# Run scraper
uoftscrapers.ScraperName.scrape(location='./data')

Placed at the root of the repo, this will import the uoftscrapers from the source and not the one pip3 is managing.