Home

Testing scrapers

I test the scrapers using a script similar to the following:

import uoftscrapers
import logging
import sys

logger = logging.getLogger("uoftscrapers")
logger.setLevel(logging.INFO)
ch = logging.StreamHandler(sys.stdout)
ch.setLevel(logging.INFO)
logger.addHandler(ch)

uoftscrapers.ScraperName.scrape(location='./data')

Placed at the root of the repo, this will import the uoftscrapers from the source and not the one pip3 is managing.