Patterns and Conventions

This suite of projects started out life as a collection of scrapers running on morph.io. When we migrated them all away from morph, we inherited some of morph's design decisions (which might seem strange viewed in complete isolation).

• The entry point for bot/scraper repos is always a file called scraper.py, even if its not really a scraper. (TODO: improve this https://github.com/DemocracyClub/dc-bots/issues/8)
• Credentials are stored in env vars. Env vars for bots are always prefixed with MORPH_
• All bots/scrapers have access to the same shared env vars. (TODO: improve this https://github.com/DemocracyClub/dc-bots/issues/10)
• All bots/scrapers run in a single shared virtualenv. This means they all need to have a compatible set of dependencies. (TODO: improve this https://github.com/DemocracyClub/dc-bots/issues/6)
• You should be able to run your scraper locally without sending slack notifications/opening GH issues/etc if no credentials are set. e.g:
  • https://github.com/DemocracyClub/womblr/blob/1f1085ca911479e371c1732fccd0dfba960d5912/scraper.py#L24-L27
  • https://github.com/DemocracyClub/womblr/blob/1f1085ca911479e371c1732fccd0dfba960d5912/scraper.py#L234-L235