Ultimate Marketplace Web Scraper

--

Description

This project is for scraping what you want from popular marketplaces of craigslist, offerup, and facebook marketplace. It is for local use and will send you a notification when new search appears.

Features

• scrapes craigslist, offerup, and facebook marketplace
• frontend UI - table of results
• can scrape multiple searches every min
• if new result comes up then will notify you on email or text
• shows logs, data, avg scrape time, avg memory use , total runs so far for backend

Requirements

• Need own server/system to run continuously
• Install scrapy, scrapy-playwright, mySQL, pandas,
• Email or Phone number for notifications

My Own Additional Challenges

When Setting up

• python3 -m venv env_name
• source env_name/bin/activate
• pip3 install scrapy scrapy-playwright
• playwright install
• pip3 install requests
• pip3 install beautifulsoup4
• pip3 install pandas

shout out to Foster Singletary for the idea