This is a basic ETL (Extract, Transform, Load) pipeline for Reddit data. The pipeline scrapes data from the League of Legends subreddit, specifically the top posts of the day, and loads it into a PostgreSQL database. This project focuses on the extraction and loading steps, omitting the transformation phase. It uses psycopg2 for PostgreSQL interaction and stores database credentials in a keys.conf configuration file.
Before running the ETL pipeline, ensure you have the following prerequisites:
-
Python 3.x installed.
-
PostgreSQL server installed and running.
-
Install the
psycopg2library using pip:pip install psycopg2. -
Install the
configparserlibrary using pip:pip install configparser -
Create a
keys.confconfiguration file with PostgreSQL credentials in the following format:[pg_config] hostname = localhost dbname = postgres user = postgres password = your_password
-
Clone this repository to your local machine.
-
Create the
keys.conffile with your PostgreSQL credentials (as described above) and save it in the project directory. -
Execute the ETL pipeline script using the following command:
python etl_pipeline.py
This will run the pipeline, scraping Reddit data and loading it into the database.
The etl_pipeline.py script performs the following tasks:
- Scrapes data from Reddit's League of Legends subreddit.
- Extracts information such as title, author, posting date/time, number of comments, and upvotes.
- Stores the extracted data in a JSON file.
- Reads PostgreSQL credentials from
keys.conf. - Connects to the PostgreSQL server.
- Checks if the
redditdbdatabase exists; if not, it creates the database. - Defines the table structure with appropriate data types to store the Reddit data.
- Connects to the
redditdbdatabase. - Creates a table in the database to store Reddit data.
- Loads the extracted data from the JSON file into the database table.
This is a basic project intended for educational purposes. Depending on your requirements, you may want to implement error handling, logging, and additional data validation or transformation steps to make the pipeline more robust and suitable for production use.