International Space Station Data Harvesting and Warehousing (Live Data)

Overview

In this project, the raw API data of the International Space Station (ISS) is transformed into MongoDB documents and SQL records. The project aims to unlock valuable insights that are helpful for communication timestamp analysis, message content analysis, and more. This is achieved through the effective storage capabilities of MongoDB's document-oriented storage and SQL's relational database management.

Features

Data Transformation: Raw ISS API data is transformed into MongoDB documents and SQL records.
Storage Efficiency: MongoDB's document-oriented storage and SQL's relational database management provide efficient data storage solutions.

Getting Started

Step 1: Install/Import Required Libraries

Ensure the necessary libraries are installed or imported: Streamlit, Pandas, JSON, PyMongo, Requests, Time, and Psycopg2.

Step 2: Utilize the 'API2MongoSQL' Class

Inside the class, there are six methods that handle the transformation and management of data:
API iss data converted into MongoDB Documents
Fetch MongoDB Documents
Converting MongoDB Documents Into SQL Records
Fetch SQL Table Records
Delete Document in Mongo Collection
Delete SQL Records
Note : Streamlit is integrated for GUI purposes.

Step 3: Run the Project with Streamlit

Open the command prompt in the directory where the "DataSyncPro.py" file is located. Execute the command: streamlit run DataSyncPro.py. It will open in a web browser.4

Step 4: Configure Databases

Make sure to connect your MongoDB Atlas and PostgresSQL DBMS in your local setup.

Technical Skills

Python (Scripting)
Data Collection
MongoDB
SQL
API Integration
Data Management using MongoDB (Atlas) and PostgresSQL
Integrated Development Environment (IDE): PyCharm Community Version