A production-inspired Data Engineering platform for collecting, processing, monitoring, analyzing, and visualizing real-time environmental intelligence for cities worldwide.
Atmos is an end-to-end Data Engineering project that demonstrates how modern data systems ingest, validate, transform, store, analyze, monitor, and visualize real-world data.
The platform gathers live weather and air-quality information from external APIs, processes the incoming data through a structured ETL pipeline, stores the results in a relational database, generates analytical metrics, tracks pipeline performance, exports datasets, and presents insights through an interactive dashboard.
The project was intentionally built around practical Data Engineering concepts and workflows commonly found in industry environments rather than purely academic exercises.
- Global city search
- Geocoding integration
- Weather data ingestion
- Air quality ingestion
- Dynamic city onboarding
- Extract data from external APIs
- Transform raw responses into structured records
- Validate records before storage
- Load validated data into a database
- Schema validation using Pydantic
- Type enforcement
- Data quality checks
- Invalid record protection
- SQLite database
- SQLAlchemy ORM
- Relational modeling
- Foreign-key relationships
- Weather scoring
- AQI scoring
- Readiness scoring
- Risk classification
- Pipeline execution tracking
- Runtime measurement
- Success/failure tracking
- Historical execution logs
- City readiness rankings
- City explorer
- Pipeline monitoring
- Dataset explorer
- Dynamic city search
- Dockerized application
- Docker Compose support
- Portable deployment workflow
User
│
▼
Streamlit Dashboard
│
▼
Pipeline Layer
│
├── Geocoding API
├── Weather API
└── Air Quality API
│
▼
Validation Layer
│
▼
Transformation Layer
│
▼
SQLite Database
│
├── Cities
├── Weather
├── Air Quality
├── Daily Metrics
└── Pipeline Runs
│
▼
Analytics Layer
│
▼
CSV Export Layer
│
▼
Dashboard Visualizations
| Category | Technology |
|---|---|
| Language | Python 3.13 |
| Dashboard | Streamlit |
| Database | SQLite |
| ORM | SQLAlchemy |
| Data Processing | Pandas |
| Validation | Pydantic |
| API Communication | Requests |
| CLI | Typer |
| Containerization | Docker |
| Orchestration | Docker Compose |
Atmos/
│
├── app.py
├── main.py
├── requirements.txt
├── Dockerfile
├── docker-compose.yml
│
├── data/
│ └── exports/
│
├── src/
│ ├── analytics/
│ ├── clients/
│ ├── dashboard/
│ ├── monitoring/
│ ├── pipeline/
│ ├── transformers/
│ ├── validators/
│ ├── database.py
│ ├── models.py
│ ├── config.py
│ └── create_database.py
│
└── README.md
Atmos follows a traditional ETL architecture.
The system retrieves:
- City metadata
- Geographic coordinates
- Weather information
- Air quality information
from the Open-Meteo ecosystem.
Raw API responses are normalized into a consistent internal structure.
Typical transformations include:
- Datetime conversion
- Field renaming
- Type conversion
- Score generation
- Data cleaning
All transformed records pass through Pydantic validation before entering the database.
Validation ensures:
- Correct datatypes
- Required fields
- Consistent schemas
- Reliable downstream analytics
Validated records are stored using SQLAlchemy ORM and persisted in SQLite.
Stores metadata about every city processed by the system.
| Field | Description |
|---|---|
| id | Primary Key |
| city_name | City Name |
| country | Country Code |
| latitude | Latitude |
| longitude | Longitude |
| population | Population |
| timezone | Timezone |
Stores weather observations.
| Field | Description |
|---|---|
| city_id | City Reference |
| date | Observation Date |
| temperature | Temperature |
| humidity | Humidity |
| wind_speed | Wind Speed |
| precipitation | Rainfall |
| condition | Weather Condition |
Stores environmental pollution measurements.
| Field | Description |
|---|---|
| city_id | City Reference |
| date | Observation Date |
| aqi | Air Quality Index |
| pm25 | PM2.5 |
| pm10 | PM10 |
| o3 | Ozone |
| no2 | Nitrogen Dioxide |
Stores generated analytics.
| Field | Description |
|---|---|
| weather_score | Weather Quality Score |
| aqi_score | Air Quality Score |
| readiness_score | Final Composite Score |
| risk_level | Risk Classification |
Stores monitoring information.
| Field | Description |
|---|---|
| city_name | Processed City |
| start_time | Pipeline Start |
| end_time | Pipeline End |
| duration_seconds | Runtime |
| status | Success/Failure |
| records_processed | Processed Records |
| error_message | Failure Reason |
Atmos generates several derived metrics.
Measures environmental favorability using weather conditions.
Evaluates pollution conditions using AQI and pollutant measurements.
Combines environmental indicators into a single city readiness metric.
Cities are categorized into:
- LOW
- MODERATE
- HIGH
based on their environmental conditions.
Every pipeline execution is tracked.
Recorded information includes:
- City processed
- Start time
- End time
- Runtime
- Success status
- Failure status
- Records processed
- Error messages
This monitoring data powers the Pipeline Monitoring dashboard page.
Displays:
- Active cities
- Weather records
- AQI records
- Analytics records
- Readiness rankings
- Risk distribution
- Weather comparisons
- AQI comparisons
Provides city-specific analytics including:
- Readiness score
- Weather score
- AQI score
- Risk classification
- Historical metrics
- Environmental measurements
Displays:
- Total runs
- Success rate
- Average runtime
- Historical executions
Provides direct access to exported datasets.
git clone https://github.com/AA-KH/Atmos
cd Atmospython3 -m venv .venv
source .venv/bin/activatepython -m venv .venv
.venv\Scripts\activatepip install -r requirements.txtCreate all database tables:
python -m src.create_databasePopulate the system with the default city set:
python main.py run-allExport datasets:
python main.py exportView generated metrics:
python main.py metricsView pipeline statistics:
python main.py statsView pipeline history:
python main.py historystreamlit run app.pyOpen:
http://localhost:8501
Atmos allows users to search and ingest cities directly from the dashboard.
Workflow:
- Enter city name
- Fetch coordinates
- Retrieve weather data
- Retrieve air-quality data
- Validate records
- Store data
- Generate analytics
- Export datasets
- Refresh dashboard
docker build -t atmos .docker run -p 8501:8501 atmosdocker compose up --buildStop:
docker compose downRun default pipeline:
python main.py run-allRun single city:
python main.py run-city DelhiExport datasets:
python main.py exportView metrics:
python main.py metricsView pipeline statistics:
python main.py statsView pipeline history:
python main.py historyAtmos uses Open-Meteo APIs.
- Geocoding API
- Weather Forecast API
- Air Quality API
- ETL Pipelines
- Data Validation
- Data Quality Enforcement
- API Integration
- Data Modeling
- SQLAlchemy ORM
- Relational Databases
- Monitoring Systems
- Analytics Generation
- Dashboard Development
- Docker Deployment
- Production-Oriented Architecture
User Searches City
│
▼
Geocoding API
│
▼
Weather API + AQI API
│
▼
Transformation Layer
│
▼
Validation Layer
│
▼
SQLite Database
│
▼
Analytics Generation
│
▼
CSV Export Layer
│
▼
Streamlit Dashboard
A complete Data Engineering platform demonstrating practical ETL pipelines, data validation, monitoring, analytics, dashboarding, and Dockerized deployment using real-world environmental intelligence data.