Multimodal Weather Forecasting via Sim2Real Hybrid Fusion

A multimodal deep learning system for high-resolution 72-hour weather forecasting.
Fusing Spatial (Satellite Imagery) and Temporal (Historical Weather Data) through Attention Mechanism.

---

Project Overview

Traditional weather forecasting often relies on isolated data streams—either looking at historical numerical data or analyzing satellite imagery independently. This project bridges that gap by introducing a Sim2Real Hybrid Fusion System that comprehensively analyzes both modalities parallelly.

By employing a Temporal Transformer Encoder and an Attention-inspired Fusion Module, the model dynamically balances spatial context and temporal sequences, identifying intricate meteorological patterns to produce highly accurate 72-hour forecasts for Temperature, Humidity, Wind Speed, and Precipitation.

Key Achievements (Recruiter's Quick Look)

• Engineered a Multimodal Pipeline: Successfully processed and fused disparate data types (time-series + image data) utilizing custom PyTorch Dataset and DataLoader classes to handle multimodal integration.
• Implemented Advanced Architectures: Built and trained a Transformer-based time-series encoder alongside a CNN spatial feature extractor from scratch.
• Designed Attention-based Fusion: Replaced static concatenation with an adaptive attention mechanism, calculating dynamic weights for spatial vs. temporal features depending on the immediate meteorological context.
• Robust Evaluation & Benchmarking: Outperformed standard LSTM and CNN+LSTM baselines, achieving a +61% improvement in R² score compared to traditional isolated methods.
• Production Ready Inference: Developed an optimized inferencing script for real-world application and real-time forecasting.

---

Technology Stack

• Core ML Framework: PyTorch, Torchvision
• Data Engineering: Pandas, NumPy, Scikit-learn
• Visualization: Matplotlib, Seaborn
• Development & Logging: Python 3.11, logging, TQDM

---

Repository Structure

Multimodal-Vision-Transformer-Weather/
├── report/               # LaTeX report
├── results/              # Output directory for plots and metrics
├── src/                  # Source code directory
│   ├── baselines.py      # LSTM and CNN+LSTM Baseline implementations
│   ├── clean_csv.py      # Data preprocessing and cleaning script
│   ├── dataset.py        # Multimodal DataLoader and data transformations
│   ├── evaluate.py       # Standard evaluation script
│   ├── evaluate_advanced.py  # Report generation and metrics visualization
│   ├── inference.py      # Real-time forecasting invocation
│   ├── main.py           # Main entry point for execution
│   ├── models.py         # Sim2Real Hybrid Architecture (Core logic)
│   └── train.py          # Model training loop
├── .gitignore            # Git ignore configuration
├── requirements.txt      # Project dependencies
└── README.md             

---

System Architecture

graph TD
    A[Historical Weather Data] -->|Time-Series| B(Temporal Transformer Encoder)
    C[Satellite Imagery] -->|Spatial| D(CNN Feature Extractor)
    
    B --> E{Attention Fusion Module}
    D --> E
    
    E -->|Context-Aware Hidden State| F(Recursive MLP Forecaster)
    F -->|t+1, t+2... t+72| G[72-Hour Predictions]
    
    classDef data fill:#2d3436,stroke:#74b9ff,stroke-width:2px,color:#fff;
    classDef model fill:#0984e3,stroke:#74b9ff,stroke-width:2px,color:#fff;
    classDef fusion fill:#6c5ce7,stroke:#a29bfe,stroke-width:2px,color:#fff;
    classDef out fill:#00b894,stroke:#55efc4,stroke-width:2px,color:#fff;
    
    class A,C data;
    class B,D,F model;
    class E fusion;
    class G out;

Loading

Components Breakdown:

1. CNN Spatial Encoder: Extracts high-level spatial meteorological features from input satellite grids/images.
2. Temporal Transformer Encoder: Leverages self-attention mechanisms to capture complex long-range dependencies in weather time-series, mitigating issues found in standard RNNs.
3. Attention Fusion Module: Calculates attention scores to dynamically weigh the importance of spatial features versus temporal historical trends.
4. Forecasting Network: A highly optimized recursive Multilayer Perceptron (MLP) generating autoregressive multi-step forecasts (Temperature, Humidity, Wind Speed, Precipitation).

---

Results & Performance

Extensive benchmarking against traditional architectures highlights the power of multimodal fusion:

Architecture	RMSE (↓)	R² Score (↑)	Performance Note
LSTM Only	0.426	0.335	Temporal Baseline
CNN + LSTM	0.382	0.541	Static Multi-modal
Sim2Real Hybrid	0.381	0.543	Dynamic Attention (SOTA)

The Hybrid model shows exceptional stability in predicting extreme weather spikes compared to baseline models. Refer to the results/ folder for comprehensive RMSE growth analyses, prediction residuals, and attention maps.

---

Getting Started

1. Prerequisites

• Python: 3.11 or higher.
• Compute: NVIDIA GPU (e.g., RTX 4050 or above) with CUDA support is highly recommended for accelerated training.

2. Installation

Clone the repository and install the required dependencies:

git clone https://github.com/Kri311/Multimodal-Weather-Forecasting-LSTM.git
cd Multimodal-Weather-Forecasting-LSTM

# Install via requirements.txt
pip install -r requirements.txt

3. Usage

Evaluation Pipeline (Benchmarking) Run the automated script to evaluate pre-trained models and generate all comparison plots (20+ detailed graphs).

python evaluate_advanced.py

Real-time Inference Deploy the model architecture for immediate 72-hour future forecasting.

python inference.py

Note:

The Image dataset has been taken from NASA [https://worldview.earthdata.nasa.gov/] and the numerical dataset has been taken from automated script from Open-Meteo [https://open-meteo.com/] over Amrita Vishwa Vidyapeetham, Coimbatore over a year from June 2024 - June 2025 and I have uploaded just the numerical dataset in the repo and eliminated the image dataset due to larger size. There still exist some issues with the model training and performance, which maybe I'll modify in the future