Adds system for generating and updating data collection status on README.md

Author: JohnPalmer

README.md

@@ -2,95 +2,255 @@
 
 This repository provides scripts to download, update, and manage weather data from AEMET weather stations across Spain, producing three comprehensive datasets for analysis and research.
 
+
+## 📊 Current Data Collection Status
+
+*Last updated: 2025-08-25 21:23:49*
+
+### Dataset 1: Daily Station Data
+- **Records**: 2,250 station-days
+- **Stations**: 838 weather stations
+- **Coverage**: 2025-08-17 to 2025-08-25
+- **Data Quality**: Coverage analysis pending
+- **Latest File**: `daily_station_aggregated_2025-08-25.csv.gz` (0 MB)
+
+### Dataset 2: Municipal Daily Data  
+- **Records**: 2,001 municipality-days
+- **Municipalities**: 724 municipalities
+- **Historical Data**: 1,910 records
+- **Forecast Data**: 91 records (7 days coverage)
+- **Coverage**: 2025-08-17 to 2025-08-31
+- **Data Quality**: Coverage analysis pending
+- **Latest File**: `municipal_aggregated_2025-08-25.csv` (0.3 MB)
+
+### Dataset 3: Hourly Station Data
+- **Records**: 180,393 hourly observations
+- **Stations**: ~752 stations (sample estimate)
+- **Variables**: 7 meteorological measures
+- **Coverage**: 2025-08-25 to 2025-08-25
+- **Recent Activity**: Analysis pending observations (last 30 days)
+- **Archive Size**: 0.8 MB compressed
+
+### 🔄 Collection System Status
+- **Collection Method**: Hybrid system using `climaemet` package + custom API calls
+- **Performance**: ~5.4x faster than previous approach
+- **Schedule**: Daily collection at 2 AM via crontab
+- **Last Gap Analysis**: Not available
+
+### 📈 Data Growth Tracking
+| Dataset | Current Size | Growth Rate | Last Updated |
+|---------|-------------|-------------|--------------|
+| Station Daily | 0 MB | ~281 records/day | 2025-08-25 |
+| Municipal Data | 0.3 MB | ~250 records/day | 2025-08-25 |
+| Hourly Archive | 0.8 MB | ~TBD records/day | 2025-08-25 |
+
+---
+
+
 ## Three Output Datasets
 
-### 📊 Dataset 1: Daily Station Data (`daily_station_historical.csv.gz`)
-Daily aggregated weather data by station, combining:
-- Historical data from AEMET historical endpoint (2013+)
-- Recent data from current observations (last 4 days)
-- Variables: daily min/max/mean temperature, precipitation totals, etc.
+### Dataset 1: Daily Station Data
+**File**: `daily_station_aggregated_YYYY-MM-DD.csv.gz`
+
+Daily aggregated weather data by station:
+- Data sources: AEMET daily climatological endpoint + hourly observations aggregated to daily
+- Variables: daily min/max/mean temperature, precipitation, wind, humidity, pressure
+- Coverage: Active weather stations across Spain
+- Quality control: Temperature range validation, realistic value bounds
+
+### Dataset 2: Municipal Daily Data  
+**File**: `municipal_aggregated_YYYY-MM-DD.csv.gz`
+
+Daily weather data by municipality:
+- Data sources: Station data aggregated by municipality + 7-day municipal forecasts
+- Coverage: 8,129 Spanish municipalities
+- Temporal range: Historical station aggregates through 7-day forecasts
+- Source tracking: Distinguishes between station-derived data and forecast data
+
+### Dataset 3: Hourly Station Data
+**File**: `hourly_station_ongoing.csv.gz`
 
-### 🏘️ Dataset 2: Municipal Daily Data (`municipal_daily_combined.csv.gz`) 
-Daily weather data by municipality (8,129 Spanish municipalities), combining:
-- Station data aggregated to municipal level (historical + recent)
-- 7-day municipal forecasts from AEMET
-- Complete temporal coverage: historical through 7-day forecast
+Hourly observations from AEMET stations:
+- Data format: Long format (measure/value pairs) for 7 core variables
+- Update frequency: Daily collection with continuous archiving
+- Purpose: Building comprehensive historical hourly archive
 
-### ⏰ Dataset 3: Hourly Station Data (`hourly_station_observations.csv.gz`)
-Hourly observations from all AEMET stations:
-- Real-time collection building our own historical archive
-- Expanded variable set (7 safe variables)
-- Updated every 2 hours
+## Data Collection System
 
-## Data Collection Workflow
+### Collection Methods
+- **Station Daily Data**: Custom API calls to AEMET climatological endpoints
+- **Municipal Forecasts**: `climaemet` R package for robust API interaction
+- **Hourly Data**: Direct API calls to AEMET observational endpoints
+
+### Automation
+- **Schedule**: Daily collection via SLURM batch system
+- **Gap Detection**: Automated identification of missing data
+- **Gap Filling**: Weekly targeted collection for missing records
+- **Quality Control**: Automated validation of temperature ranges and data consistency
+
+### Data Processing Pipeline
 
 ```mermaid
 flowchart TD
-    A[AEMET API] --> B[Historical Endpoint]
-    A --> C[Current Observations]
+    A[AEMET API] --> B[Station Daily Endpoint]
+    A --> C[Hourly Observations] 
     A --> D[Municipal Forecasts]
     
-    B --> E[get_historical_data.R]
+    B --> E[get_station_daily_hybrid.R]
     C --> F[get_latest_data.R]
-    D --> G[get_forecast_data.R]
+    D --> G[get_forecast_data_hybrid.R]
     
-    E --> H[daily_station_historical.csv.gz]
-    F --> I[hourly_station_ongoing.csv.gz]
-    G --> J[Municipal Forecast Data]
+    E --> H[Station Daily Data]
+    F --> I[Hourly Archive]
+    G --> J[Municipal Forecasts]
     
-    H --> K[aggregate_municipal_daily.R]
-    J --> K
-    K --> L[daily_municipal_extended.csv.gz]
+    H --> K[aggregate_daily_station_data_hybrid.R]
+    I --> K
+    K --> L[Dataset 1: Daily Station Aggregated]
     
-    H --> M[Dataset 1: Daily Station]
-    L --> N[Dataset 2: Municipal Extended]
-    I --> O[Dataset 3: Hourly Station]
+    L --> M[aggregate_municipal_data_hybrid.R]
+    J --> M
+    M --> N[Dataset 2: Municipal Aggregated]
     
-    M --> P[Zenodo Publication]
-    N --> P
-    O --> P
+    I --> O[Dataset 3: Hourly Archive]
 ```
 
-## Dataset Temporal Coverage
+## File Structure
 
-```mermaid
-gantt
-    title Weather Data Temporal Coverage
-    dateFormat  YYYY-MM-DD
-    section Dataset 1 - Daily Station
-    Historical Records    :done, hist1, 2013-01-01, 2025-08-17
-    Recent Observations   :active, recent1, 2025-08-17, 2025-08-21
-    
-    section Dataset 2 - Municipal Extended  
-    Historical Period     :done, hist2, 2013-01-01, 2025-08-17
-    Recent Period         :active, recent2, 2025-08-17, 2025-08-21
-    Forecast Period       :forecast, 2025-08-21, 2025-08-28
-    
-    section Dataset 3 - Hourly Station
-    Accumulating Archive  :active, archive, 2025-08-01, 2025-08-21
+### Core Collection Scripts
+```
+code/
+├── get_station_daily_hybrid.R        # Station daily data collection
+├── get_forecast_data_hybrid.R        # Municipal forecast collection  
+├── get_latest_data.R                 # Hourly data collection
+├── collect_all_datasets_hybrid.R     # Coordinated collection of all datasets
+├── aggregate_daily_station_data_hybrid.R  # Station data aggregation
+└── aggregate_municipal_data_hybrid.R      # Municipal data aggregation
+```
+
+### Data Quality & Monitoring
+```
+code/
+├── check_data_gaps.R                 # Gap detection and analysis
+├── fill_data_gaps.R                  # Targeted gap filling
+├── generate_data_summary.R           # Dataset statistics generation
+└── update_readme_with_summary.R      # Automated documentation updates
+```
+
+### SLURM Integration
+```
+├── update_weather_hybrid.sh          # Main collection job
+└── CRONTAB_LINES_TO_ADD.txt         # Scheduling configuration
+```
+
+## Setup and Usage
+
+### Prerequisites
+```bash
+# Load required modules (HPC environment)
+module load GDAL/3.10.0-foss-2024a
+module load R/4.4.2-gfbf-2024a
+
+# Install required R packages
+Rscript -e "install.packages(c('climaemet', 'tidyverse', 'data.table', 'lubridate'))"
+```
+
+### Manual Collection
+```bash
+# Collect all three datasets
+sbatch update_weather_hybrid.sh
+
+# Or run individual components
+Rscript code/get_forecast_data_hybrid.R      # Municipal forecasts
+Rscript code/get_station_daily_hybrid.R      # Station daily data  
+Rscript code/get_latest_data.R               # Hourly data
 ```
 
-## Monitoring & Dashboard Integration
+### Automated Schedule
+Add these lines to your crontab:
+```bash
+# Daily collection (2 AM)
+0 2 * * * cd /path/to/weather-data-collector-spain && sbatch update_weather_hybrid.sh
+
+# Daily status update (6 AM)  
+0 6 * * * cd /path/to/weather-data-collector-spain && Rscript code/update_readme_with_summary.R
+
+# Weekly gap filling (Sunday 1 AM)
+0 1 * * 0 cd /path/to/weather-data-collector-spain && Rscript code/fill_data_gaps.R
+```
+
+## Data Quality
+
+### Coverage and Success Rates
+- **Station Daily**: Typical success rates of 30-50% per collection run (normal for AEMET API)
+- **Municipal Forecasts**: 65-95% success rates with automatic retry logic
+- **Hourly Data**: >99% success rate for active stations
 
-### 🖥️ **Real-time Monitoring**
-This project integrates with the [mosquito-alert-model-monitor](https://github.com/Mosquito-Alert/mosquito-alert-model-monitor) dashboard for real-time job monitoring.
+### Quality Control Measures
+- **Temperature validation**: Range checks (min ≤ mean ≤ max, realistic bounds)
+- **Duplicate handling**: Automatic deduplication with source prioritization
+- **Gap tracking**: Systematic identification and filling of missing data
+- **Source attribution**: Clear distinction between observed vs forecast data
 
-**Monitored Jobs:**
-- `weather-forecast`: Municipal forecasts (every 6 hours) - **CRITICAL PRIORITY**
-- `weather-hourly`: Station observations (every 2 hours) - **MEDIUM PRIORITY**  
-- `weather-historical`: Historical data updates (daily) - **LOW PRIORITY**
-- `municipal-forecast-priority`: Immediate municipal data - **CRITICAL PRIORITY**
+## Data Access
 
-**Setup Dashboard Monitoring:**
+### Output Location
+All datasets are saved in `data/output/` with date-stamped filenames:
+- `daily_station_aggregated_YYYY-MM-DD.csv[.gz]`
+- `municipal_aggregated_YYYY-MM-DD.csv[.gz]`  
+- `hourly_station_ongoing.csv.gz`
+
+### File Formats
+- **CSV format**: Compatible with R, Python, and standard analysis tools
+- **Compressed versions**: `.gz` files for efficient storage
+- **Consistent schemas**: Standardized column names across collection runs
+
+## Monitoring and Maintenance
+
+### Gap Analysis
+```bash
+# Check for missing data
+Rscript code/check_data_gaps.R
+
+# Fill identified gaps
+Rscript code/fill_data_gaps.R
+```
+
+### Data Statistics
 ```bash
-# Test integration
-./scripts/test_dashboard_integration.sh
+# Generate current dataset summary
+Rscript code/generate_data_summary.R
 
-# Check dashboard at: ~/research/mosquito-alert-model-monitor/docs/index.html
+# Update README with latest statistics  
+Rscript code/update_readme_with_summary.R
 ```
 
-**Status Reporting:**
-All SLURM scripts automatically report job status, progress, and resource usage to the monitoring dashboard.
+### Error Handling
+- **API Rate Limits**: Automatic detection and waiting
+- **SSL Connection Issues**: Built-in retry logic in `climaemet` package
+- **Server Errors**: Exponential backoff for temporary failures
+- **Missing Data**: Systematic gap detection and targeted re-collection
+
+## Technical Details
+
+### API Integration
+- **AEMET OpenData API**: Primary data source requiring valid API key
+- **Rate Limiting**: Respectful API usage with automatic throttling
+- **Error Recovery**: Robust handling of temporary failures and connection issues
+
+### Dependencies
+- **R Packages**: `climaemet`, `tidyverse`, `data.table`, `lubridate`, `httr`, `jsonlite`
+- **System Requirements**: GDAL/3.10.0, R/4.4.2
+- **Environment**: HPC cluster with SLURM job scheduler
+
+### Performance
+- **Collection Time**: Typically 2-4 hours for complete daily collection
+- **Resource Usage**: 8GB RAM, single CPU core sufficient
+- **Storage Growth**: Approximately 50-100MB per day across all datasets
+
+---
+
+*This system provides reliable, automated collection of comprehensive weather data for Spain with built-in quality control and gap management.*
 
 ## Features
 - **Real-time Observations**: Fetches current hourly weather from all AEMET stations