Add weather integration documentation and local test script for dashboard monitoring

Author: JohnPalmer

docs/weather-integration.md

@@ -0,0 +1,87 @@
+# Weather Data Collector - Dashboard Integration
+
+## ✅ Integration Complete
+
+The weather-data-collector-spain project is now fully integrated with the mosquito-alert-model-monitor dashboard.
+
+## 🎯 What's Monitored
+
+The dashboard now tracks these weather collection jobs:
+
+| Job Name | Purpose | Frequency | Priority |
+|----------|---------|-----------|----------|
+| `weather-forecast` | Municipal forecasts (8,129 municipalities) | Every 6 hours | CRITICAL |
+| `weather-hourly` | Station observations (all AEMET stations) | Every 2 hours | MEDIUM |
+| `weather-historical` | Historical data backfill | Daily | LOW |
+| `municipal-forecast-priority` | Immediate municipal data | Every 6 hours | CRITICAL |
+
+## 🔧 Key Changes Made
+
+### 1. Enhanced Status Reporting
+- **Updated**: `scripts/update_weather_status.sh` now includes git push functionality
+- **Result**: Job status changes automatically trigger dashboard rebuilds
+
+### 2. Job Script Integration
+- **Modified**: `update_weather.sh` with proper status reporting
+- **Modified**: `priority_municipal_data.sh` with status integration
+- **Result**: All weather jobs report progress, duration, and completion status
+
+### 3. Initial Status Files
+- **Created**: Status files for all 4 weather jobs in the dashboard
+- **Result**: Dashboard shows weather jobs even when not actively running
+
+## 🚀 Testing on Cluster
+
+When you're ready to test on the HPC cluster:
+
+### 1. Quick Integration Test
+```bash
+cd ~/research/weather-data-collector-spain
+./scripts/test_dashboard_integration.sh
+```
+
+### 2. Run a Real Weather Job
+```bash
+# Test the priority municipal forecast (short runtime)
+sbatch priority_municipal_data.sh
+
+# Monitor dashboard for live updates
+```
+
+### 3. Check Dashboard
+- **Local**: `~/research/mosquito-alert-model-monitor/docs/index.html`
+- **GitHub Pages**: Will auto-update within 2-3 minutes of job status changes
+
+## 📊 Dashboard Features for Weather Jobs
+
+- **Real-time Status**: Running, completed, failed, waiting
+- **Progress Tracking**: 0-100% completion
+- **Duration Monitoring**: Job execution time
+- **Resource Usage**: CPU and memory utilization
+- **Next Run Schedule**: Automatic calculation based on job type
+- **Error Alerts**: Failed jobs highlighted in dashboard
+
+## 🔄 How It Works
+
+1. **Job starts** → Updates status to "running"
+2. **Job progresses** → Periodic status updates with progress %
+3. **Job completes/fails** → Final status update
+4. **Status script** → Automatically commits and pushes to git
+5. **GitHub Actions** → Rebuilds and deploys dashboard
+6. **Dashboard** → Shows updated status within 2-3 minutes
+
+## ⚠️ Requirements on Cluster
+
+- SSH keys configured for git push
+- `mosquito-alert-monitor` conda environment activated
+- Access to `~/research/mosquito-alert-model-monitor` directory
+
+## 🎉 Benefits
+
+- **Real-time Monitoring**: See job status without logging into cluster
+- **Historical Tracking**: Dashboard maintains job history
+- **Automated Alerts**: Failed jobs are highlighted
+- **Resource Monitoring**: Track cluster resource usage
+- **Schedule Awareness**: See when jobs are expected to run next
+
+Your weather data collection pipeline is now fully visible and monitored through the dashboard!

scripts/test_weather_integration_local.sh

@@ -0,0 +1,58 @@
+#!/bin/bash
+
+# test_weather_integration_local.sh
+# --------------------------------
+# Local test to verify weather project integration with dashboard
+# This simulates the weather job status updates without running actual collection
+
+echo "=== Testing Weather Data Collector Integration (Local) ==="
+
+# Configuration
+MONITOR_REPO="/Users/palmer/research/mosquito-alert-model-monitor"
+STATUS_SCRIPT="$MONITOR_REPO/scripts/update_job_status_and_push.sh"
+
+cd "$MONITOR_REPO"
+
+echo "✅ Testing weather job status updates..."
+
+# Simulate different weather job statuses
+echo "📡 Simulating weather-forecast job..."
+$STATUS_SCRIPT "weather-forecast" "running" 120 45
+
+sleep 2
+
+echo "🌡️  Simulating weather-hourly job..."
+$STATUS_SCRIPT "weather-hourly" "completed" 300 100
+
+sleep 2
+
+echo "📚 Simulating weather-historical job..."
+$STATUS_SCRIPT "weather-historical" "running" 180 75
+
+sleep 2
+
+echo "🏘️  Simulating municipal-forecast-priority job..."
+$STATUS_SCRIPT "municipal-forecast-priority" "completed" 90 100
+
+echo ""
+echo "✅ Weather job status files created:"
+ls -la data/status/weather-*.json data/status/municipal-*.json 2>/dev/null || echo "No weather status files found"
+
+echo ""
+echo "📊 Sample status file content:"
+if [ -f "data/status/weather-forecast.json" ]; then
+    echo "--- weather-forecast.json ---"
+    cat data/status/weather-forecast.json | head -10
+else
+    echo "⚠️  weather-forecast.json not found"
+fi
+
+echo ""
+echo "🎯 Integration test complete!"
+echo "📈 Check your dashboard at: file://$MONITOR_REPO/docs/index.html"
+echo "🔄 If GitHub Actions is set up, the dashboard will rebuild automatically"
+echo ""
+echo "🚀 When you test on the cluster:"
+echo "   1. Run: cd ~/research/weather-data-collector-spain"
+echo "   2. Test: ./scripts/test_dashboard_integration.sh"
+echo "   3. Monitor: Check dashboard for live updates"