Skip to content

Commit 94583fa

Browse files
JohnPalmerJohnPalmer
committed
Add dashboard integration documentation and enhance job status reporting scripts
1 parent 7c684b3 commit 94583fa

File tree

5 files changed

+305
-15
lines changed

5 files changed

+305
-15
lines changed

‎DASHBOARD_INTEGRATION.md‎

Lines changed: 172 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,172 @@
1+
# Weather Data Collector - Dashboard Integration
2+
3+
## Overview
4+
5+
The weather-data-collector-spain project is now fully integrated with the mosquito-alert-model-monitor dashboard for real-time job monitoring and status tracking.
6+
7+
## Monitored Jobs
8+
9+
The dashboard tracks four critical weather data collection jobs:
10+
11+
### 🌦️ **weather-forecast** (CRITICAL PRIORITY)
12+
- **Purpose**: Municipal forecast collection for 8,129 Spanish municipalities
13+
- **Frequency**: Every 6 hours
14+
- **Script**: `priority_municipal_forecast.sh`
15+
- **Data Output**: 7-day municipal forecasts
16+
- **Critical for**: Model predictions requiring immediate forecast data
17+
18+
### ⏰ **weather-hourly** (MEDIUM PRIORITY)
19+
- **Purpose**: Hourly station observations from all AEMET stations
20+
- **Frequency**: Every 2 hours
21+
- **Script**: `update_weather.sh`
22+
- **Data Output**: Real-time weather observations
23+
- **Critical for**: Current conditions and short-term analysis
24+
25+
### 📊 **weather-historical** (LOW PRIORITY)
26+
- **Purpose**: Daily historical weather data backfill
27+
- **Frequency**: Daily (typically overnight)
28+
- **Script**: `update_historical_weather.sh`
29+
- **Data Output**: Historical daily climatology
30+
- **Critical for**: Long-term trend analysis and model training
31+
32+
### 🚀 **municipal-forecast-priority** (CRITICAL PRIORITY)
33+
- **Purpose**: Immediate municipal data generation for model deployment
34+
- **Frequency**: On-demand/immediate needs
35+
- **Script**: `priority_municipal_data.sh`
36+
- **Data Output**: Complete municipal dataset with forecasts
37+
- **Critical for**: Urgent model deployment needs
38+
39+
## How It Works
40+
41+
### 1. **Status Reporting**
42+
Each weather collection script uses `scripts/update_weather_status.sh` to report:
43+
- Job status (running, completed, failed, waiting)
44+
- Execution duration and progress percentage
45+
- Resource usage (CPU, memory)
46+
- Next scheduled run time
47+
- Detailed configuration and metadata
48+
49+
### 2. **Automatic Dashboard Updates**
50+
When jobs update their status:
51+
1. Status JSON file is updated in the monitor repository
52+
2. Changes are automatically committed to git
53+
3. Git push triggers GitHub Actions workflow
54+
4. Dashboard is rebuilt and deployed within 2-3 minutes
55+
5. Live dashboard shows updated job statuses
56+
57+
### 3. **Status File Format**
58+
Each job creates a JSON status file with this structure:
59+
```json
60+
{
61+
"job_name": "weather-forecast",
62+
"status": "completed",
63+
"last_updated": "2025-08-22T14:30:00Z",
64+
"start_time": "2025-08-22T14:25:00Z",
65+
"duration": 300,
66+
"progress": 100,
67+
"cpu_usage": 45.2,
68+
"memory_usage": 1024,
69+
"next_scheduled_run": "2025-08-22T20:30:00Z",
70+
"config": {
71+
"project_type": "weather_data_collection",
72+
"data_source": "AEMET OpenData API",
73+
"collection_scope": "Municipal forecasts",
74+
"frequency": "Every 6 hours",
75+
"priority": "CRITICAL"
76+
}
77+
}
78+
```
79+
80+
## Testing Integration
81+
82+
### Run Integration Test
83+
```bash
84+
cd ~/research/weather-data-collector-spain
85+
./test_integration.sh
86+
```
87+
88+
### Manual Status Update
89+
```bash
90+
./scripts/update_weather_status.sh "weather-forecast" "running" 120 75
91+
```
92+
93+
### View Dashboard
94+
- **Local**: `~/research/mosquito-alert-model-monitor/docs/index.html`
95+
- **Live**: [GitHub Pages URL when deployed]
96+
97+
## HPC Cluster Usage
98+
99+
### SLURM Job Examples
100+
```bash
101+
# Submit priority municipal forecast job
102+
sbatch priority_municipal_forecast.sh
103+
104+
# Submit regular weather update job
105+
sbatch update_weather.sh
106+
107+
# Submit historical data update
108+
sbatch update_historical_weather.sh
109+
```
110+
111+
### Cron Job Examples
112+
```bash
113+
# Every 6 hours: Municipal forecasts (critical)
114+
0 */6 * * * cd ~/research/weather-data-collector-spain && sbatch priority_municipal_forecast.sh
115+
116+
# Every 2 hours: Station observations
117+
0 */2 * * * cd ~/research/weather-data-collector-spain && sbatch update_weather.sh
118+
119+
# Daily: Historical data update
120+
0 6 * * * cd ~/research/weather-data-collector-spain && sbatch update_historical_weather.sh
121+
```
122+
123+
## Dashboard Benefits
124+
125+
1. **Real-time Monitoring**: See job status, progress, and performance metrics
126+
2. **Failure Detection**: Immediate notification when jobs fail
127+
3. **Resource Tracking**: Monitor CPU, memory usage across jobs
128+
4. **Schedule Visibility**: Know when jobs are due to run next
129+
5. **Historical Trends**: Track job performance over time
130+
6. **Remote Access**: Monitor jobs from anywhere via GitHub Pages
131+
132+
## Troubleshooting
133+
134+
### Common Issues
135+
136+
**Status not updating:**
137+
- Check git push permissions from HPC cluster
138+
- Verify monitor repository path in scripts
139+
- Ensure conda environment is activated
140+
141+
**Dashboard not rendering:**
142+
- Check GitHub Actions workflow status
143+
- Verify Quarto and R dependencies
144+
- Check for JSON syntax errors in status files
145+
146+
**Jobs not appearing:**
147+
- Ensure status files are created in correct directory
148+
- Check file permissions
149+
- Verify JSON structure matches expected format
150+
151+
### Debug Commands
152+
```bash
153+
# Check if status files exist
154+
ls -la ~/research/mosquito-alert-model-monitor/data/status/
155+
156+
# Validate JSON syntax
157+
jq . ~/research/mosquito-alert-model-monitor/data/status/weather-forecast.json
158+
159+
# Test git operations
160+
cd ~/research/mosquito-alert-model-monitor
161+
git status
162+
git log --oneline -5
163+
```
164+
165+
## Integration Maintenance
166+
167+
- **Status scripts** are automatically updated with each job run
168+
- **Dashboard rebuilds** happen automatically via GitHub Actions
169+
- **No manual intervention** required for normal operations
170+
- **Monitor git repository** should be kept up to date with latest changes
171+
172+
This integration provides comprehensive monitoring for all weather data collection activities, ensuring reliable operation and immediate visibility into job status and performance.

‎priority_municipal_data.sh‎

Lines changed: 30 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -52,22 +52,50 @@ mkdir -p logs
5252
# Create output directory
5353
mkdir -p data/output
5454

55+
# Initialize status reporting
56+
JOB_NAME="municipal-forecast-priority"
57+
STATUS_SCRIPT="./scripts/update_weather_status.sh"
58+
START_TIME=$(date +%s)
59+
60+
# Report job started
61+
$STATUS_SCRIPT "$JOB_NAME" "running" 0 5
62+
5563
# Activate renv
5664
R --slave --no-restore --file=- <<EOF
5765
renv::activate()
5866
EOF
5967

6068
echo "Starting priority municipal data generation: $(date)"
69+
$STATUS_SCRIPT "$JOB_NAME" "running" $(($(date +%s) - START_TIME)) 10
6170

6271
# Get forecasts first (immediate availability)
6372
echo "Collecting municipal forecasts for immediate model use..."
73+
$STATUS_SCRIPT "$JOB_NAME" "running" $(($(date +%s) - START_TIME)) 25
6474
R CMD BATCH --no-save --no-restore code/get_forecast_data.R logs/priority_forecast_$(date +%Y%m%d_%H%M%S).out
6575

76+
if [ $? -eq 0 ]; then
77+
$STATUS_SCRIPT "$JOB_NAME" "running" $(($(date +%s) - START_TIME)) 60
78+
echo "✅ Municipal forecasts collected successfully"
79+
else
80+
$STATUS_SCRIPT "$JOB_NAME" "failed" $(($(date +%s) - START_TIME)) 25
81+
echo "❌ Municipal forecast collection failed"
82+
exit 1
83+
fi
84+
6685
# Generate backwards municipal data
6786
echo "Generating municipal data backwards from present..."
87+
$STATUS_SCRIPT "$JOB_NAME" "running" $(($(date +%s) - START_TIME)) 75
6888
R CMD BATCH --no-save --no-restore code/generate_municipal_priority.R logs/priority_municipal_$(date +%Y%m%d_%H%M%S).out
6989

70-
echo "Priority municipal data generation completed: $(date)"
71-
echo "Models can now use: data/output/daily_municipal_extended.csv.gz"
90+
if [ $? -eq 0 ]; then
91+
FINAL_DURATION=$(($(date +%s) - START_TIME))
92+
$STATUS_SCRIPT "$JOB_NAME" "completed" $FINAL_DURATION 100
93+
echo "✅ Priority municipal data generation completed: $(date)"
94+
echo "Models can now use: data/output/daily_municipal_extended.csv.gz"
95+
else
96+
$STATUS_SCRIPT "$JOB_NAME" "failed" $(($(date +%s) - START_TIME)) 75
97+
echo "❌ Municipal data generation failed"
98+
exit 1
99+
fi
72100

73101
# Submit: sbatch priority_municipal_data.sh

‎scripts/update_weather_status.sh‎

Lines changed: 28 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -24,8 +24,8 @@ conda activate mosquito-alert-monitor
2424

2525
# scripts/update_weather_status.sh
2626
# ---------------------------------
27-
# Integration script for mosquito-alert-model-monitor dashboard
28-
# Reports weather data collection job status
27+
# Enhanced integration script for mosquito-alert-model-monitor dashboard
28+
# Reports weather data collection job status AND triggers dashboard rebuild via git push
2929

3030
# Configuration
3131
MONITOR_REPO_PATH="$HOME/research/mosquito-alert-model-monitor" # Path to monitor dashboard
@@ -129,12 +129,32 @@ echo "Monitor repo: $MONITOR_REPO_PATH"
129129
if [ -f "$STATUS_DIR/${JOB_NAME}.json" ]; then
130130
echo "✅ Status file created successfully"
131131
echo "File size: $(stat -c%s "$STATUS_DIR/${JOB_NAME}.json" 2>/dev/null || echo "unknown") bytes"
132+
133+
# Push changes to git to trigger dashboard rebuild
134+
cd "$MONITOR_REPO_PATH"
135+
if [ -d ".git" ]; then
136+
echo "🔄 Triggering dashboard rebuild via git push..."
137+
138+
# Add the status file
139+
git add "data/status/${JOB_NAME}.json"
140+
141+
# Create commit message with job details
142+
COMMIT_MSG="Update ${JOB_NAME} status: ${STATUS} ($(date '+%Y-%m-%d %H:%M:%S'))"
143+
144+
# Commit changes
145+
if git commit -m "$COMMIT_MSG" > /dev/null 2>&1; then
146+
# Push to trigger GitHub Actions
147+
if git push origin main > /dev/null 2>&1; then
148+
echo "✅ Dashboard rebuild triggered - will be live in ~2-3 minutes"
149+
else
150+
echo "⚠️ Git push failed - dashboard may not update automatically"
151+
fi
152+
else
153+
echo "ℹ️ No changes to commit (status unchanged)"
154+
fi
155+
else
156+
echo "⚠️ Monitor repo is not a git repository - no automatic rebuild"
157+
fi
132158
else
133159
echo "❌ Failed to create status file"
134160
fi
135-
136-
# Optional: Trigger dashboard update if running locally
137-
if [ -f "$MONITOR_REPO_PATH/index.qmd" ] && [ "$AUTO_RENDER" = "true" ]; then
138-
cd "$MONITOR_REPO_PATH"
139-
quarto render index.qmd >/dev/null 2>&1 &
140-
fi

‎test_integration.sh‎

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
#!/bin/bash
2+
3+
# Test script to verify weather-data-collector-spain integration with dashboard monitoring
4+
# Run this from the weather-data-collector-spain directory
5+
6+
echo "=== Testing Weather Data Collector Dashboard Integration ==="
7+
8+
# Check if we're in the right directory
9+
if [ ! -f "README.md" ] || [ ! -d "scripts" ]; then
10+
echo "❌ Please run this script from the weather-data-collector-spain directory"
11+
exit 1
12+
fi
13+
14+
# Check if monitor dashboard exists
15+
MONITOR_REPO="$HOME/research/mosquito-alert-model-monitor"
16+
if [ ! -d "$MONITOR_REPO" ]; then
17+
echo "❌ Monitor dashboard not found at: $MONITOR_REPO"
18+
echo "Please ensure the mosquito-alert-model-monitor repo is cloned at the expected location"
19+
exit 1
20+
fi
21+
22+
echo "✅ Found monitor dashboard at: $MONITOR_REPO"
23+
24+
# Test status update script
25+
echo ""
26+
echo "Testing status update functionality..."
27+
28+
# Test each weather job type
29+
./scripts/update_weather_status.sh "weather-forecast" "running" 120 45
30+
echo "✅ Updated weather-forecast status"
31+
32+
./scripts/update_weather_status.sh "weather-hourly" "completed" 300 100
33+
echo "✅ Updated weather-hourly status"
34+
35+
./scripts/update_weather_status.sh "weather-historical" "running" 180 25
36+
echo "✅ Updated weather-historical status"
37+
38+
./scripts/update_weather_status.sh "municipal-forecast-priority" "completed" 90 100
39+
echo "✅ Updated municipal-forecast-priority status"
40+
41+
# Check if status files were created
42+
echo ""
43+
echo "Verifying status files..."
44+
for job in weather-forecast weather-hourly weather-historical municipal-forecast-priority; do
45+
if [ -f "$MONITOR_REPO/data/status/${job}.json" ]; then
46+
echo "✅ ${job}.json created successfully"
47+
else
48+
echo "❌ ${job}.json not found"
49+
fi
50+
done
51+
52+
# Display sample status file
53+
echo ""
54+
echo "Sample status file content:"
55+
echo "=========================="
56+
cat "$MONITOR_REPO/data/status/weather-forecast.json" | jq .
57+
58+
echo ""
59+
echo "=== Integration Test Complete ==="
60+
echo ""
61+
echo "🎯 Weather data collection jobs are now integrated with the monitoring dashboard"
62+
echo "📊 Dashboard location: $MONITOR_REPO/docs/index.html"
63+
echo "🔄 Status updates will automatically trigger dashboard rebuilds via git push"
64+
echo ""
65+
echo "Next steps:"
66+
echo "1. Run a weather collection job to see live updates"
67+
echo "2. Check the dashboard shows the weather jobs with their statuses"
68+
echo "3. Deploy the dashboard to GitHub Pages for remote monitoring"

‎update_weather.sh‎

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -45,12 +45,12 @@ mkdir -p logs
4545
mkdir -p data/output
4646

4747
# Initialize status reporting
48-
JOB_NAME="weather-data-collector"
48+
JOB_NAME="weather-hourly"
4949
STATUS_SCRIPT="./scripts/update_weather_status.sh"
5050
START_TIME=$(date +%s)
5151

5252
# Report job started
53-
$STATUS_SCRIPT "$JOB_NAME" "running" 0 0
53+
$STATUS_SCRIPT "$JOB_NAME" "running" 0 5
5454

5555
# Initialize renv if first run
5656
if [ ! -f "renv.lock" ]; then
@@ -79,7 +79,7 @@ $STATUS_SCRIPT "$JOB_NAME" "running" $(($(date +%s) - START_TIME)) 20
7979

8080
# Priority 1: Municipal forecasts (immediate model needs)
8181
echo "Collecting municipal forecasts..."
82-
$STATUS_SCRIPT "weather-forecast" "running" $(($(date +%s) - START_TIME)) 30
82+
$STATUS_SCRIPT "weather-forecast" "running" $(($(date +%s) - START_TIME)) 25
8383
R CMD BATCH --no-save --no-restore code/get_forecast_data.R logs/get_forecast_data_$(date +%Y%m%d_%H%M%S).out
8484

8585
if [ $? -eq 0 ]; then
@@ -90,12 +90,14 @@ fi
9090

9191
# Priority 2: Hourly observations
9292
echo "Collecting hourly observations..."
93-
$STATUS_SCRIPT "weather-hourly" "running" $(($(date +%s) - START_TIME)) 50
93+
$STATUS_SCRIPT "$JOB_NAME" "running" $(($(date +%s) - START_TIME)) 50
9494
R CMD BATCH --no-save --no-restore code/get_latest_data.R logs/get_latest_data_$(date +%Y%m%d_%H%M%S).out
9595

9696
if [ $? -eq 0 ]; then
97-
$STATUS_SCRIPT "weather-hourly" "completed" $(($(date +%s) - START_TIME)) 100
97+
$STATUS_SCRIPT "$JOB_NAME" "completed" $(($(date +%s) - START_TIME)) 100
9898
else
99+
$STATUS_SCRIPT "$JOB_NAME" "failed" $(($(date +%s) - START_TIME)) 50
100+
fi
99101
$STATUS_SCRIPT "weather-hourly" "failed" $(($(date +%s) - START_TIME)) 50
100102
fi
101103

0 commit comments

Comments
 (0)