Skip to content

Commit 8fa62c0

Browse files
JohnPalmerJohnPalmer
committed
logs from failed cluster run
1 parent 3b2a32f commit 8fa62c0

6 files changed

+355
-0
lines changed

‎logs/generate_datasets_20250822_115809_dataset1.out‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
2+
R version 4.4.3 (2025-02-28) -- "Trophy Case"
3+
Copyright (C) 2025 The R Foundation for Statistical Computing
4+
Platform: x86_64-conda-linux-gnu
5+
6+
R is free software and comes with ABSOLUTELY NO WARRANTY.
7+
You are welcome to redistribute it under certain conditions.
8+
Type 'license()' or 'licence()' for distribution details.
9+
10+
R is a collaborative project with many contributors.
11+
Type 'contributors()' for more information and
12+
'citation()' on how to cite R or R packages in publications.
13+
14+
Type 'demo()' for some demos, 'help()' for on-line help, or
15+
'help.start()' for an HTML browser interface to help.
16+
Type 'q()' to quit R.
17+
18+
- Project '~/research/weather-data-collector-spain' loaded. [renv 1.1.4]
19+
- One or more packages recorded in the lockfile are not installed.
20+
- Use `renv::status()` for more details.
21+
> #!/usr/bin/env Rscript
22+
>
23+
> # aggregate_daily_station_data.R
24+
> # -------------------------------
25+
> # Purpose: Create daily aggregated weather data by station from hourly observations
26+
> #
27+
> # This script processes the hourly expanded weather data to create daily summaries
28+
> # by station. It combines historical daily data with aggregated current observations
29+
> # to provide a complete time series from 2013 to present.
30+
> #
31+
> # Output: Daily means, minimums, maximums, and totals by weather station
32+
> #
33+
> # Data Sources:
34+
> # 1. Historical daily data (2013 to T-4 days) from AEMET climatological endpoint
35+
> # 2. Current hourly data (T-4 days to present) aggregated to daily values
36+
> #
37+
> # Author: John Palmer
38+
> # Date: 2025-08-20
39+
>
40+
> rm(list=ls())
41+
>
42+
> # Dependencies ####
43+
> library(tidyverse)
44+
Error in library(tidyverse) : there is no package called ‘tidyverse’
45+
Execution halted

‎logs/get_forecast_data_20250822_115728.out‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
2+
R version 4.4.3 (2025-02-28) -- "Trophy Case"
3+
Copyright (C) 2025 The R Foundation for Statistical Computing
4+
Platform: x86_64-conda-linux-gnu
5+
6+
R is free software and comes with ABSOLUTELY NO WARRANTY.
7+
You are welcome to redistribute it under certain conditions.
8+
Type 'license()' or 'licence()' for distribution details.
9+
10+
R is a collaborative project with many contributors.
11+
Type 'contributors()' for more information and
12+
'citation()' on how to cite R or R packages in publications.
13+
14+
Type 'demo()' for some demos, 'help()' for on-line help, or
15+
'help.start()' for an HTML browser interface to help.
16+
Type 'q()' to quit R.
17+
18+
- Project '~/research/weather-data-collector-spain' loaded. [renv 1.1.4]
19+
- One or more packages recorded in the lockfile are not installed.
20+
- Use `renv::status()` for more details.
21+
> #!/usr/bin/env Rscript
22+
>
23+
> # Simple forecast data collection based on proven working patterns
24+
> library(jsonlite)
25+
Error in library(jsonlite) : there is no package called ‘jsonlite’
26+
Execution halted

‎logs/get_historical_data_20250822_115756.out‎

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
2+
R version 4.4.3 (2025-02-28) -- "Trophy Case"
3+
Copyright (C) 2025 The R Foundation for Statistical Computing
4+
Platform: x86_64-conda-linux-gnu
5+
6+
R is free software and comes with ABSOLUTELY NO WARRANTY.
7+
You are welcome to redistribute it under certain conditions.
8+
Type 'license()' or 'licence()' for distribution details.
9+
10+
R is a collaborative project with many contributors.
11+
Type 'contributors()' for more information and
12+
'citation()' on how to cite R or R packages in publications.
13+
14+
Type 'demo()' for some demos, 'help()' for on-line help, or
15+
'help.start()' for an HTML browser interface to help.
16+
Type 'q()' to quit R.
17+
18+
- Project '~/research/weather-data-collector-spain' loaded. [renv 1.1.4]
19+
- One or more packages recorded in the lockfile are not installed.
20+
- Use `renv::status()` for more details.
21+
> # get_historical_data.R
22+
> # ----------------------
23+
> # Purpose: Download and update historical daily weather data for Spain from the AEMET OpenData API.
24+
> #
25+
> # This script fetches historical daily climatological data using the 7 core variables
26+
> # that are compatible across current observations, historical data, and forecast endpoints.
27+
> #
28+
> # Core Variables (Standardized):
29+
> # - ta: Air temperature (°C) - from tmed
30+
> # - tamax: Maximum temperature (°C) - from tmax
31+
> # - tamin: Minimum temperature (°C) - from tmin
32+
> # - hr: Relative humidity (%) - from hrMedia
33+
> # - prec: Precipitation (mm) - from prec
34+
> # - vv: Wind speed (km/h) - from velmedia
35+
> # - pres: Atmospheric pressure (hPa) - from presMax
36+
> #
37+
> # Concurrency Control:
38+
> # - Set PREVENT_CONCURRENT_RUNS = TRUE to enable lockfile-based run prevention
39+
> # - Set PREVENT_CONCURRENT_RUNS = FALSE (default) to allow multiple concurrent runs
40+
> #
41+
> # Main Steps:
42+
> # 1. Load dependencies and API key.
43+
> # 2. Determine which dates are missing from the local dataset.
44+
> # 3. Download missing data in chunks, handling API rate limits and errors.
45+
> # 4. Append new data to the historical dataset.
46+
> #
47+
> # Usage:
48+
> # - Requires a valid API key in 'auth/keys.R' as 'my_api_key'.
49+
> # - Run as an R script. Output is written to 'data/output/daily_station_historical.csv.gz'.
50+
> #
51+
> # Dependencies: tidyverse, lubridate, data.table, curl, jsonlite
52+
> #
53+
> # Author: John Palmer
54+
> # Date: 2025-08-22 (Updated for 7-variable standardization)
55+
>
56+
> # Title ####
57+
> # For downloading and preparing historical weather data.
58+
>
59+
> rm(list=ls())
60+
>
61+
> ####Dependencies####
62+
> library(tidyverse)
63+
Error in library(tidyverse) : there is no package called ‘tidyverse’
64+
Execution halted

‎logs/get_latest_data_20250822_115742.out‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
2+
R version 4.4.3 (2025-02-28) -- "Trophy Case"
3+
Copyright (C) 2025 The R Foundation for Statistical Computing
4+
Platform: x86_64-conda-linux-gnu
5+
6+
R is free software and comes with ABSOLUTELY NO WARRANTY.
7+
You are welcome to redistribute it under certain conditions.
8+
Type 'license()' or 'licence()' for distribution details.
9+
10+
R is a collaborative project with many contributors.
11+
Type 'contributors()' for more information and
12+
'citation()' on how to cite R or R packages in publications.
13+
14+
Type 'demo()' for some demos, 'help()' for on-line help, or
15+
'help.start()' for an HTML browser interface to help.
16+
Type 'q()' to quit R.
17+
18+
- Project '~/research/weather-data-collector-spain' loaded. [renv 1.1.4]
19+
- One or more packages recorded in the lockfile are not installed.
20+
- Use `renv::status()` for more details.
21+
> # get_latest_data_expanded.R
22+
> # ----------------------
23+
> # Purpose: Download and update the latest observation data from AEMET stations across Spain.
24+
> #
25+
> # This script fetches weather observations from the AEMET OpenData API using the 7 core variables
26+
> # that are compatible across current observations, historical data, and forecast endpoints.
27+
> #
28+
> # Core Variables (Safe for all endpoints):
29+
> # - ta: Air temperature (°C)
30+
> # - tamax: Maximum temperature (°C)
31+
> # - tamin: Minimum temperature (°C)
32+
> # - hr: Relative humidity (%)
33+
> # - prec: Precipitation (mm)
34+
> # - vv: Wind speed (km/h)
35+
> # - pres: Atmospheric pressure (hPa)
36+
> #
37+
> # Main Steps:
38+
> # 1. Load dependencies and API key.
39+
> # 2. Define functions to request and process data from the AEMET API, with error handling and retries.
40+
> # 3. Download the latest data and reshape it for storage.
41+
> # 4. Append new data to the local CSV file, ensuring no duplicates.
42+
> #
43+
> # Usage:
44+
> # - Requires a valid API key in 'auth/keys.R' as 'my_api_key'.
45+
> # - Run as an R script. Output is written to 'data/spain_weather_expanded.csv.gz'.
46+
> #
47+
> # Dependencies: tidyverse, lubridate, curl, jsonlite, data.table, R.utils
48+
> #
49+
> # Author: John Palmer
50+
> # Date: 2025-08-20 (Updated for 7-variable expansion)
51+
>
52+
> # Title ####
53+
> # For downloading latest observation data from AEMET stations all over Spain. This needs to be run at least every 12 hours, but better to run it every 2 because of API limits, failures etc.
54+
>
55+
> rm(list=ls())
56+
>
57+
>
58+
> # Dependencies ####
59+
> library(tidyverse)
60+
Error in library(tidyverse) : there is no package called ‘tidyverse’
61+
Execution halted

‎logs/weather_collection_17296.err‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
Identity added: /home/j.palmer/.ssh/id_rsa (/home/j.palmer/.ssh/id_rsa)
2+
Identity added: /home/j.palmer/.ssh/id_rsa (/home/j.palmer/.ssh/id_rsa)
3+
4+
CondaError: Run 'conda init' before 'conda activate'
5+
6+
From github.com:Mosquito-Alert/weather-data-collector-spain
7+
* branch main -> FETCH_HEAD
8+
Identity added: /home/j.palmer/.ssh/id_rsa (/home/j.palmer/.ssh/id_rsa)
9+
10+
CondaError: Run 'conda init' before 'conda activate'
11+
12+
Identity added: /home/j.palmer/.ssh/id_rsa (/home/j.palmer/.ssh/id_rsa)
13+
14+
CondaError: Run 'conda init' before 'conda activate'
15+
16+
Identity added: /home/j.palmer/.ssh/id_rsa (/home/j.palmer/.ssh/id_rsa)
17+
18+
CondaError: Run 'conda init' before 'conda activate'
19+
20+
Identity added: /home/j.palmer/.ssh/id_rsa (/home/j.palmer/.ssh/id_rsa)
21+
22+
CondaError: Run 'conda init' before 'conda activate'
23+
24+
Identity added: /home/j.palmer/.ssh/id_rsa (/home/j.palmer/.ssh/id_rsa)
25+
26+
CondaError: Run 'conda init' before 'conda activate'
27+
28+
Identity added: /home/j.palmer/.ssh/id_rsa (/home/j.palmer/.ssh/id_rsa)
29+
30+
CondaError: Run 'conda init' before 'conda activate'
31+
32+
Identity added: /home/j.palmer/.ssh/id_rsa (/home/j.palmer/.ssh/id_rsa)
33+
34+
CondaError: Run 'conda init' before 'conda activate'
35+
36+
Identity added: /home/j.palmer/.ssh/id_rsa (/home/j.palmer/.ssh/id_rsa)
37+
38+
CondaError: Run 'conda init' before 'conda activate'
39+
40+
Identity added: /home/j.palmer/.ssh/id_rsa (/home/j.palmer/.ssh/id_rsa)
41+
42+
CondaError: Run 'conda init' before 'conda activate'
43+
44+
Identity added: /home/j.palmer/.ssh/id_rsa (/home/j.palmer/.ssh/id_rsa)
45+
46+
CondaError: Run 'conda init' before 'conda activate'
47+

‎logs/weather_collection_17296.out‎

Lines changed: 112 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,112 @@
1+
Agent pid 447003
2+
Agent pid 447011
3+
Status updated for weather-hourly: running
4+
Status file written to: /home/j.palmer/research/mosquito-alert-model-monitor/data/status/weather-hourly.json
5+
Monitor repo: /home/j.palmer/research/mosquito-alert-model-monitor
6+
✅ Status file created successfully
7+
File size: 482 bytes
8+
🔄 Triggering dashboard rebuild via git push...
9+
⚠️ Git push failed - dashboard may not update automatically
10+
# Bootstrapping renv 1.1.4 ---------------------------------------------------
11+
- Downloading renv ... OK
12+
- Installing renv ... OK
13+
14+
- One or more packages recorded in the lockfile are not installed.
15+
- Use `renv::status()` for more details.
16+
Already up to date.
17+
Starting weather data collection: Fri Aug 22 11:57:19 CEST 2025
18+
Agent pid 447196
19+
Status updated for weather-hourly: running
20+
Status file written to: /home/j.palmer/research/mosquito-alert-model-monitor/data/status/weather-hourly.json
21+
Monitor repo: /home/j.palmer/research/mosquito-alert-model-monitor
22+
✅ Status file created successfully
23+
File size: 486 bytes
24+
🔄 Triggering dashboard rebuild via git push...
25+
⚠️ Git push failed - dashboard may not update automatically
26+
Collecting municipal forecasts...
27+
Agent pid 447244
28+
Status updated for weather-forecast: running
29+
Status file written to: /home/j.palmer/research/mosquito-alert-model-monitor/data/status/weather-forecast.json
30+
Monitor repo: /home/j.palmer/research/mosquito-alert-model-monitor
31+
✅ Status file created successfully
32+
File size: 488 bytes
33+
🔄 Triggering dashboard rebuild via git push...
34+
⚠️ Git push failed - dashboard may not update automatically
35+
Agent pid 447331
36+
Status updated for weather-forecast: failed
37+
Status file written to: /home/j.palmer/research/mosquito-alert-model-monitor/data/status/weather-forecast.json
38+
Monitor repo: /home/j.palmer/research/mosquito-alert-model-monitor
39+
✅ Status file created successfully
40+
File size: 487 bytes
41+
🔄 Triggering dashboard rebuild via git push...
42+
⚠️ Git push failed - dashboard may not update automatically
43+
Collecting hourly observations...
44+
Agent pid 447370
45+
Status updated for weather-hourly: running
46+
Status file written to: /home/j.palmer/research/mosquito-alert-model-monitor/data/status/weather-hourly.json
47+
Monitor repo: /home/j.palmer/research/mosquito-alert-model-monitor
48+
✅ Status file created successfully
49+
File size: 485 bytes
50+
🔄 Triggering dashboard rebuild via git push...
51+
⚠️ Git push failed - dashboard may not update automatically
52+
Agent pid 447452
53+
Status updated for weather-hourly: failed
54+
Status file written to: /home/j.palmer/research/mosquito-alert-model-monitor/data/status/weather-hourly.json
55+
Monitor repo: /home/j.palmer/research/mosquito-alert-model-monitor
56+
✅ Status file created successfully
57+
File size: 485 bytes
58+
🔄 Triggering dashboard rebuild via git push...
59+
⚠️ Git push failed - dashboard may not update automatically
60+
Updating historical data...
61+
Agent pid 447491
62+
Status updated for weather-historical: running
63+
Status file written to: /home/j.palmer/research/mosquito-alert-model-monitor/data/status/weather-historical.json
64+
Monitor repo: /home/j.palmer/research/mosquito-alert-model-monitor
65+
✅ Status file created successfully
66+
File size: 490 bytes
67+
🔄 Triggering dashboard rebuild via git push...
68+
⚠️ Git push failed - dashboard may not update automatically
69+
Agent pid 447573
70+
Status updated for weather-historical: failed
71+
Status file written to: /home/j.palmer/research/mosquito-alert-model-monitor/data/status/weather-historical.json
72+
Monitor repo: /home/j.palmer/research/mosquito-alert-model-monitor
73+
✅ Status file created successfully
74+
File size: 488 bytes
75+
🔄 Triggering dashboard rebuild via git push...
76+
⚠️ Git push failed - dashboard may not update automatically
77+
Generating aggregated datasets...
78+
Agent pid 447612
79+
Status updated for weather-aggregation: running
80+
Status file written to: /home/j.palmer/research/mosquito-alert-model-monitor/data/status/weather-aggregation.json
81+
Monitor repo: /home/j.palmer/research/mosquito-alert-model-monitor
82+
✅ Status file created successfully
83+
File size: 491 bytes
84+
🔄 Triggering dashboard rebuild via git push...
85+
⚠️ Git push failed - dashboard may not update automatically
86+
=== GENERATING ALL REQUIRED DATASETS ===
87+
Started at: Fri Aug 22 11:58:09 CEST 2025
88+
Log files will be saved with prefix: logs/generate_datasets_20250822_115809
89+
90+
Checking for required input files...
91+
Input file check complete.
92+
93+
=== STEP 1/3: GENERATING DATASET 1 (Daily Station Data) ===
94+
Running aggregate_daily_station_data.R...
95+
Agent pid 447700
96+
Status updated for weather-aggregation: failed
97+
Status file written to: /home/j.palmer/research/mosquito-alert-model-monitor/data/status/weather-aggregation.json
98+
Monitor repo: /home/j.palmer/research/mosquito-alert-model-monitor
99+
✅ Status file created successfully
100+
File size: 491 bytes
101+
🔄 Triggering dashboard rebuild via git push...
102+
⚠️ Git push failed - dashboard may not update automatically
103+
❌ Dataset generation failed
104+
Weather data collection completed: Fri Aug 22 11:58:19 CEST 2025
105+
Agent pid 447740
106+
Status updated for weather-hourly: completed
107+
Status file written to: /home/j.palmer/research/mosquito-alert-model-monitor/data/status/weather-hourly.json
108+
Monitor repo: /home/j.palmer/research/mosquito-alert-model-monitor
109+
✅ Status file created successfully
110+
File size: 490 bytes
111+
🔄 Triggering dashboard rebuild via git push...
112+
⚠️ Git push failed - dashboard may not update automatically

0 commit comments

Comments
 (0)