Skip to content

Commit d9354fc

Browse files
JohnPalmerJohnPalmer
committed
logs from cluster
1 parent 74d93ba commit d9354fc

6 files changed

+723
-0
lines changed

‎logs/generate_datasets_20250822_122808_dataset1.out‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
2+
R version 4.4.3 (2025-02-28) -- "Trophy Case"
3+
Copyright (C) 2025 The R Foundation for Statistical Computing
4+
Platform: x86_64-conda-linux-gnu
5+
6+
R is free software and comes with ABSOLUTELY NO WARRANTY.
7+
You are welcome to redistribute it under certain conditions.
8+
Type 'license()' or 'licence()' for distribution details.
9+
10+
R is a collaborative project with many contributors.
11+
Type 'contributors()' for more information and
12+
'citation()' on how to cite R or R packages in publications.
13+
14+
Type 'demo()' for some demos, 'help()' for on-line help, or
15+
'help.start()' for an HTML browser interface to help.
16+
Type 'q()' to quit R.
17+
18+
- Project '~/research/weather-data-collector-spain' loaded. [renv 1.1.4]
19+
- One or more packages recorded in the lockfile are not installed.
20+
- Use `renv::status()` for more details.
21+
> #!/usr/bin/env Rscript
22+
>
23+
> # aggregate_daily_station_data.R
24+
> # -------------------------------
25+
> # Purpose: Create daily aggregated weather data by station from hourly observations
26+
> #
27+
> # This script processes the hourly expanded weather data to create daily summaries
28+
> # by station. It combines historical daily data with aggregated current observations
29+
> # to provide a complete time series from 2013 to present.
30+
> #
31+
> # Output: Daily means, minimums, maximums, and totals by weather station
32+
> #
33+
> # Data Sources:
34+
> # 1. Historical daily data (2013 to T-4 days) from AEMET climatological endpoint
35+
> # 2. Current hourly data (T-4 days to present) aggregated to daily values
36+
> #
37+
> # Author: John Palmer
38+
> # Date: 2025-08-20
39+
>
40+
> rm(list=ls())
41+
>
42+
> # Dependencies ####
43+
> library(tidyverse)
44+
Error in library(tidyverse) : there is no package called ‘tidyverse’
45+
Execution halted

‎logs/get_forecast_data_20250822_122724.out‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
2+
R version 4.4.3 (2025-02-28) -- "Trophy Case"
3+
Copyright (C) 2025 The R Foundation for Statistical Computing
4+
Platform: x86_64-conda-linux-gnu
5+
6+
R is free software and comes with ABSOLUTELY NO WARRANTY.
7+
You are welcome to redistribute it under certain conditions.
8+
Type 'license()' or 'licence()' for distribution details.
9+
10+
R is a collaborative project with many contributors.
11+
Type 'contributors()' for more information and
12+
'citation()' on how to cite R or R packages in publications.
13+
14+
Type 'demo()' for some demos, 'help()' for on-line help, or
15+
'help.start()' for an HTML browser interface to help.
16+
Type 'q()' to quit R.
17+
18+
- Project '~/research/weather-data-collector-spain' loaded. [renv 1.1.4]
19+
- One or more packages recorded in the lockfile are not installed.
20+
- Use `renv::status()` for more details.
21+
> #!/usr/bin/env Rscript
22+
>
23+
> # Simple forecast data collection based on proven working patterns
24+
> library(jsonlite)
25+
Error in library(jsonlite) : there is no package called ‘jsonlite’
26+
Execution halted

‎logs/get_historical_data_20250822_122754.out‎

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
2+
R version 4.4.3 (2025-02-28) -- "Trophy Case"
3+
Copyright (C) 2025 The R Foundation for Statistical Computing
4+
Platform: x86_64-conda-linux-gnu
5+
6+
R is free software and comes with ABSOLUTELY NO WARRANTY.
7+
You are welcome to redistribute it under certain conditions.
8+
Type 'license()' or 'licence()' for distribution details.
9+
10+
R is a collaborative project with many contributors.
11+
Type 'contributors()' for more information and
12+
'citation()' on how to cite R or R packages in publications.
13+
14+
Type 'demo()' for some demos, 'help()' for on-line help, or
15+
'help.start()' for an HTML browser interface to help.
16+
Type 'q()' to quit R.
17+
18+
- Project '~/research/weather-data-collector-spain' loaded. [renv 1.1.4]
19+
- One or more packages recorded in the lockfile are not installed.
20+
- Use `renv::status()` for more details.
21+
> # get_historical_data.R
22+
> # ----------------------
23+
> # Purpose: Download and update historical daily weather data for Spain from the AEMET OpenData API.
24+
> #
25+
> # This script fetches historical daily climatological data using the 7 core variables
26+
> # that are compatible across current observations, historical data, and forecast endpoints.
27+
> #
28+
> # Core Variables (Standardized):
29+
> # - ta: Air temperature (°C) - from tmed
30+
> # - tamax: Maximum temperature (°C) - from tmax
31+
> # - tamin: Minimum temperature (°C) - from tmin
32+
> # - hr: Relative humidity (%) - from hrMedia
33+
> # - prec: Precipitation (mm) - from prec
34+
> # - vv: Wind speed (km/h) - from velmedia
35+
> # - pres: Atmospheric pressure (hPa) - from presMax
36+
> #
37+
> # Concurrency Control:
38+
> # - Set PREVENT_CONCURRENT_RUNS = TRUE to enable lockfile-based run prevention
39+
> # - Set PREVENT_CONCURRENT_RUNS = FALSE (default) to allow multiple concurrent runs
40+
> #
41+
> # Main Steps:
42+
> # 1. Load dependencies and API key.
43+
> # 2. Determine which dates are missing from the local dataset.
44+
> # 3. Download missing data in chunks, handling API rate limits and errors.
45+
> # 4. Append new data to the historical dataset.
46+
> #
47+
> # Usage:
48+
> # - Requires a valid API key in 'auth/keys.R' as 'my_api_key'.
49+
> # - Run as an R script. Output is written to 'data/output/daily_station_historical.csv.gz'.
50+
> #
51+
> # Dependencies: tidyverse, lubridate, data.table, curl, jsonlite
52+
> #
53+
> # Author: John Palmer
54+
> # Date: 2025-08-22 (Updated for 7-variable standardization)
55+
>
56+
> # Title ####
57+
> # For downloading and preparing historical weather data.
58+
>
59+
> rm(list=ls())
60+
>
61+
> ####Dependencies####
62+
> library(tidyverse)
63+
Error in library(tidyverse) : there is no package called ‘tidyverse’
64+
Execution halted

‎logs/get_latest_data_20250822_122739.out‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
2+
R version 4.4.3 (2025-02-28) -- "Trophy Case"
3+
Copyright (C) 2025 The R Foundation for Statistical Computing
4+
Platform: x86_64-conda-linux-gnu
5+
6+
R is free software and comes with ABSOLUTELY NO WARRANTY.
7+
You are welcome to redistribute it under certain conditions.
8+
Type 'license()' or 'licence()' for distribution details.
9+
10+
R is a collaborative project with many contributors.
11+
Type 'contributors()' for more information and
12+
'citation()' on how to cite R or R packages in publications.
13+
14+
Type 'demo()' for some demos, 'help()' for on-line help, or
15+
'help.start()' for an HTML browser interface to help.
16+
Type 'q()' to quit R.
17+
18+
- Project '~/research/weather-data-collector-spain' loaded. [renv 1.1.4]
19+
- One or more packages recorded in the lockfile are not installed.
20+
- Use `renv::status()` for more details.
21+
> # get_latest_data_expanded.R
22+
> # ----------------------
23+
> # Purpose: Download and update the latest observation data from AEMET stations across Spain.
24+
> #
25+
> # This script fetches weather observations from the AEMET OpenData API using the 7 core variables
26+
> # that are compatible across current observations, historical data, and forecast endpoints.
27+
> #
28+
> # Core Variables (Safe for all endpoints):
29+
> # - ta: Air temperature (°C)
30+
> # - tamax: Maximum temperature (°C)
31+
> # - tamin: Minimum temperature (°C)
32+
> # - hr: Relative humidity (%)
33+
> # - prec: Precipitation (mm)
34+
> # - vv: Wind speed (km/h)
35+
> # - pres: Atmospheric pressure (hPa)
36+
> #
37+
> # Main Steps:
38+
> # 1. Load dependencies and API key.
39+
> # 2. Define functions to request and process data from the AEMET API, with error handling and retries.
40+
> # 3. Download the latest data and reshape it for storage.
41+
> # 4. Append new data to the local CSV file, ensuring no duplicates.
42+
> #
43+
> # Usage:
44+
> # - Requires a valid API key in 'auth/keys.R' as 'my_api_key'.
45+
> # - Run as an R script. Output is written to 'data/spain_weather_expanded.csv.gz'.
46+
> #
47+
> # Dependencies: tidyverse, lubridate, curl, jsonlite, data.table, R.utils
48+
> #
49+
> # Author: John Palmer
50+
> # Date: 2025-08-20 (Updated for 7-variable expansion)
51+
>
52+
> # Title ####
53+
> # For downloading latest observation data from AEMET stations all over Spain. This needs to be run at least every 12 hours, but better to run it every 2 because of API limits, failures etc.
54+
>
55+
> rm(list=ls())
56+
>
57+
>
58+
> # Dependencies ####
59+
> library(tidyverse)
60+
Error in library(tidyverse) : there is no package called ‘tidyverse’
61+
Execution halted

‎logs/weather_collection_17298.err‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
Identity added: /home/j.palmer/.ssh/id_rsa (/home/j.palmer/.ssh/id_rsa)
2+
Identity added: /home/j.palmer/.ssh/id_rsa (/home/j.palmer/.ssh/id_rsa)
3+
4+
CondaError: Run 'conda init' before 'conda activate'
5+
6+
Identity added: /home/j.palmer/.ssh/id_rsa (/home/j.palmer/.ssh/id_rsa)
7+
8+
CondaError: Run 'conda init' before 'conda activate'
9+
10+
rpm: no arguments given for query
11+
/home/j.palmer/.conda/envs/mosquito-alert-monitor/lib/R/bin/R --vanilla -s -f '/tmp/RtmpKOiKwm/renv-install-6d74b521480df.R'
12+
================================================================================
13+
14+
Error in dyn.load(file, DLLpath = DLLpath, ...) :
15+
unable to load shared object '/home/j.palmer/research/weather-data-collector-spain/renv/staging/1/ragg/libs/ragg.so':
16+
libtiff.so.5: cannot open shared object file: No such file or directory
17+
Calls: loadNamespace -> library.dynam -> dyn.load
18+
Execution halted
19+
20+
Error: error testing if 'ragg' can be loaded [error code 1]
21+
Execution halted
22+
From github.com:Mosquito-Alert/weather-data-collector-spain
23+
* branch main -> FETCH_HEAD
24+
Identity added: /home/j.palmer/.ssh/id_rsa (/home/j.palmer/.ssh/id_rsa)
25+
26+
CondaError: Run 'conda init' before 'conda activate'
27+
28+
Identity added: /home/j.palmer/.ssh/id_rsa (/home/j.palmer/.ssh/id_rsa)
29+
30+
CondaError: Run 'conda init' before 'conda activate'
31+
32+
Identity added: /home/j.palmer/.ssh/id_rsa (/home/j.palmer/.ssh/id_rsa)
33+
34+
CondaError: Run 'conda init' before 'conda activate'
35+
36+
Identity added: /home/j.palmer/.ssh/id_rsa (/home/j.palmer/.ssh/id_rsa)
37+
38+
CondaError: Run 'conda init' before 'conda activate'
39+
40+
Identity added: /home/j.palmer/.ssh/id_rsa (/home/j.palmer/.ssh/id_rsa)
41+
42+
CondaError: Run 'conda init' before 'conda activate'
43+
44+
Identity added: /home/j.palmer/.ssh/id_rsa (/home/j.palmer/.ssh/id_rsa)
45+
46+
CondaError: Run 'conda init' before 'conda activate'
47+
48+
Identity added: /home/j.palmer/.ssh/id_rsa (/home/j.palmer/.ssh/id_rsa)
49+
50+
CondaError: Run 'conda init' before 'conda activate'
51+
52+
Identity added: /home/j.palmer/.ssh/id_rsa (/home/j.palmer/.ssh/id_rsa)
53+
54+
CondaError: Run 'conda init' before 'conda activate'
55+
56+
Identity added: /home/j.palmer/.ssh/id_rsa (/home/j.palmer/.ssh/id_rsa)
57+
58+
CondaError: Run 'conda init' before 'conda activate'
59+
60+
Identity added: /home/j.palmer/.ssh/id_rsa (/home/j.palmer/.ssh/id_rsa)
61+
62+
CondaError: Run 'conda init' before 'conda activate'
63+

0 commit comments

Comments
 (0)