Skip to content

Commit ed26fef

Browse files
michielkallenbergmichielkallenberg
committed
scripts and README
1 parent 1f3eca1 commit ed26fef

5 files changed

Lines changed: 176 additions & 153 deletions

File tree

‎data_preparation/global_AgERA5/README.md‎

Lines changed: 24 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
# Agrometeorological indicators from 1979 to present derived from reanalysis
22

33
## Short description
4-
This dataset provides daily surface meteorological data for the period from 1979 to present as input for agriculture and agro-ecological studies.This dataset is based on the hourly ECMWF ERA5 data at surface level and is referred to as AgERA5.
4+
This dataset provides daily surface meteorological data for the period from 1979 to present as input for agriculture and agro-ecological studies. This dataset is based on the hourly ECMWF ERA5 data at surface level and is referred to as AgERA5.
55

66
## Link
77
https://doi.org/10.24381/cds.6c68c9bb
@@ -30,34 +30,39 @@ Abdelrahman Amr Ali, Dilli R. Paudel
3030
| Temporal resolution | Daily |
3131
| File format | NetCDF-4 |
3232
| Conventions | Climate and Forecast (CF) Metadata Convention v1.7 |
33-
| Versions | 1.0, 1.1 |
34-
| Update frequency | Monthly |
33+
| Versions | 1.1, 2.0 |
34+
| Update frequency | Daily |
3535

3636
## Main variables
3737

38-
| Name | Units | Description |
39-
|--------------------------------------|---------------|-------------|
40-
| 10m wind speed | m s-1 | Mean wind speed at a height of 10 metres above the surface over the period 00h-24h local time. |
41-
| 2m dewpoint temperature | K | Mean dewpoint temperature at a height of 2 metres above the surface over the period 00h-24h local time. The dew point is the temperature to which air must be cooled to become saturated with water vapor. In combination with the air temperature, it is used to assess relative humidity. |
42-
| 2m relative humidity | % | Relative humidity at 06h, 09h, 12h. 15h, 18h (local time) at a height of 2 metres above the surface. This variable describes the amount of water vapour present in air expressed as a percentage of the amount needed for saturation at the same temperature. |
43-
| 2m temperature | K | Air temperature at a height of 2 metres above the surface. |
44-
| Cloud cover | Dimensionless | The number of hours with clouds over the period 00h-24h local time divided by 24 hours. |
45-
| Liquid precipitation duration fraction | Dimensionless | The number of hours with precipitation over the period 00h-24h local time divided by 24 hours and per unit of area. Liquid precipitation is equivalent to the height of the layer of water that would have formed from precipitation had the water not penetrated the soil, run off, or evaporated. |
46-
| Precipitation flux | mm day-1 | Total volume of liquid water (mm³) precipitated over the period 00h-24h local time per unit of area (mm²), per day. |
47-
| Snow thickness | cm | Mean snow depth over the period 00h-24h local time measured as volume of snow (cm³) per unit area (cm²). |
48-
| Snow thickness LWE | cm | Mean snow depth liquid water equivalent (LWE) over the period 00h-24h local time measured as volume of snow (cm³) per unit area (cm²) if all the snow had melted and had not penetrated the soil, runoff, or evaporated. |
49-
| Solar radiation flux | J m-2 day-1 | Total amount of energy provided by solar radiation at the surface over the period 00-24h local time per unit area and time. |
50-
| Solid precipitation duration fraction | Dimensionless | The number of hours with solid precipitation (freezing rain, snow, wet snow, mixture of rain and snow, and ice pellets) over the period 00h-24h local time divided by 24 hours and per unit of area. |
51-
| Vapour pressure | hPa | Contribution to the total atmospheric pressure provided by the water vapour over the period 00-24h local time per unit of time. |
38+
| Name | Units | Description |
39+
|--------------------------------------------|---------------|-------------|
40+
| 10m wind speed | m s⁻¹ | Mean wind speed at a height of 10 metres above the surface over the period 00h–24h local time. |
41+
| 2m dewpoint temperature | K | Mean dewpoint temperature at a height of 2 metres above the surface over the period 00h–24h local time. The dew point is the temperature to which air must be cooled to become saturated with water vapor. In combination with the air temperature it is used to assess relative humidity. |
42+
| 2m relative humidity | % | Relative humidity at 06h, 09h, 12h, 15h, 18h (local time) at a height of 2 metres above the surface. This variable describes the amount of water vapour present in air expressed as a percentage of the amount needed for saturation at the same temperature. |
43+
| 2m relative humidity derived | % | Derived relative humidity provides the daily minimum and maximum relative humidity at 2 meters above the surface. These layers are derived from the AgERA5 daily mean vapour pressure and temperature layers as a post-processing step rather than the original ERA5 inputs. |
44+
| 2m temperature | K | Air temperature at a height of 2 metres above the surface. |
45+
| Cloud cover | Dimensionless | The number of hours with clouds over the period 00h–24h local time divided by 24 hours. |
46+
| Liquid precipitation duration fraction | Dimensionless | The number of hours with precipitation over the period 00h–24h local time divided by 24 hours and per unit of area. Liquid precipitation is equivalent to the height of water that would accumulate if there were no infiltration, runoff, or evaporation. It is calculated using the ERA5 variable *ptype* by determining the fraction of the day during which precipitation occurs in liquid form. |
47+
| Precipitation duration fraction | Dimensionless | The fraction of a day during which precipitation occurs. It is calculated using the ERA5 variable *tp* by determining the number of hours of the day (00h–24h local time) where the precipitation rate is greater than 0.1 mm/hr divided by 24 hours and per unit of area. |
48+
| Precipitation flux | mm day⁻¹ | Total volume of liquid water (mm³) precipitated over the period 00h–24h local time per unit of area (mm²), per day. |
49+
| Reference evapotranspiration | mm day⁻¹ | Calculated using the Penman-Monteith method as described by the FAO56 guidelines, it represents the rate at which a well-watered reference crop loses water to the atmosphere through evapotranspiration and transpiration. |
50+
| Snow thickness | cm | Mean snow depth over the period 00h–24h local time measured as volume of snow (cm³) per unit area (cm²). |
51+
| Snow thickness LWE | cm | Mean snow depth liquid water equivalent (LWE) over the period 00h–24h local time measured as volume of snow (cm³) per unit area (cm²) if all the snow had melted and had not penetrated the soil, runoff, or evaporated. |
52+
| Solar radiation flux | J m⁻² day⁻¹ | Total amount of energy provided by solar radiation at the surface over the period 00–24h local time per unit area and time. |
53+
| Solid precipitation duration fraction | Dimensionless | The number of hours with solid precipitation (freezing rain, snow, wet snow, mixture of rain and snow, and ice pellets) over the period 00h–24h local time divided by 24 hours and per unit of area. It is calculated using the ERA5 variable *ptype* by determining the fraction of the day during which precipitation occurs in solid form. |
54+
| Vapour pressure | hPa | Contribution to the total atmospheric pressure provided by the water vapour over the period 00–24h local time per unit of time. |
55+
| Vapour pressure deficit at daily max temp. | hPa | The difference between the amount of moisture in the air and the maximum amount of moisture the air can hold when it is saturated. It is calculated using the highest temperature recorded in a day, which usually corresponds to the afternoon when evaporation and transpiration rates are highest. |
56+
5257

5358
## Provenance
54-
[Documentation](https://cds.climate.copernicus.eu/cdsapp#!/dataset/10.24381/cds.6c68c9bb?tab=doc) includes
59+
[Documentation](https://cds.climate.copernicus.eu/datasets/sis-agrometeorological-indicators?tab=documentation) includes
5560
* [product user guide and specification](https://confluence.ecmwf.int/x/3FmaE)
5661
* [algorithm theoretical basis](https://confluence.ecmwf.int/x/yFmaE)
5762
* [downscaling and bias correction](https://confluence.ecmwf.int/x/4lmaE)
5863

5964
## License
60-
[License to use Copernicus Products](https://cds.climate.copernicus.eu/api/v2/terms/static/licence-to-use-copernicus-products.pdf)
65+
CC-BY
6166

6267
## How to cite
6368
Cite the dataset as follows:

‎data_preparation/global_AgERA5/download_agera5.py‎

Lines changed: 48 additions & 134 deletions
Original file line numberDiff line numberDiff line change
@@ -1,146 +1,60 @@
11
import os
22
import cdsapi
3-
import concurrent.futures
4-
5-
# Most of the code here comes from Abdelrahman's library.
6-
# https://github.com/AbdelrahmanAmr3/earthstat
3+
import zipfile
4+
import argparse
75

6+
# Mapping of parameters to ERA5 variable/statistic
87
AgERA5_params = {
98
"Maximum_Temperature": ("2m_temperature", "24_hour_maximum"),
109
"Minimum_Temperature": ("2m_temperature", "24_hour_minimum"),
1110
"Mean_Temperature": ("2m_temperature", "24_hour_mean"),
1211
"Solar_Radiation_Flux": ("solar_radiation_flux", None),
1312
"Precipitation_Flux": ("precipitation_flux", None),
14-
# NOTE: Not used. Uncomment if required.
15-
# 'Wind_Speed': {'10m_wind_speed': '24_hour_mean'},
16-
# 'Vapour_Pressure': {'vapour_pressure': '24_hour_mean'},
13+
"Reference_Evapotranspiration": ("reference_evapotranspiration", None),
14+
"Vapour_Pressure_Deficit": ("vapour_pressure_deficit_at_maximum_temperature", None)
1715
}
1816

17+
download_root = '/lustre/backup/SHARED/AIN/agml/predictors/AgERA5/v2.0'
18+
19+
def get_agera5_params(sel_param, year):
20+
var, stat = AgERA5_params[sel_param]
21+
retrieve_params = {
22+
"version": "2_0",
23+
"format": "zip",
24+
"variable": var,
25+
"year": [str(year)],
26+
"month": [f"{m:02d}" for m in range(1, 13)],
27+
"day": [f"{d:02d}" for d in range(1, 32)],
28+
}
29+
if stat is not None:
30+
retrieve_params["statistic"] = [stat]
31+
return retrieve_params
32+
33+
def is_zip_ok(zip_path):
34+
return os.path.isfile(zip_path) and zipfile.is_zipfile(zip_path)
35+
36+
def download_agera5(cds, year, param):
37+
zipfile_path = os.path.join(download_root, param, f"{param}_{year}.zip")
38+
if is_zip_ok(zipfile_path):
39+
print(f"Skipping {zipfile_path} (already exists and OK)")
40+
return
41+
42+
os.makedirs(os.path.join(download_root, param), exist_ok=True)
43+
retrieve_params = get_agera5_params(param, year)
44+
45+
try:
46+
print(f"Downloading {param} {year} ...")
47+
cds.retrieve("sis-agrometeorological-indicators", retrieve_params).download(zipfile_path)
48+
print(f"Download complete: {param} {year}")
49+
except Exception as e:
50+
print(f"Error downloading {param} {year}: {e}")
51+
52+
if __name__ == "__main__":
53+
parser = argparse.ArgumentParser()
54+
parser.add_argument("--year", type=int, required=True)
55+
parser.add_argument("--param", type=str, required=True, choices=list(AgERA5_params.keys()))
56+
args = parser.parse_args()
57+
58+
cds = cdsapi.Client(progress=True)
59+
download_agera5(cds, args.year, args.param)
1960

20-
# @author: Abdelrahman Saleh
21-
def get_agera5_params(sel_param, year, area=None):
22-
if sel_param in AgERA5_params:
23-
var, stat = AgERA5_params[sel_param]
24-
25-
# Construct the return dictionary
26-
retrieve_params = {
27-
"version": "1_1",
28-
"format": "zip",
29-
"variable": var,
30-
"year": str(year),
31-
"month": [
32-
"01",
33-
"02",
34-
"03",
35-
"04",
36-
"05",
37-
"06",
38-
"07",
39-
"08",
40-
"09",
41-
"10",
42-
"11",
43-
"12",
44-
],
45-
"day": [
46-
"01",
47-
"02",
48-
"03",
49-
"04",
50-
"05",
51-
"06",
52-
"07",
53-
"08",
54-
"09",
55-
"10",
56-
"11",
57-
"12",
58-
"13",
59-
"14",
60-
"15",
61-
"16",
62-
"17",
63-
"18",
64-
"19",
65-
"20",
66-
"21",
67-
"22",
68-
"23",
69-
"24",
70-
"25",
71-
"26",
72-
"27",
73-
"28",
74-
"29",
75-
"30",
76-
"31",
77-
],
78-
}
79-
80-
if stat is not None:
81-
retrieve_params["statistic"] = stat
82-
83-
if area is not None:
84-
retrieve_params["area"] = area
85-
86-
return retrieve_params
87-
else:
88-
raise Exception(f"parameter '{sel_param}' is not supported")
89-
90-
91-
def download_agera5_year(cds, year, sel_param, download_path):
92-
download_dir = os.path.join(download_path, sel_param)
93-
os.makedirs(download_dir, exist_ok=True)
94-
retrieve_params = get_agera5_params(sel_param, year)
95-
cds.retrieve(
96-
"sis-agrometeorological-indicators",
97-
retrieve_params,
98-
f"{download_dir}/{sel_param}_{year}.zip",
99-
)
100-
101-
102-
def download_agera5(cds, num_requests, start_year, end_year, download_path):
103-
# NOTE: $HOME/.cdsapirc needs to contain API key
104-
# Example contents of .cdsapirc:
105-
# url: https://cds.climate.copernicus.eu/api
106-
# key: {cdsapi.api_key}
107-
# NOTE: To get a cdsapi token, you will need an ECMWF account.
108-
# See here for details:
109-
# https://confluence.ecmwf.int/x/uINmFw and
110-
# https://cds.climate.copernicus.eu/how-to-api
111-
# Also you need to accept license terms before you can actually download data.
112-
# For AgERA5, accept terms here:
113-
# https://cds.climate.copernicus.eu/datasets/sis-agrometeorological-indicators?tab=download#manage-licenses
114-
if not os.path.exists(os.path.join(os.environ["HOME"], ".cdsapirc")):
115-
raise Exception(".cdsapirc not found in $HOME")
116-
117-
assert num_requests <= 16, Exception("Suggested number of requests is <= 16")
118-
with concurrent.futures.ThreadPoolExecutor(max_workers=num_requests) as executor:
119-
tasks = [
120-
executor.submit(download_agera5_year, cds, year, parameter, download_path)
121-
for year in range(start_year, end_year + 1)
122-
for parameter in AgERA5_params
123-
]
124-
125-
for future in concurrent.futures.as_completed(tasks):
126-
future.result()
127-
128-
129-
# NOTE:
130-
# 1. Make sure you have enough disk space when you run the full script.
131-
# 2. Some zip files seem to have errors. You may need to download them again.
132-
cds = cdsapi.Client(progress=False)
133-
download_path = "/path/to/downloads"
134-
download_agera5(cds, 16, 2001, 2023, download_path)
135-
136-
# NOTE:
137-
# After downloading data, you need 2 more steps before you can
138-
# use `../predictor_data_prep.py` to mask and aggregate AgERA5 data to admin regions.
139-
# 1. Data will be downloaded to folders named "Maximum_Temperature",
140-
# "Minimum_Temperature", etc. Run the `rename_agera5_files.py` in each folder.
141-
# 2. Rename folder names as follows:
142-
# "Maximum_Temperature" -> tmax
143-
# "Minimum_Temperature" -> tmin
144-
# "Mean_Temperature" -> tavg
145-
# "Precipitation_Flux" -> prec
146-
# "Solar_Radiation_Flux" -> rad

‎data_preparation/global_AgERA5/download_agera5_slurm.sh‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
#!/bin/bash
2+
#SBATCH --job-name=agera5_download
3+
#SBATCH --output=output/agera5/output_%A_%a_%j.txt
4+
#SBATCH --error=output/agera5/error_%A_%a_%j.txt
5+
#SBATCH --mem-per-cpu=16G
6+
#SBATCH --ntasks=1
7+
#SBATCH --cpus-per-task=1
8+
#SBATCH --time=4-00:00:00
9+
#SBATCH --array=0-160%10 # This will cover all year×param combinations
10+
11+
# Initialize modules
12+
module load 2024
13+
module load Python/3.12.3-GCCcore-13.3.0
14+
15+
# Change to working directory
16+
cd /lustre/backup/SHARED/AIN/agml/AgML-CY-Bench/
17+
18+
# Define years and parameters
19+
YEARS=(2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023)
20+
PARAMS=("Maximum_Temperature" "Minimum_Temperature" "Mean_Temperature" "Solar_Radiation_Flux" "Precipitation_Flux" "Reference_Evapotranspiration" "Vapour_Pressure_Deficit")
21+
22+
# Map SLURM_ARRAY_TASK_ID to year and param
23+
TOTAL_PARAMS=${#PARAMS[@]}
24+
YEAR_INDEX=$(( SLURM_ARRAY_TASK_ID / TOTAL_PARAMS ))
25+
PARAM_INDEX=$(( SLURM_ARRAY_TASK_ID % TOTAL_PARAMS ))
26+
27+
YEAR=${YEARS[$YEAR_INDEX]}
28+
PARAM=${PARAMS[$PARAM_INDEX]}
29+
30+
echo "Downloading $PARAM $YEAR"
31+
32+
# Run the Python script
33+
poetry run python /lustre/backup/SHARED/AIN/agml/predictors/AgERA5/download_agera5.py --year $YEAR --param $PARAM
34+

‎data_preparation/global_AgERA5/merge_agera5.py‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
import os
2+
import sys
3+
import pandas as pd
4+
5+
if len(sys.argv) != 3:
6+
print("Usage: python merge_region.py <crop> <country>")
7+
sys.exit(1)
8+
9+
crop = sys.argv[1]
10+
country = sys.argv[2]
11+
12+
data_path = "/lustre/backup/SHARED/AIN/agml/python-output"
13+
cn_dir = os.path.join(data_path, crop, country)
14+
15+
if not os.path.isdir(cn_dir):
16+
print(f"No directory found for {crop}-{country}")
17+
sys.exit(0)
18+
19+
indicators = ["tmin", "tmax", "prec", "rad", "tavg", "et0"]
20+
21+
print(f"Processing {crop}-{country}")
22+
23+
agera5_df = None
24+
for ind in indicators:
25+
csv_file = os.path.join(cn_dir, ind, f"{ind}_{crop}_{country}.csv")
26+
if not os.path.exists(csv_file):
27+
print(f" Missing: {ind}_{crop}_{country}.csv")
28+
continue
29+
30+
ind_df = pd.read_csv(csv_file)
31+
32+
if agera5_df is None:
33+
agera5_df = ind_df
34+
else:
35+
agera5_df = agera5_df.merge(ind_df, on=["crop_name", "adm_id", "date"])
36+
37+
if agera5_df is not None:
38+
if "prec" in agera5_df and "et0" in agera5_df:
39+
agera5_df["cwb"] = agera5_df["prec"] - agera5_df["et0"]
40+
agera5_df = agera5_df.round(3)
41+
out_file = os.path.join(cn_dir, f"meteo_{crop}_{country}.csv")
42+
agera5_df.to_csv(out_file, index=False)
43+
print(f" Saved {out_file}")
44+
else:
45+
print(f"No data merged for {crop}-{country}")

‎data_preparation/global_AgERA5/merge_agera5_slurm.sh‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
#!/bin/bash
2+
#
3+
#SBATCH --job-name=merge_meteo
4+
#SBATCH --output=output/merge/output_%A_%a.txt
5+
#SBATCH --error=output/merge/error_%A_%a.txt
6+
#SBATCH --mem-per-cpu=8G
7+
#SBATCH --cpus-per-task=2
8+
#SBATCH --ntasks=1
9+
#SBATCH --time=1-00:00:00
10+
#SBATCH --array=0-0
11+
12+
module load 2024
13+
module load Python/3.12.3-GCCcore-13.3.0
14+
15+
# Get line from list (only crop-country, ignore indicator)
16+
line=$(awk "NR==${SLURM_ARRAY_TASK_ID}+1" /lustre/backup/SHARED/AIN/agml/crop_country_list_simple.txt)
17+
crop=$(echo "$line" | cut -d'_' -f1)
18+
country=$(echo "$line" | cut -d'_' -f2)
19+
20+
# Change to working directory
21+
cd /lustre/backup/SHARED/AIN/agml/AgML-CY-Bench/
22+
23+
# Run merge script
24+
echo "Merging region: $crop $country"
25+
poetry run python /lustre/backup/SHARED/AIN/agml/predictors/merge_agera5.py "$crop" "$country"

0 commit comments

Comments
 (0)