IBF-drought-model

Drought forecast pipeline for Zimbabwe RCS.

Introduction

This repo contains the code to:

1. Get latest ENSO data
2. Get latest CHIRPS data
3. Get latest VCI data
4. Forecast drought per province based on the latest data listed above.
5. Calculate impacts of drought province

Data is on MS Azure Datalake ibf.

Functionalities

The pipeline is developed in Docker format. It can run locally in your desktop (with Dock installed, see below). A logic app is set up (510-ibf-drought) and to start the container and perform automated execution (monthly).

settings.py contains basic settings for the pipeline:

• To set test API for posting output and disable email notification
• To switch between operation and dummy mode using a boolean switch dummy
• To define lead-time based on the running mode and the month of execution
• To define a switch of the forecast model depending on the month of execution
• Data sources of ENSO, CHIRPS, VCI

utils.py contains main functions for the pipeline. For now in dummy mode, only the last 2 functions will be executed.

• get_new_senso(): get latest ENSO data from data source
• get_new_chirps(): get latest daily CHIRPS data from data source and calculate monthly accumulation and dryspell per district
• get_new_vci(): get latest observed VCI from data source and calculate monthly average VCI values per district
• arrange_data(): prepare an input data for model 2 by combining ENSO and CHIRPS (and VCI) data into one
• forecast_model1(): forecast drought per province based on the latest ENSO data, using trained XGBoost models
• forecast_model2(): forecast drought per province based on the latest ENSO, monthly rainfall and 14-day dry spell data, using trained XGBoost models
• forecast_model3(): forecast drought per province based on the latest ENSO, monthly rainfall, 14-day dry spell, and VCI data, using trained XGBoost models
• calculate_impact(): calculate exposed population, cattles, ruminants per drought-predicted province(s)
• post_output(): the processed data (drought forecast and impacts) will be posted to the IBF dashboard via IBF API

Setup

with Docker

1. Install Docker

2. Build the docker image from the root directory

docker build -t rodekruis/ibf-drought-model .

3. Run and access the docker container

docker run -it --entrypoint /bin/bash rodekruis/ibf-drought-model

4. Set necessary credentials (in bitwarden) for the pipeline

Usage

Run the pipeline with the command:

run-drought-model

Versions

You can find the versions in the tags of the commits. See below table to find which version of the pipeline corresponds to which version of IBF-Portal.

Drought Pipeline version	IBF-Portal version	Changes
0.2.6	0.305.2	Convert to new warning/trigger setup
0.2.4	0.261.5	Update python env Update xgboost package Reorder input data in arrange_data() Update objects of forecast model 2 and 3 based on new data order
0.2.3	0.247.3	Correct leadtime_str of API payload
0.2.2	0.224.5	Correct arrange_data() to get all past data Add date to API payloads
0.2.1	0.189.1	Change VCI data name
0.2.0	0.170.0	Model 3 added to the pipeline Function to download VCI data from NOAA added Function to arrange data adjusted for VCI data Enable downloading data in off-season
0.1.3	0.152.0	Add function to post non-trigger in off-season
0.1.2	0.129.0	Corrected generation of link to raw chirps file Fixed misdownloading a processed rainfall from datalake Fixed raw chirps files listing for calculating zonal statistics Minor fixes
0.1.1	-	ENSO+rainfall model added Minor fixes
0.1.0	-	Initial version, ENSO-only model