Skip to content

Dev #53

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
p-phung merged 3 commits into from
Dec 18, 2025
Merged

Dev #53

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
ac6d08e
update logic app json
p-phung Dec 18, 2025
691d3e9
adding dataset placeholders to readme
p-phung Dec 18, 2025
8034a49
Merge pull request #52 from rodekruis/update-logic-app
p-phung Dec 18, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 14 additions & 4 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,12 @@ Furthermore, for the ECMWF extractors one needs to accept the *Terms of use*.
This is done on the ECMWF website and requires you to be logged in (corresponding to the account of the api-key).
For example, for the extreme heat dataset see: https://cds.climate.copernicus.eu/datasets/derived-utci-historical?tab=download.

#### CHIRPS
TBA

#### WorldPop
TBA

#### IOM DTM
IOM DTM data requires API key to access data. See more about the [DTM API](https://dtm.iom.int/data-and-analysis/dtm-api) latest version V3.
Set the api_key in environment variable `DTM_API_KEY` in the `.env` file.
Expand All @@ -71,13 +77,14 @@ Set the api_key in environment variable `DTM_API_KEY` in the `.env` file.
For IPC extractor, API key is needed.
Set api key: TBA

#### FEWS NET
TBA

### Data transformation

While the retrieval pipeline focuses on downloading raw data, the transform pipeline turns those
datasets into district‑level climate indices suitable for multi‑criteria prioritisation. It operates on
After extractions, the transformations turns those datasets into district‑level suitable for multi‑criteria prioritisation. It operates on
three broad climate hazards—drought, flood and extreme heat—and produces baseline, recent and
trend indicators for each. These pipelines live in the `transformpipeline/` package and are
assembled from reusable `Step` classes defined in `transformpipeline.core`.
trend indicators for each, as well as other transformations.

### Indicators

Expand All @@ -94,6 +101,9 @@ The table below summarises what each climate indicator measures and the datasets
| **Drought trend** | Evaluates 30‑year trends in drought risk by fitting linear regressions to time series of drought frequency, intensity and persistency. Negative slopes are clipped to zero before normalisation. | CHIRPS monthly NetCDF; SPI‑based metrics are computed for each year, trends estimated by linear regression and normalised before combination. |
| **Flood trend** | Tracks trends in flood extent over 1984–2021 by fitting a linear regression to annual flood percentages. Negative trends are set to zero and positive slopes normalised. | JRC Monthly Water History v1.4 rasters; annual flooded area percentages are computed, trends estimated via linear regression and normalised. |
| **Extreme heat trend** | Assesses trends in extreme heat frequency, intensity and persistency over the last 30 years using UTCI data. Negative trends are clipped and remaining slopes normalised and combined. | ERA5‑derived UTCI hourly NetCDF; annual time series of exceedance frequency/intensity/persistency are built, linear trends computed and normalised. |
| **Displacement** | TBA | TBA |
| **Food insecurity** | TBA | TBA |
| **Population** | TBA | TBA |

## To contribute

Expand Down
127 changes: 101 additions & 26 deletions logic_app/demo_logic_app.json → logic_app/logic-app-prod.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,18 @@
"triggers": {
"When_an_HTTP_request_is_received": {
"type": "Request",
"kind": "Http"
"kind": "Http",
"inputs": {
"method": "POST",
"schema": {
"properties": {
"country": {
"type": "string"
}
},
"type": "object"
}
}
}
},
"actions": {
Expand All @@ -15,7 +26,7 @@
"inputs": {
"host": {
"connection": {
"name": "@parameters('$connections')['keyvault']['connectionId']"
"name": "@parameters('$connections')['keyvault-1']['connectionId']"
}
},
"method": "get",
Expand Down Expand Up @@ -44,6 +55,12 @@
"Get_container-registry-password": [
"Succeeded"
],
"Get_dtm-api-key": [
"Succeeded"
],
"Get_ipc-api-key": [
"Succeeded"
],
"containerIsDone_=_false": [
"Succeeded"
],
Expand All @@ -55,7 +72,7 @@
"inputs": {
"host": {
"connection": {
"name": "@parameters('$connections')['aci']['connectionId']"
"name": "@parameters('$connections')['aci-1']['connectionId']"
}
},
"method": "put",
Expand All @@ -64,9 +81,17 @@
"properties": {
"containers": [
{
"name": "tjebbe-demo",
"name": "container",
"properties": {
"image": "tjebbedemo-gnh9duaghsdfhyfx.azurecr.io/etl-pipeline:latest",
"command": [
"poetry",
"run",
"python",
"run_pipeline.py",
"--country-iso3",
"@triggerBody()?['country']"
],
"image": "datapipelineregistry-a5eudbdee8djhnh8.azurecr.io/exposure-vulnerability-retrieval:prod",
"environmentVariables": [
{
"name": "BLOB_ACCOUNT_NAME",
Expand All @@ -79,6 +104,14 @@
{
"name": "ECMWF_DATASTORES_KEY",
"secureValue": "@body('Get_ecmwf-datastores-key')?['value']"
},
{
"name": "IPC_API_KEY",
"secureValue": "@body('Get_ipc-api-key')?['value']"
},
{
"name": "DTM_API_KEY",
"secureValue": "@body('Get_dtm-api-key')?['value']"
}
],
"resources": {
Expand All @@ -93,8 +126,8 @@
"osType": "Linux",
"imageRegistryCredentials": [
{
"server": "tjebbedemo-gnh9duaghsdfhyfx.azurecr.io",
"username": "tjebbedemo",
"server": "datapipelineregistry-a5eudbdee8djhnh8.azurecr.io",
"username": "datapipelineregistry",
"password": "@body('Get_container-registry-password')?['value']"
}
],
Expand All @@ -103,11 +136,11 @@
"identity": {
"type": "UserAssigned",
"userAssignedIdentities": {
"/subscriptions/57b0d17a-5429-4dbb-8366-35c928e3ed94/resourceGroups/ibf-data-exploratory-rg/providers/Microsoft.ManagedIdentity/userAssignedIdentities/tjebbe-demo": {}
"/subscriptions/57b0d17a-5429-4dbb-8366-35c928e3ed94/resourceGroups/exposure-vulnerability/providers/Microsoft.ManagedIdentity/userAssignedIdentities/exposure-etl-managed-identity": {}
}
}
},
"path": "/subscriptions/@{encodeURIComponent('57b0d17a-5429-4dbb-8366-35c928e3ed94')}/resourceGroups/@{encodeURIComponent('ibf-data-exploratory-rg')}/providers/Microsoft.ContainerInstance/containerGroups/@{encodeURIComponent('tjebbe-demo')}",
"path": "/subscriptions/@{encodeURIComponent('57b0d17a-5429-4dbb-8366-35c928e3ed94')}/resourceGroups/@{encodeURIComponent('exposure-vulnerability')}/providers/Microsoft.ContainerInstance/containerGroups/@{encodeURIComponent('exposure-vulnerability-retrieval-prod-',toLower(triggerBody()?['country']),'-cg')}",
"queries": {
"x-ms-api-version": "2023-05-01"
}
Expand All @@ -123,11 +156,11 @@
"inputs": {
"host": {
"connection": {
"name": "@parameters('$connections')['aci']['connectionId']"
"name": "@parameters('$connections')['aci-1']['connectionId']"
}
},
"method": "get",
"path": "/subscriptions/@{encodeURIComponent('57b0d17a-5429-4dbb-8366-35c928e3ed94')}/resourceGroups/@{encodeURIComponent('ibf-data-exploratory-rg')}/providers/Microsoft.ContainerInstance/containerGroups/@{encodeURIComponent('tjebbe-demo')}/containers/@{encodeURIComponent('tjebbe-demo')}/logs",
"path": "/subscriptions/@{encodeURIComponent('57b0d17a-5429-4dbb-8366-35c928e3ed94')}/resourceGroups/@{encodeURIComponent('exposure-vulnerability')}/providers/Microsoft.ContainerInstance/containerGroups/@{encodeURIComponent('exposure-vulnerability-retrieval-prod-',toLower(triggerBody()?['country']),'-cg')}/containers/@{encodeURIComponent('container')}/logs",
"queries": {
"x-ms-api-version": "2023-05-01"
}
Expand All @@ -146,11 +179,11 @@
"inputs": {
"host": {
"connection": {
"name": "@parameters('$connections')['aci']['connectionId']"
"name": "@parameters('$connections')['aci-1']['connectionId']"
}
},
"method": "delete",
"path": "/subscriptions/@{encodeURIComponent('57b0d17a-5429-4dbb-8366-35c928e3ed94')}/resourceGroups/@{encodeURIComponent('ibf-data-exploratory-rg')}/providers/Microsoft.ContainerInstance/containerGroups/@{encodeURIComponent('tjebbe-demo')}",
"path": "/subscriptions/@{encodeURIComponent('57b0d17a-5429-4dbb-8366-35c928e3ed94')}/resourceGroups/@{encodeURIComponent('exposure-vulnerability')}/providers/Microsoft.ContainerInstance/containerGroups/@{encodeURIComponent('exposure-vulnerability-retrieval-prod-',toLower(triggerBody()?['country']),'-cg')}",
"queries": {
"x-ms-api-version": "2023-05-01"
}
Expand All @@ -163,11 +196,11 @@
"inputs": {
"host": {
"connection": {
"name": "@parameters('$connections')['aci']['connectionId']"
"name": "@parameters('$connections')['aci-1']['connectionId']"
}
},
"method": "get",
"path": "/subscriptions/@{encodeURIComponent('57b0d17a-5429-4dbb-8366-35c928e3ed94')}/resourceGroups/@{encodeURIComponent('ibf-data-exploratory-rg')}/providers/Microsoft.ContainerInstance/containerGroups/@{encodeURIComponent('tjebbe-demo')}",
"path": "/subscriptions/@{encodeURIComponent('57b0d17a-5429-4dbb-8366-35c928e3ed94')}/resourceGroups/@{encodeURIComponent('exposure-vulnerability')}/providers/Microsoft.ContainerInstance/containerGroups/@{encodeURIComponent('exposure-vulnerability-retrieval-prod-',toLower(triggerBody()?['country']),'-cg')}",
"queries": {
"x-ms-api-version": "2023-05-01"
}
Expand Down Expand Up @@ -238,7 +271,7 @@
"inputs": {
"host": {
"connection": {
"name": "@parameters('$connections')['keyvault']['connectionId']"
"name": "@parameters('$connections')['keyvault-1']['connectionId']"
}
},
"method": "get",
Expand All @@ -259,7 +292,7 @@
"inputs": {
"host": {
"connection": {
"name": "@parameters('$connections')['keyvault']['connectionId']"
"name": "@parameters('$connections')['keyvault-1']['connectionId']"
}
},
"method": "get",
Expand All @@ -280,7 +313,7 @@
"inputs": {
"host": {
"connection": {
"name": "@parameters('$connections')['keyvault']['connectionId']"
"name": "@parameters('$connections')['keyvault-1']['connectionId']"
}
},
"method": "get",
Expand Down Expand Up @@ -314,15 +347,57 @@
"inputs": {
"host": {
"connection": {
"name": "@parameters('$connections')['aci']['connectionId']"
"name": "@parameters('$connections')['aci-1']['connectionId']"
}
},
"method": "delete",
"path": "/subscriptions/@{encodeURIComponent('57b0d17a-5429-4dbb-8366-35c928e3ed94')}/resourceGroups/@{encodeURIComponent('ibf-data-exploratory-rg')}/providers/Microsoft.ContainerInstance/containerGroups/@{encodeURIComponent('tjebbe-demo')}",
"path": "/subscriptions/@{encodeURIComponent('57b0d17a-5429-4dbb-8366-35c928e3ed94')}/resourceGroups/@{encodeURIComponent('exposure-vulnerability')}/providers/Microsoft.ContainerInstance/containerGroups/@{encodeURIComponent('exposure-vulnerability-retrieval-prod-',toLower(triggerBody()?['country']),'-cg')}",
"queries": {
"x-ms-api-version": "2023-05-01"
}
}
},
"Get_dtm-api-key": {
"runAfter": {},
"type": "ApiConnection",
"inputs": {
"host": {
"connection": {
"name": "@parameters('$connections')['keyvault-1']['connectionId']"
}
},
"method": "get",
"path": "/secrets/@{encodeURIComponent('dtm-api-key')}/value"
},
"runtimeConfiguration": {
"secureData": {
"properties": [
"inputs",
"outputs"
]
}
}
},
"Get_ipc-api-key": {
"runAfter": {},
"type": "ApiConnection",
"inputs": {
"host": {
"connection": {
"name": "@parameters('$connections')['keyvault-1']['connectionId']"
}
},
"method": "get",
"path": "/secrets/@{encodeURIComponent('ipc-api-key')}/value"
},
"runtimeConfiguration": {
"secureData": {
"properties": [
"inputs",
"outputs"
]
}
}
}
},
"outputs": {},
Expand All @@ -337,20 +412,20 @@
"$connections": {
"type": "Object",
"value": {
"keyvault": {
"keyvault-1": {
"id": "/subscriptions/57b0d17a-5429-4dbb-8366-35c928e3ed94/providers/Microsoft.Web/locations/westeurope/managedApis/keyvault",
"connectionId": "/subscriptions/57b0d17a-5429-4dbb-8366-35c928e3ed94/resourceGroups/ibf-data-exploratory-rg/providers/Microsoft.Web/connections/keyvault",
"connectionName": "keyvault",
"connectionId": "/subscriptions/57b0d17a-5429-4dbb-8366-35c928e3ed94/resourceGroups/exposure-vulnerability/providers/Microsoft.Web/connections/keyvault-1",
"connectionName": "keyvault-1",
"connectionProperties": {
"authentication": {
"type": "ManagedServiceIdentity"
}
}
},
"aci": {
"aci-1": {
"id": "/subscriptions/57b0d17a-5429-4dbb-8366-35c928e3ed94/providers/Microsoft.Web/locations/westeurope/managedApis/aci",
"connectionId": "/subscriptions/57b0d17a-5429-4dbb-8366-35c928e3ed94/resourceGroups/ibf-data-exploratory-rg/providers/Microsoft.Web/connections/aci",
"connectionName": "aci",
"connectionId": "/subscriptions/57b0d17a-5429-4dbb-8366-35c928e3ed94/resourceGroups/exposure-vulnerability/providers/Microsoft.Web/connections/aci-1",
"connectionName": "aci-1",
"connectionProperties": {
"authentication": {
"type": "ManagedServiceIdentity"
Expand Down