Disasters/add STAC metadata

aws_dags_variables.json

@@ -0,0 +1,12 @@
+{
+  "EVENT_BUCKET": "veda-tf-state-shared-disasters",
+  "STAC_INGESTOR_API_URL": "https://dev.disasters.openveda.cloud/api/ingest/",
+  "STAC_URL": "https://dev.disasters.openveda.cloud/api/stac",
+  "INGEST_API_KEYCLOAK_APP_SECRET": "",
+  "ASSUME_ROLE_READ_ARN": "",
+  "ASSUME_ROLE_WRITE_ARN": "",
+  "VECTOR_SECRET_NAME": "null",
+  "SM2A_BASE_URL": "https://airflow.disasters.openveda.cloud",
+  "CLOUDFRONT_TO_INVALIDATE": null,
+  "CLOUDFRONT_PATH_TO_INVALIDATE": null
+}

aws_dags_variables_instructions.md

@@ -0,0 +1,81 @@
+# AWS DAGs Variables Configuration
+
+This JSON file contains the configuration variables that Airflow needs to run the VEDA data pipelines.
+
+## How to Use
+
+### Option 1: Upload to AWS Secrets Manager (Recommended for Production)
+```bash
+aws secretsmanager create-secret \
+  --name veda-pipeline-dev/airflow/variables/aws_dags_variables \
+  --secret-string file://aws_dags_variables.json \
+  --region us-west-2
+```
+
+Or update an existing secret:
+```bash
+aws secretsmanager update-secret \
+  --secret-id veda-pipeline-dev/airflow/variables/aws_dags_variables \
+  --secret-string file://aws_dags_variables.json \
+  --region us-west-2
+```
+
+### Option 2: Add via Airflow UI (Quick Fix for Development)
+1. Navigate to Airflow UI → Admin → Variables
+2. Click "+" to add a new variable
+3. Key: `aws_dags_variables`
+4. Value: Paste the JSON content from `aws_dags_variables.json`
+5. Click Save
+
+## Required Fields to Update
+
+Before using this file, you MUST update the following placeholder values:
+
+### 1. INGEST_API_KEYCLOAK_APP_SECRET
+- **Description**: Keycloak client secret for authenticating with the STAC ingestor API
+- **How to get it**: Contact your Keycloak administrator or retrieve from AWS Secrets Manager
+- **Example**: `"a1b2c3d4-e5f6-7g8h-9i0j-k1l2m3n4o5p6"`
+
+### 2. ASSUME_ROLE_READ_ARN
+- **Description**: IAM Role ARN that Airflow workers assume to read data from S3
+- **Format**: `arn:aws:iam::ACCOUNT_ID:role/ROLE_NAME`
+- **Example**: `"arn:aws:iam::114506680961:role/veda-data-access-role-dev"`
+
+### 3. ASSUME_ROLE_WRITE_ARN
+- **Description**: IAM Role ARN that Airflow workers assume to write data to S3
+- **Format**: `arn:aws:iam::ACCOUNT_ID:role/ROLE_NAME`
+- **Example**: `"arn:aws:iam::114506680961:role/veda-data-write-role-dev"`
+
+## Pre-configured Fields (Disasters-specific)
+
+These fields are already configured for the disasters environment:
+
+- **EVENT_BUCKET**: `veda-tf-state-shared-disasters` - S3 bucket for events and processing data
+- **STAC_INGESTOR_API_URL**: `https://dev.disasters.openveda.cloud/api/ingest/` - Disasters STAC ingestor endpoint
+- **STAC_URL**: `https://dev.disasters.openveda.cloud/api/stac` - Public STAC catalog endpoint
+- **SM2A_BASE_URL**: `https://airflow.disasters.openveda.cloud` - Airflow base URL
+
+## Optional Fields
+
+These fields are set to `null` or default values and may not be needed:
+
+- **VECTOR_SECRET_NAME**: AWS Secrets Manager secret name for vector database credentials
+- **CLOUDFRONT_TO_INVALIDATE**: CloudFront distribution ID to invalidate after updates
+- **CLOUDFRONT_PATH_TO_INVALIDATE**: CloudFront path pattern to invalidate
+
+## Terraform Integration
+
+This configuration matches the terraform variables in `infrastructure/terraform.tfvars`. 
+When deployed via Terraform, these values are automatically synced to AWS Secrets Manager.
+
+## Verification
+
+After updating the Airflow variable, verify it's accessible:
+
+```python
+from airflow.models.variable import Variable
+import json
+
+vars = Variable.get("aws_dags_variables", deserialize_json=True)
+print(vars["STAC_INGESTOR_API_URL"])  # Should print: https://dev.disasters.openveda.cloud/api/ingest/
+```

aws_dags_variables_minified.json

@@ -0,0 +1 @@
+{"EVENT_BUCKET": "veda-tf-state-shared-disasters", "STAC_INGESTOR_API_URL": "https://dev.disasters.openveda.cloud/api/ingest/", "STAC_URL": "https://dev.disasters.openveda.cloud/api/stac", "INGEST_API_KEYCLOAK_APP_SECRET": "", "ASSUME_ROLE_READ_ARN": "", "ASSUME_ROLE_WRITE_ARN": "", "VECTOR_SECRET_NAME": "null", "SM2A_BASE_URL": "https://airflow.disasters.openveda.cloud", "CLOUDFRONT_TO_INVALIDATE": null, "CLOUDFRONT_PATH_TO_INVALIDATE": null}

dags/veda_data_pipeline/utils/LUT/event-hazard-location.json

@@ -0,0 +1,54 @@
+{
+    "202603_Flood_HI": {
+        "hazard": ["Flood", "Landslide"], 
+        "location": ["Hawaii"]
+    },
+    "202601_WinterWx_US": {
+        "hazard": ["WinterWx"], 
+        "location": ["Virginia", "Texas", "Tennessee", "South Carolina", "Mississippi"]
+    },
+    "202512_Flood_WA": {
+        "hazard": ["Flood"], 
+        "location": ["Washington", "Landslide"]
+    },
+    "202510_Hurricane_Melissa": {
+        "hazard": ["Tropical Cyclone", "Hurricane", "Flood", "Landslide"], 
+        "location": ["Jamaica"]
+    },
+    "202510_Flood_AK": {
+        "hazard": ["Flood"], 
+        "location": ["Alaska"]
+    },
+    "202507_Flood_TX": {
+        "hazard": ["Flood"], 
+        "location": ["Texas"]
+    },
+    "202506_Fire_NM": {
+        "hazard": ["Fire", "Flood"], 
+        "location": ["New Mexico"]
+    },
+    "202504_SevereWx_US": {
+        "hazard": ["SevereWx", "Flood"], 
+        "location": ["Arkansas", "Alabama", "Louisiana", "Mississippi", "Missouri", "Tennessee"]
+    },
+    "202503_SevereWx_US": {
+        "hazard": ["SevereWx"], 
+        "location": ["Arkansas", "Alabama", "Kentucky", "Louisiana", "Mississippi", "Missouri", "Tennessee"]
+    },
+    "202502_Flood_OhioValley": {
+        "hazard": ["Flood", "Landslide"], 
+        "location": ["Kentucky", "Tennessee", "Virginia", "West Virginia"]
+    },
+    "202501_Fire_CA": {
+        "hazard": ["Fire"], 
+        "location": ["California"]
+    },
+    "202410_Hurricane_Milton": {
+        "hazard": ["Tropical Cyclone", "Hurricane", "Flood"], 
+        "location": ["Florida"]
+    },
+    "202409_Hurricane_Helene": {
+        "hazard": ["Tropical Cyclone", "Hurricane", "Flood", "Landslide"], 
+        "location": ["Alabama", "North Carolina", "South Carolina", "Tennessee", "Virginia"]
+    }
+}