Research and identify solutions for workflow monitoring and observation.

[nikki-t]

We need to be able to automatically execute the Confluence workflow on a monthly basis. We should set up the Confluence workflow so it can be executed by Airflow.

This documentation covers how to execute a Step Function state machine using Airflow: https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/operators/step_functions.html

We will want to install Airflow in whatever AWS venue we decide to execute within and write a DAG that uses the StepFunctionStartExecutionOperator to start the execution. It may be helpful to use the StepFunctionExecutionSensor to wait for the execution to complete.

[nikki-t]

I have set up a small Step Function that integrates a single Confluence module (init_workflow) for testing in the services SIT environment. It is called: svc-confluence-sit-workflow

The inputs needed by the Step Function are:

    "version": "0001",
    "run_type": "constrained",
    "reach_subset_file": "reaches_of_interest.json",
    "temporal_range": "&start_time=2020-09-01T00:00:00Z&end_time=2025-10-30T00:00:00Z&",
    "tolerated_failure_percentage": "50",
    "run_gbpriors": "false",
    "run_postdiagnostics": "true",
    "counter": "test-run-0001"

Where the counter gets incremented with each run (e.g. 0001 --> 0002).

Here are some initial thoughts on how to define each of these in whatever triggers the Step Function:

• Create a step-function-init.json file that keeps some state between executions and store this in the config S3 bucket. This file gets updated by the Airflow DAG that triggers the workflow after the execution is complete.
• version: Incremented with each run and is kept in the step-function-init.json
• run_type: This value alternates between constrained and unconstrained with every other run. This is kept in the step-function-init.json file.
• reach_subset_file: reaches_of_interest.json is a constant value and does not typically change between executions.
• temporal_range: Keep a start time in step-function-init.json file and add an end time for current data and time with each execution.
• tolerated_failure_percentage: Constant and does not change
• run_gbpriors: Constant and does not change often. The Confluence dev team will let us know when to run these.
• run_postdiagnostics: Constant and is always set to true.
• counter: Incremented with each run and is kept in the step-function-init.json file with a start value of global-0001.

We also need to define the name of the step function execution as follows: svc-confluence-{venue}-{version}-{runtype}-{counter}

[nikki-t]

Let's plan for a complete demo on May 6 with the SIT Airflow deployment.

[nikki-t]

Config S3 bucket for SIT environment: svc-confluence-sit-config

[nikki-t]

Publish CNM module: https://github.com/podaac/confluence_publish_cnm

[nikki-t]

Let's create a separate directory to hold the report.py file in the GitHub repo: https://github.com/podaac/airflow-dags so that we can organized DAG-specific files under that folder.