Varo packager

packages/varo/.changeset/README.md

@@ -0,0 +1,8 @@
+# Changesets
+
+Hello and welcome! This folder has been automatically generated by `@changesets/cli`, a build tool that works
+with multi-package repos, or single-package repos to help you version and publish your code. You can
+find the full documentation for it [in our repository](https://github.com/changesets/changesets)
+
+We have a quick list of common questions to get you started engaging with this project in
+[our documentation](https://github.com/changesets/changesets/blob/main/docs/common-questions.md)

packages/varo/.changeset/config.json

@@ -0,0 +1,11 @@
+{
+  "$schema": "https://unpkg.com/@changesets/[email protected]/schema.json",
+  "changelog": "@changesets/cli/changelog",
+  "commit": false,
+  "fixed": [],
+  "linked": [],
+  "access": "restricted",
+  "baseBranch": "main",
+  "updateInternalDependencies": "patch",
+  "ignore": []
+}

packages/varo/.changeset/yellow-papayas-perform.md

@@ -0,0 +1,6 @@
+---
+'@openfn/language-varo': minor
+---
+
+Added components to support converting collection data into Varo message
+packages.

packages/varo/README.md

@@ -1,34 +1,26 @@
 # language-varo <img src='./assets/square.png' width="30" height="30"/>
 
-The Varo adaptor works in conjunction with the Gmail adaptor. It takes proprietary data formats found in `state.data` and attempts to convert them to EMS format.
+The Varo adaptor is designed to work in conjunction with the Gmail adaptor.
 
-## Setup the primary workflow and jobs
+The following workflows have been illustrated:
+- Gmail -> EMS report
+  Take proprietary data formats found in Gmail messages and convert them to EMS-compliant format.
+- CCDX data collection -> Gmail
+  Retrieve data found in an OpenFn collection, convert it to a EMS report, assemble it as a Varo package and send it to a Varo inbox to be processed by Pogo LT.
+
+# Gmail -> EMS report
 
 This will demonstrate how to pull messages from a Gmail account and pass them to the Varo adaptor which will convert them into EMS format to be used for downstream consumers in the workflow.
 
 The workflow requires some pre-configuration, namely the Gmail `access_token` and the OpenFn `collection_token`.
 
-### Token configuration
-
-#### Gmail token
-
-Use Postman to retrieve an access token. This is a short-lived token will last 60 minutes and will have to be manually retrieved. See the documentation in the [Gmail adaptor readme](https://docs.openfn.org/adaptors/packages/gmail-readme#use-the-postman-application-to-query-the-oauth-enpoint-and-retrieve-an-access-token) for a guide on how to configure Postman to retrieve the access token.
-
-#### OpenFn collections token
+## Example workflow
 
-The workflow tracks the previously processed messages by storing the processed IDs in an OpenFn collection. OpenFn requires authorization to access a given collection.
-
-1. Access the [Collections](http://localhost:4000/settings/collections) configuration in the administration area.
-1. Ensure the collection named `gmail-processed-ids` exists.
-1. Create a new token in the [Personal Access Tokens](http://localhost:4000/profile/tokens) configuration.
+Place these files in the openfn/adaptors/workflows/readVaroPackages folder.
 
-### Workflow
+### workflow.json
 
-Place these files in the openfn/adaptors/workflows folder
-
-#### workflow.json
-
-```
+```js
 {
   "options": {
     "start": "getContentsFromMessages"
@@ -63,13 +55,11 @@ Place these files in the openfn/adaptors/workflows folder
 }
 ```
 
-### Jobs
-
-#### jobGmail.js
+### jobGmail.js
 
 This job will define message parts of interest including: metadata, data and fridgeTag. Also important is a collection of previously-processed messageIds. This job will grab ids from the `gmail-processed-ids` collection and pass them into the request. Once the request is complete, the new, processed messageIds are placed into the collection for future retrieval.
 
-```
+```js
 const contents = [
   { name: "metadata", file: /\d{12}_\d{8}T\d{6}Z\.json$/ },
   { name: "data", file: /_CURRENT_DATA_.*\.json$/ },
@@ -97,13 +87,13 @@ fnIf(
 );
 ```
 
-#### jobVaro.js
+### jobVaro.js
 
 The function `convertToEms` expects an array of message contents. This property contains text files in EMS-like Varo format or FridgeTag format and transforms them into EMS-structured data. Tip: This format is automatically provided by the Gmail adaptor. 
 
 Expected data structure:
 
-```
+```js
 [
   {
     metadata: {
@@ -119,7 +109,7 @@ Expected data structure:
 ```
 
 
-```
+```js
 convertToEms($.data);
 
 fn((state) => {
@@ -128,20 +118,103 @@ fn((state) => {
 });
 ```
 
-### Running the workflow
+# CCDX data collection -> Gmail
+
+Following is an example demonstrating how to retrieve data found in an OpenFn collection, convert it to a EMS report, assemble it as a Varo package and send it to a Varo inbox to be processed by Pogo LT.
+
+## Example workflow
+
+Place these files in the openfn/adaptors/workflows/sendVaroPackage folder.
+
+### workflow.js
+```js
+{
+  "workflow": {
+    "steps": [
+      {
+        "id": "convertRecordsToMessageContent",
+        "adaptor": "varo",
+        "expression": "jobVaro.js",
+        "next": {
+          "sendMessage": "state.data != null"
+        }
+      },
+      {
+        "id": "sendMessage",
+        "adaptor": "gmail",
+        "expression": "jobGmail.js",
+        "configuration": {
+          "access_token": "[redacted]"
+        }
+      }
+    ]
+  }
+}
+```
+
+### jobVaro.js
+
+```js
+// Pull the data from the collection into the state.
+collections.get("ccdx-ems", "*");
+
+// Convert the raw collection records into an EMS report.
+convertRecordsToReport($.data);
+
+// Convert the EMS report to message contents (subject, data file).
+convertReportToMessageContent($.data, "ems");
+```
+
+### jobGmail.js
+
+```js
+fn((state) => {
+  const { subject, data } = state.data;
+
+  /*
+    subject = "OpenFn | EMS";
+    data = {
+      filename: "data.json", 
+      content: "{ ... EMS report converted to JSON string ... }",
+    };
+  */
+
+  // Compress the data.json file into a ZIP file.
+  const dataArchive = {
+    filename: "data.zip",
+    archive: [data],
+  };
+
+  state.data = {
+    to: "[email protected]",
+    subject,
+    attachments: [dataArchive],
+  };
+
+  return state;
+});
+
+sendMessage($.data);
+```
+
+# Running workflow
+
+We can look in more detail at the Gmail -> EMS report workflow and illustrate some advanced techniques to enhance the development experience and operation.
+
+## Basics
 
 The `-m` flag tells OpenFn to use the monorepo instead of its own adaptor cache.
 
 ```
-cd openfn/adaptors/workflows
+cd openfn/adaptors/workflows/readVaroPackages
 openfn workflow.json -m
 ```
 
 ## Advanced workflow operation
 
 It's beneficial to isolate the Varo adaptor during development to avoid the redundant step of repeatedly querying Gmail which also requires refreshing the access token each hour. We can use advanced functionality of the OpenFn CLI to cache the output of the Gmail step which will enable us reuse its output while we are enhancing the Varo adaptor.
 
-### Cache Gmail adaptor output
+### Cache the output from a step
 
 We can configure the workflow to retrieve this message content from a local file which will bypass the need to use the Gmail adaptor to retrieve this information.
 
@@ -153,7 +226,7 @@ We can configure the workflow to retrieve this message content from a local file
 openfn workflow.json -m --cache-steps --only getContentsFromMessages
 ```
 
-### Running the workflow with cached Gmail adaptor output
+### Running with a cached step
 
 - `-m` Use the monorepo.
 - `--only convertToEms` Execute only the convertToEms step.
@@ -162,6 +235,22 @@ openfn workflow.json -m --cache-steps --only getContentsFromMessages
 openfn workflow.json -m --only convertToEms
 ```
 
+# Token configuration
+
+Some workflows required authorization to access the resources.
+
+## Gmail token
+
+Use Postman to retrieve an access token. This is a short-lived token will last 60 minutes and will have to be manually retrieved. See the documentation in the [Gmail adaptor readme](https://docs.openfn.org/adaptors/packages/gmail-readme#use-the-postman-application-to-query-the-oauth-enpoint-and-retrieve-an-access-token) for a guide on how to configure Postman to retrieve the access token.
+
+## OpenFn collections token
+
+A workflow may track the previously processed messages by storing the processed IDs in an OpenFn collection. OpenFn requires authorization to access a given collection.
+
+1. Access the [Collections](http://localhost:4000/settings/collections) configuration in the administration area.
+1. Ensure the collection named `gmail-processed-ids` exists.
+1. Create a new token in the [Personal Access Tokens](http://localhost:4000/profile/tokens) configuration.
+
 # Enhancing/developing the Varo adaptor
 
 These instructions will illustrate how to install OpenFn and the adaptor monorepo. The monorepo gives you access to in-development versions of the adaptors.
@@ -176,7 +265,6 @@ openfn
     └── workflows
 ```
 
-
 ## Install OpenFn
 
 Prerequisite is Docker in installed, up-to-date, and running on your computer.

packages/varo/ast.json

@@ -47,6 +47,111 @@
         ]
       },
       "valid": true
+    },
+    {
+      "name": "convertRecordsToReport",
+      "params": [
+        "records"
+      ],
+      "docs": {
+        "description": "Read a collection of EMS-like data records and convert them to a standard EMS report/record format.\nSystematically separates report properties from record properties.",
+        "tags": [
+          {
+            "title": "public",
+            "description": null,
+            "type": null
+          },
+          {
+            "title": "function",
+            "description": null,
+            "name": null
+          },
+          {
+            "title": "param",
+            "description": "Array of EMS-like JSON objects.",
+            "type": {
+              "type": "NameExpression",
+              "name": "Array"
+            },
+            "name": "records"
+          },
+          {
+            "title": "state",
+            "description": "{Array} data - The converted, EMS-compliant report with records."
+          },
+          {
+            "title": "returns",
+            "description": null,
+            "type": {
+              "type": "NameExpression",
+              "name": "Operation"
+            }
+          },
+          {
+            "title": "example",
+            "description": "convertRecordsToReport(\n  [\n    { \"ASER\": \"BJBC 08 30\", \"ABST\": \"20241205T004440Z\", \"TVC\": 5.0 },\n    { \"ASER\": \"BJBC 08 30\", \"ABST\": \"20241205T005440Z\", \"TVC\": 5.2 },\n  ]\n);\n\nstate.data becomes:\n{\n  \"ASER\": \"BJBC 08 30\",\n  records: [\n    { \"ABST\": \"20241205T004440Z\", \"TVC\": 5.0 },\n    { \"ABST\": \"20241205T005440Z\", \"TVC\": 5.2 },\n  ],\n}",
+            "caption": "Convert data to EMS-compliant data."
+          }
+        ]
+      },
+      "valid": true
+    },
+    {
+      "name": "convertReportToMessageContent",
+      "params": [
+        "report",
+        "reportType"
+      ],
+      "docs": {
+        "description": "Converts an EMS-compliant report into Varo-compatible message components.",
+        "tags": [
+          {
+            "title": "public",
+            "description": null,
+            "type": null
+          },
+          {
+            "title": "function",
+            "description": null,
+            "name": null
+          },
+          {
+            "title": "param",
+            "description": "EMS-compliant report object.",
+            "type": {
+              "type": "NameExpression",
+              "name": "Object"
+            },
+            "name": "report"
+          },
+          {
+            "title": "param",
+            "description": "Optional. Source of the report, e.g., \"ems\" or \"rtmd\".",
+            "type": {
+              "type": "OptionalType",
+              "expression": {
+                "type": "NameExpression",
+                "name": "string"
+              }
+            },
+            "name": "reportType",
+            "default": "'unknown'"
+          },
+          {
+            "title": "returns",
+            "description": "An operation function that receives `state` and returns updated message content.",
+            "type": {
+              "type": "NameExpression",
+              "name": "Function"
+            }
+          },
+          {
+            "title": "example",
+            "description": "// Convert EMS-compliant record to Varo message components.\nconvertReportToMessageContent(emsReport, \"ems\");"
+          }
+        ]
+      },
+      "valid": true
     }
   ],
   "exports": [],