[IT-4485] Add endpoint to emit current balances as CSV

.pre-commit-config.yaml

@@ -17,7 +17,7 @@ repos:
     hooks:
     -   id: yamllint
 -   repo: https://github.com/awslabs/cfn-python-lint
-    rev: v1.38.1
+    rev: v1.37.1
     hooks:
     -   id: cfn-python-lint
         files: template\.(json|yml|yaml)$

README.md

@@ -1,18 +1,19 @@
 # lambda-mips-api
 
-An AWS Lambda microservice presenting MIPS chart of accounts data
+An AWS Lambda microservice providing limited read-only access to MIP Cloud.
 
 ## Architecture
 
-This microservice is designed to retrieve a chart of accounts from a third-party
-API and present the data in a useful format.
+This microservice is designed to retrieve data using the MIP Cloud API and
+present the data in a useful format.
 
 Formats available:
 
-| API Route | Description                                                              |
-| --------- | ------------------------------------------------------------------------ |
-| /accounts | A dictionary mapping the chart of accounts to their friendly names.      |
-| /tags     | A list of valid tag values for either `CostCenter` or `CostCenterOther`. |
+| API Route | Description                                                                 |
+| --------- | --------------------------------------------------------------------------- |
+| /accounts | A dictionary mapping the chart of program accounts to their friendly names. |
+| /balances | A CSV listing GL account balances, appropriate for FloQast consumption.     |
+| /tags     | A list of valid tag values for either `CostCenter` or `CostCenterOther`.    |
 
 Since we reach out to a third-party API across the internet, responses are
 cached to minimize interaction with the API and mitigate potential environmental
@@ -28,7 +29,7 @@ data to be stored in Cloudfront for a default of one day.
 In the event of a cache hit, Cloudfront will return the cached value without
 triggering an API gateway event.
 
-### Default Behavior
+### Chart of Accounts Behavior
 
 By default, the lambda will process the chart of accounts received to remove
 inactive codes, deduplicate the significant portion of active codes, and add a
@@ -41,6 +42,15 @@ the `CodesToOmit` template parameter. Remaining codes will be returned in
 numeric order as either a list of strings or a json dictionary depending on the
 API route.
 
+### Current Balances
+
+When retrieving a CSV of current balances, the full chart of accounts will be
+processed according to query-string parameters, and then collated with MIP trial
+balance data.
+
+During the first week of the month, the balance will be calculated for the
+previous month; otherwise a month-to-date balance will be calculated.
+
 ### Required Secure Parameters
 
 User credentials for logging in to the finance system are stored as secure
@@ -65,24 +75,31 @@ environment:
 
 | Template Parameter | Environment Variable | Description                                        |
 | ------------------ | -------------------- | -------------------------------------------------- |
-| MipsOrganization   | MipsOrg              | Log in to this organization in the finance system. |
+| MipOrganization    | MipOrg               | Log in to this organization in the finance system. |
 | SsmParamPrefix     | SsmPath              | Path prefix for required secure parameters.        |
 | CodesToOmit        | CodesToOmit          | List of numeric codes to remove from output.       |
 | NoProgramCode      | NoProgramCode        | Numeric code to use for "No Program" entry.        |
 | OtherCode          | OtherCode            | Numeric code to use for "Other" entry.             |
 
 ### Query String Parameters
 
-Several query-string parameters are available for either endpoint to configure
-response output.
+Several query-string parameters are shared by the endpoints to configure the
+list of accounts in the response output. All query string parameters except for
+`year_to_date` effect the `/accounts` and `/tags` endpoints. The only parameters
+that effect the `/balances` endpoint are `show_inactive_codes` and
+`target_date`.
 
-| Query String Parameter | Allowed Values                        |
-| ---------------------- | ------------------------------------- |
-| show_other_code        | "on" or "yes" or "true"               |
-| hide_no_program_code   | "on" or "yes" or "true"               |
-| show_inactive_codes    | "on" or "yes" or "true"               |
-| priority_codes         | Comma-separated list of numeric codes |
-| limit                  | Integer                               |
+| Query String Parameter | Allowed Values                        | Default       | Supported Endpoints               |
+| ---------------------- | ------------------------------------- | ------------- | --------------------------------- |
+| show_other_code        | Boolean value                         | False         | `/accounts`, `/tags`              |
+| hide_no_program_code   | Boolean value                         | False         | `/accounts`, `/tags`              |
+| show_inactive_codes    | Boolean value                         | False         | `/accounts`, `/balances`, `/tags` |
+| priority_codes         | Comma-separated list of numeric codes | Empty list    | `/accounts`, `/tags`              |
+| limit                  | Integer                               | 0 (unlimited) | `/accounts`, `/tags`              |
+| target_date            | ISO 8601 string                       | Today         | `/balances`                       |
+
+Boolean values: any value other than "no", "off", or "false" will be interpreted
+as true, including the empty string.
 
 A `show_other_code` parameter is available to optionally include an "Other"
 entry in the output with a value from the `OtherCode` parameter. Defining any
@@ -105,6 +122,9 @@ prioritized when included. Example value: `123456,654321`.
 A `limit` parameter is available to restrict the number of items returned. This
 value must be a positive integer, a value of zero disables the parameter.
 
+A `target_date` parameter is available to calculate a balance period from a day
+other than today.
+
 ### Triggering
 
 The CloudFormation template will output all available endpoint URLs for

mip_api/__init__.py

@@ -1,15 +1,15 @@
-import json
 import logging
 
-from mip_api import chart, s3, ssm, upstream, util
+from mip_api import balances, chart, s3, ssm, upstream, util
 
 
 LOG = logging.getLogger(__name__)
 LOG.setLevel(logging.DEBUG)
 
 
 def lambda_handler(event, context):
-    """Entry Point for Lambda
+    """
+    Entry Point for Lambda
 
     Collect configuration from environment variables and query-string parameters,
     determine data requested based on the API endpoint called, and finally
@@ -38,68 +38,115 @@ def lambda_handler(event, context):
         Return doc: https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-lambda-proxy-integrations.html
     """
 
-    # helper functions to encapsulate the body, headers, and status code
-    def _build_return(code, body):
-        return {
-            "statusCode": code,
-            "body": json.dumps(body, indent=2),
-        }
-
     try:
         # collect environment variables
-        mip_org = util.get_os_var("MipsOrg")
+        mip_org = util.get_os_var("MipOrg")
         ssm_path = util.get_os_var("SsmPath")
         s3_bucket = util.get_os_var("CacheBucket")
-        s3_path = util.get_os_var("CacheBucketPath")
+        s3_prefix = util.get_os_var("CacheBucketPrefix")
 
         code_other = util.get_os_var("OtherCode")
         code_no_program = util.get_os_var("NoProgramCode")
 
-        api_routes = {
-            "ApiChartOfAccounts": util.get_os_var("ApiChartOfAccounts"),
-            "ApiValidTags": util.get_os_var("ApiValidTags"),
-        }
+        api_route_coa = util.get_os_var("ApiChartOfAccounts")
+        api_route_tags = util.get_os_var("ApiValidTags")
+        api_route_balances = util.get_os_var("ApiTrialBalances")
 
         _to_omit = util.get_os_var("CodesToOmit")
         omit_codes_list = util.parse_codes(_to_omit)
 
+        # collect query-string parameters
+        params = util.params_dict(event)
+        hide_inactive = params["hide_inactive"]
+
+        # build S3 cache paths, with separate paths for each combination
+        # of endpoint and relevant parameters
+        s3_path_gl_coa = s3_prefix + "gl-coa"
+        if not hide_inactive:
+            s3_path_gl_coa += "-full"
+        s3_path_gl_coa += ".json"
+
+        s3_path_program_coa = s3_prefix + "program-coa"
+        if not hide_inactive:
+            s3_path_program_coa += "-full"
+        s3_path_program_coa += ".json"
+
+        s3_path_balances = s3_prefix + "balances"
+        if not hide_inactive:
+            s3_path_balances += "-full"
+        s3_path_balances += ".json"
+
         # get secure parameters
         ssm_secrets = ssm.get_secrets(ssm_path)
 
-        # get chart of accounts from mip cloud
-        raw_chart = chart.get_chart(mip_org, ssm_secrets, s3_bucket, s3_path)
-        LOG.debug(f"Raw chart data: {raw_chart}")
-
-        # collect query-string parameters
-        params = {}
-        if "queryStringParameters" in event:
-            params = event["queryStringParameters"]
-            LOG.debug(f"Query-string parameters: {params}")
-
         # parse the path and return appropriate data
         if "path" in event:
             event_path = event["path"]
 
-            # always process the chart of accounts
-            mip_chart = chart.process_chart(
-                params, raw_chart, omit_codes_list, code_other, code_no_program
+            if event_path == api_route_balances:
+                # get chart of general ledger accounts
+                gl_chart = chart.get_gl_chart(
+                    mip_org,
+                    ssm_secrets,
+                    s3_bucket,
+                    s3_path_gl_coa,
+                    hide_inactive,
+                )
+                LOG.debug(f"Raw chart data: {gl_chart}")
+
+                # get balance data
+                raw_bal = balances.get_balances(
+                    mip_org,
+                    ssm_secrets,
+                    s3_bucket,
+                    s3_path_balances,
+                    params["date"],
+                )
+
+                # combine them into CSV output
+                balances_csv = balances.format_csv(raw_bal, gl_chart)
+                return util.build_return_text(200, balances_csv)
+
+            # common processing for '/accounts' and '/tags'
+
+            # get chart of Program accounts
+            _raw_program_chart = chart.get_program_chart(
+                mip_org,
+                ssm_secrets,
+                s3_bucket,
+                s3_path_program_coa,
+                hide_inactive,
             )
+            LOG.debug(f"Raw chart data: {_raw_program_chart}")
+
+            # always process the chart of Program accounts
+            _program_chart = chart.process_chart(
+                _raw_program_chart,
+                omit_codes_list,
+                code_other,
+                code_no_program,
+                params,
+            )
+
+            # always limit the size of the chart
+            program_chart = chart.limit_chart(_program_chart, params["limit"])
 
-            if event_path == api_routes["ApiChartOfAccounts"]:
-                # conditionally limit the size of the output
-                return_chart = chart.limit_chart(params, mip_chart)
-                return _build_return(200, return_chart)
+            if event_path == api_route_coa:
+                # no more processing, return chart
+                return util.build_return_json(200, program_chart)
 
-            elif event_path == api_routes["ApiValidTags"]:
+            elif event_path == api_route_tags:
                 # build a list of strings from the processed dictionary
-                valid_tags = chart.list_tags(params, mip_chart)
-                return _build_return(200, valid_tags)
+                valid_tags = chart.list_tags(program_chart)
+                return util.build_return_json(200, valid_tags)
 
-            else:
-                return _build_return(404, {"error": "Invalid request path"})
+            else:  # unknown API endpoint
+                return util.build_return_json(404, {"error": "Invalid request path"})
 
-        return _build_return(400, {"error": f"Invalid event: No path found: {event}"})
+        return util.build_return_json(
+            400, {"error": f"Invalid event: No path found: {event}"}
+        )
 
     except Exception as exc:
         LOG.exception(exc)
-        return _build_return(500, {"error": str(exc)})
+        return util.build_return_json(500, {"error": str(exc)})