Skip to content

Commit 93d6e4e

Browse files
ConsoleCatzirlConsoleCatzirl
committed
[IT-4485] Add endpoint to show GL account balances
Add a new API endpoint `/balances` to output a CSV of GL account balances appropriate for sending to FloQast. During the first week of the month, process the balance for the previous month, otherwise process the month-to-date balance. The pre-existing endpoints used the Program segment of the chart of accounts, but the new endpoint uses the GL segment of the chart of accounts. Refactor fetching the chart of accounts to support arbitrary segments.
1 parent 49bb9b3 commit 93d6e4e

File tree

12 files changed

+1238
-543
lines changed

12 files changed

+1238
-543
lines changed

‎Pipfile.lock‎

Lines changed: 308 additions & 296 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎README.md‎

Lines changed: 38 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -1,18 +1,19 @@
11
# lambda-mips-api
22

3-
An AWS Lambda microservice presenting MIPS chart of accounts data
3+
An AWS Lambda microservice providing limited read-only access to MIP Cloud.
44

55
## Architecture
66

7-
This microservice is designed to retrieve a chart of accounts from a third-party
8-
API and present the data in a useful format.
7+
This microservice is designed to retrieve data using the MIP Cloud API and
8+
present the data in a useful format.
99

1010
Formats available:
1111

12-
| API Route | Description |
13-
| --------- | ------------------------------------------------------------------------ |
14-
| /accounts | A dictionary mapping the chart of accounts to their friendly names. |
15-
| /tags | A list of valid tag values for either `CostCenter` or `CostCenterOther`. |
12+
| API Route | Description |
13+
| --------- | --------------------------------------------------------------------------- |
14+
| /accounts | A dictionary mapping the chart of program accounts to their friendly names. |
15+
| /balances | A CSV listing GL account balances, appropriate for FloQast consumption. |
16+
| /tags | A list of valid tag values for either `CostCenter` or `CostCenterOther`. |
1617

1718
Since we reach out to a third-party API across the internet, responses are
1819
cached to minimize interaction with the API and mitigate potential environmental
@@ -28,7 +29,7 @@ data to be stored in Cloudfront for a default of one day.
2829
In the event of a cache hit, Cloudfront will return the cached value without
2930
triggering an API gateway event.
3031

31-
### Default Behavior
32+
### Chart of Accounts Behavior
3233

3334
By default, the lambda will process the chart of accounts received to remove
3435
inactive codes, deduplicate the significant portion of active codes, and add a
@@ -41,6 +42,15 @@ the `CodesToOmit` template parameter. Remaining codes will be returned in
4142
numeric order as either a list of strings or a json dictionary depending on the
4243
API route.
4344

45+
### Current Balances
46+
47+
When retrieving a CSV of current balances, the full chart of accounts will be
48+
processed according to query-string parameters, and then collated with MIP trial
49+
balance data.
50+
51+
During the first week of the month, the balance will be calculated for the
52+
previous month; otherwise a month-to-date balance will be calculated.
53+
4454
### Required Secure Parameters
4555

4656
User credentials for logging in to the finance system are stored as secure
@@ -65,24 +75,31 @@ environment:
6575

6676
| Template Parameter | Environment Variable | Description |
6777
| ------------------ | -------------------- | -------------------------------------------------- |
68-
| MipsOrganization | MipsOrg | Log in to this organization in the finance system. |
78+
| MipOrganization | MipOrg | Log in to this organization in the finance system. |
6979
| SsmParamPrefix | SsmPath | Path prefix for required secure parameters. |
7080
| CodesToOmit | CodesToOmit | List of numeric codes to remove from output. |
7181
| NoProgramCode | NoProgramCode | Numeric code to use for "No Program" entry. |
7282
| OtherCode | OtherCode | Numeric code to use for "Other" entry. |
7383

7484
### Query String Parameters
7585

76-
Several query-string parameters are available for either endpoint to configure
77-
response output.
86+
Several query-string parameters are shared by the endpoints to configure the
87+
list of accounts in the response output. All query string parameters except for
88+
`year_to_date` effect the `/accounts` and `/tags` endpoints. The only parameters
89+
that effect the `/balances` endpoint are `show_inactive_codes` and
90+
`target_date`.
7891

79-
| Query String Parameter | Allowed Values |
80-
| ---------------------- | ------------------------------------- |
81-
| show_other_code | "on" or "yes" or "true" |
82-
| hide_no_program_code | "on" or "yes" or "true" |
83-
| show_inactive_codes | "on" or "yes" or "true" |
84-
| priority_codes | Comma-separated list of numeric codes |
85-
| limit | Integer |
92+
| Query String Parameter | Allowed Values | Default | Supported Endpoints |
93+
| ---------------------- | ------------------------------------- | ------------- | --------------------------------- |
94+
| show_other_code | Boolean value | False | `/accounts`, `/tags` |
95+
| hide_no_program_code | Boolean value | False | `/accounts`, `/tags` |
96+
| show_inactive_codes | Boolean value | False | `/accounts`, `/balances`, `/tags` |
97+
| priority_codes | Comma-separated list of numeric codes | Empty list | `/accounts`, `/tags` |
98+
| limit | Integer | 0 (unlimited) | `/accounts`, `/tags` |
99+
| target_date | ISO 8601 string | Today | `/balances` |
100+
101+
Boolean values: any value other than "no", "off", or "false" will be interpreted
102+
as true, including the empty string.
86103

87104
A `show_other_code` parameter is available to optionally include an "Other"
88105
entry in the output with a value from the `OtherCode` parameter. Defining any
@@ -105,6 +122,9 @@ prioritized when included. Example value: `123456,654321`.
105122
A `limit` parameter is available to restrict the number of items returned. This
106123
value must be a positive integer, a value of zero disables the parameter.
107124

125+
A `target_date` parameter is available to calculate a balance period from a day
126+
other than today.
127+
108128
### Triggering
109129

110130
The CloudFormation template will output all available endpoint URLs for

‎docs/lambda-mips-api_components.drawio.png‎

-16.8 KB
Loading

‎docs/lambda-mips-api_components.drawio.svg‎

Lines changed: 1 addition & 1 deletion
Loading

‎mip_api/__init__.py‎

Lines changed: 81 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,6 @@
1-
import json
21
import logging
32

4-
from mip_api import chart, s3, ssm, upstream, util
3+
from mip_api import balances, chart, s3, ssm, upstream, util
54

65

76
LOG = logging.getLogger(__name__)
@@ -38,68 +37,116 @@ def lambda_handler(event, context):
3837
Return doc: https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-lambda-proxy-integrations.html
3938
"""
4039

41-
# helper functions to encapsulate the body, headers, and status code
42-
def _build_return(code, body):
43-
return {
44-
"statusCode": code,
45-
"body": json.dumps(body, indent=2),
46-
}
47-
4840
try:
4941
# collect environment variables
50-
mip_org = util.get_os_var("MipsOrg")
42+
mip_org = util.get_os_var("MipOrg")
5143
ssm_path = util.get_os_var("SsmPath")
5244
s3_bucket = util.get_os_var("CacheBucket")
53-
s3_path = util.get_os_var("CacheBucketPath")
45+
s3_prefix = util.get_os_var("CacheBucketPrefix")
5446

5547
code_other = util.get_os_var("OtherCode")
5648
code_no_program = util.get_os_var("NoProgramCode")
5749

5850
api_routes = {
5951
"ApiChartOfAccounts": util.get_os_var("ApiChartOfAccounts"),
6052
"ApiValidTags": util.get_os_var("ApiValidTags"),
53+
"ApiTrialBalances": util.get_os_var("ApiTrialBalances"),
6154
}
6255

6356
_to_omit = util.get_os_var("CodesToOmit")
6457
omit_codes_list = util.parse_codes(_to_omit)
6558

66-
# get secure parameters
67-
ssm_secrets = ssm.get_secrets(ssm_path)
59+
# collect query-string parameters
60+
params = util.params_dict(event)
6861

69-
# get chart of accounts from mip cloud
70-
raw_chart = chart.get_chart(mip_org, ssm_secrets, s3_bucket, s3_path)
71-
LOG.debug(f"Raw chart data: {raw_chart}")
62+
# build S3 cache paths, with separate paths for each combination
63+
# of endpoint and relevant parameters
64+
s3_path_gl_coa = s3_prefix + "gl-coa"
65+
if not params["hide_inactive"]:
66+
s3_path_gl_coa += "-full"
67+
s3_path_gl_coa += ".json"
7268

73-
# collect query-string parameters
74-
params = {}
75-
if "queryStringParameters" in event:
76-
params = event["queryStringParameters"]
77-
LOG.debug(f"Query-string parameters: {params}")
69+
s3_path_program_coa = s3_prefix + "program-coa"
70+
if not params["hide_inactive"]:
71+
s3_path_program_coa += "-full"
72+
s3_path_program_coa += ".json"
73+
74+
s3_path_balances = s3_prefix + "balances"
75+
if not params["hide_inactive"]:
76+
s3_path_balances += "-full"
77+
s3_path_balances += ".json"
78+
79+
# get secure parameters
80+
ssm_secrets = ssm.get_secrets(ssm_path)
7881

7982
# parse the path and return appropriate data
8083
if "path" in event:
8184
event_path = event["path"]
8285

83-
# always process the chart of accounts
84-
mip_chart = chart.process_chart(
85-
params, raw_chart, omit_codes_list, code_other, code_no_program
86-
)
86+
if event_path == api_routes["ApiTrialBalances"]:
87+
# get chart of GL accounts
88+
gl_chart = chart.get_gl_chart(
89+
mip_org,
90+
ssm_secrets,
91+
s3_bucket,
92+
s3_path_gl_coa,
93+
params["hide_inactive"],
94+
)
95+
LOG.debug(f"Raw chart data: {gl_chart}")
96+
97+
# get upstream balance data
98+
raw_bal = balances.get_balances(
99+
mip_org,
100+
ssm_secrets,
101+
s3_bucket,
102+
s3_path_balances,
103+
params["date"],
104+
)
105+
106+
# combine them into CSV output
107+
balances_csv = balances.format_csv(raw_bal, gl_chart)
108+
return util.build_return_text(200, balances_csv)
109+
110+
else: # common processing for '/accounts' and '/tags'
111+
112+
# get chart of Program accounts
113+
_raw_program_chart = chart.get_program_chart(
114+
mip_org,
115+
ssm_secrets,
116+
s3_bucket,
117+
s3_path_program_coa,
118+
params["hide_inactive"],
119+
)
120+
LOG.debug(f"Raw chart data: {_raw_program_chart}")
121+
122+
# always process the chart of Program accounts
123+
_program_chart = chart.process_chart(
124+
_raw_program_chart,
125+
omit_codes_list,
126+
code_other,
127+
code_no_program,
128+
params,
129+
)
130+
131+
# always limit the size of the chart
132+
program_chart = chart.limit_chart(_program_chart, params["limit"])
87133

88134
if event_path == api_routes["ApiChartOfAccounts"]:
89-
# conditionally limit the size of the output
90-
return_chart = chart.limit_chart(params, mip_chart)
91-
return _build_return(200, return_chart)
135+
# no more processing, return chart
136+
return util.build_return_json(200, program_chart)
92137

93138
elif event_path == api_routes["ApiValidTags"]:
94139
# build a list of strings from the processed dictionary
95-
valid_tags = chart.list_tags(params, mip_chart)
96-
return _build_return(200, valid_tags)
140+
valid_tags = chart.list_tags(program_chart)
141+
return util.build_return_json(200, valid_tags)
97142

98-
else:
99-
return _build_return(404, {"error": "Invalid request path"})
143+
else: # unknown API endpoint
144+
return util.build_return_json(404, {"error": "Invalid request path"})
100145

101-
return _build_return(400, {"error": f"Invalid event: No path found: {event}"})
146+
return util.build_return_json(
147+
400, {"error": f"Invalid event: No path found: {event}"}
148+
)
102149

103150
except Exception as exc:
104151
LOG.exception(exc)
105-
return _build_return(500, {"error": str(exc)})
152+
return util.build_return_json(500, {"error": str(exc)})

‎mip_api/balances.py‎

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
import logging
2+
3+
from mip_api import s3, upstream
4+
5+
import csv
6+
import io
7+
8+
LOG = logging.getLogger(__name__)
9+
LOG.setLevel(logging.DEBUG)
10+
11+
12+
def get_balances(org_name, secrets, bucket, path, when=None):
13+
# get the upstream API response
14+
LOG.info("Read balances from upstream API")
15+
upstream_dict = upstream.trial_balances(org_name, secrets, when)
16+
LOG.debug(f"Upstream API response: {upstream_dict}")
17+
18+
bal_dict = s3.cache(upstream_dict, bucket, path)
19+
return bal_dict
20+
21+
22+
def process_balance(bal_dict, coa_dict):
23+
24+
# check for success
25+
if "executionResult" not in bal_dict:
26+
LOG.error(f"No execution result found: '{bal_dict}'")
27+
raise KeyError("No 'executionResult' key found")
28+
29+
result = bal_dict["executionResult"]
30+
if result != "SUCCESS":
31+
LOG.error(f"Execution result is not 'SUCCESS': '{result}'")
32+
raise ValueError("Execution result is not 'SUCCESS'")
33+
34+
# collate api response into a dict
35+
_data = {}
36+
37+
_detail = []
38+
for k, v in bal_dict["extraInformation"].items():
39+
if k != "Level1":
40+
LOG.error(f"Unexpected key (not 'Level1'): {k}")
41+
raise KeyError("No 'Level1' key found")
42+
else:
43+
_detail = v
44+
45+
for d in _detail:
46+
account_id = d["DBDETAIL_SUM_SEGMENT_N0"]
47+
if account_id not in _data:
48+
_data[account_id] = {}
49+
50+
if d["DBDETAIL_SUM_TYPE"] == 1:
51+
_data[account_id]["balance_start"] = d["DBDETAIL_SUM_POSTEDAMT"]
52+
elif d["DBDETAIL_SUM_TYPE"] == 2:
53+
_data[account_id]["activity"] = d["DBDETAIL_SUM_POSTEDAMT"]
54+
elif d["DBDETAIL_SUM_TYPE"] == 3:
55+
_data[account_id]["balance_end"] = d["DBDETAIL_SUM_POSTEDAMT"]
56+
else:
57+
LOG.error(f"Unknown balance type: {d['DBDETAIL_SUM_DESC']}")
58+
59+
LOG.debug(f"Raw internal balance dict: {_data}")
60+
61+
# List of rows in CSV
62+
out_rows = []
63+
64+
# Add header row
65+
headers = [
66+
"AccountNumber",
67+
"AccountName",
68+
"PeriodStart",
69+
"PeriodEnd",
70+
"StartBalance",
71+
"Activity",
72+
"EndBalance",
73+
]
74+
out_rows.append(headers)
75+
76+
# Generate rows from input dict
77+
for k, v in _data.items():
78+
if k not in coa_dict:
79+
LOG.error(f"Key {k} not found in chart of accounts")
80+
LOG.debug(f"List of accounts: {coa_dict.keys()}")
81+
continue
82+
name = coa_dict[k]
83+
84+
row = [
85+
k,
86+
name,
87+
bal_dict["period_from"],
88+
bal_dict["period_to"],
89+
v["balance_start"],
90+
v["activity"],
91+
v["balance_end"],
92+
]
93+
out_rows.append(row)
94+
95+
return out_rows
96+
97+
98+
def format_csv(bal_dict, coa_dict):
99+
csv_out = io.StringIO()
100+
csv_writer = csv.writer(csv_out)
101+
102+
csv_rows = process_balance(bal_dict, coa_dict)
103+
for row in csv_rows:
104+
csv_writer.writerow(row)
105+
106+
return csv_out.getvalue()

0 commit comments

Comments
 (0)