Skip to content

Commit 804a4c6

Browse files
ConsoleCatzirlConsoleCatzirl
committed
[IT-4485] Add endpoint to show GL account balances
Add a new API endpoint `/balances` to output a CSV of GL account balances appropriate for sending to FloQast. During the first week of the month, process the balance for the previous month, otherwise process the month-to-date balance.
1 parent da73f45 commit 804a4c6

File tree

9 files changed

+675
-46
lines changed

9 files changed

+675
-46
lines changed

‎README.md‎

Lines changed: 37 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -1,18 +1,19 @@
11
# lambda-mips-api
22

3-
An AWS Lambda microservice presenting MIPS chart of accounts data
3+
An AWS Lambda microservice providing limited read-only access to MIP Cloud.
44

55
## Architecture
66

7-
This microservice is designed to retrieve a chart of accounts from a third-party
8-
API and present the data in a useful format.
7+
This microservice is designed to retrieve data using the MIP Cloud API and
8+
present the data in a useful format.
99

1010
Formats available:
1111

12-
| API Route | Description |
13-
| --------- | ------------------------------------------------------------------------ |
14-
| /accounts | A dictionary mapping the chart of accounts to their friendly names. |
15-
| /tags | A list of valid tag values for either `CostCenter` or `CostCenterOther`. |
12+
| API Route | Description |
13+
| --------- | --------------------------------------------------------------------------- |
14+
| /accounts | A dictionary mapping the chart of program accounts to their friendly names. |
15+
| /balances | A CSV listing GL account balances, appropriate for FloQast consumption. |
16+
| /tags | A list of valid tag values for either `CostCenter` or `CostCenterOther`. |
1617

1718
Since we reach out to a third-party API across the internet, responses are
1819
cached to minimize interaction with the API and mitigate potential environmental
@@ -28,7 +29,7 @@ data to be stored in Cloudfront for a default of one day.
2829
In the event of a cache hit, Cloudfront will return the cached value without
2930
triggering an API gateway event.
3031

31-
### Default Behavior
32+
### Chart of Accounts Behavior
3233

3334
By default, the lambda will process the chart of accounts received to remove
3435
inactive codes, deduplicate the significant portion of active codes, and add a
@@ -41,6 +42,15 @@ the `CodesToOmit` template parameter. Remaining codes will be returned in
4142
numeric order as either a list of strings or a json dictionary depending on the
4243
API route.
4344

45+
### Current Balances
46+
47+
When retrieving a CSV of current balances, the full chart of accounts will be
48+
processed according to query-string parameters, and then collated with MIP trial
49+
balance data.
50+
51+
During the first week of the month, the balance will be calculated for the
52+
previous month; otherwise a month-to-date balance will be calculated.
53+
4454
### Required Secure Parameters
4555

4656
User credentials for logging in to the finance system are stored as secure
@@ -73,16 +83,23 @@ environment:
7383

7484
### Query String Parameters
7585

76-
Several query-string parameters are available for either endpoint to configure
77-
response output.
86+
Several query-string parameters are shared by the endpoints to configure the
87+
list of accounts in the response output. All query string parameters except for
88+
`year_to_date` effect the `/accounts` and `/tags` endpoints. The only parameters
89+
that effect the `/balances` endpoint are `show_inactive_codes` and
90+
`target_date`.
7891

79-
| Query String Parameter | Allowed Values |
80-
| ---------------------- | ------------------------------------- |
81-
| show_other_code | "on" or "yes" or "true" |
82-
| hide_no_program_code | "on" or "yes" or "true" |
83-
| show_inactive_codes | "on" or "yes" or "true" |
84-
| priority_codes | Comma-separated list of numeric codes |
85-
| limit | Integer |
92+
| Query String Parameter | Allowed Values | Default | Supported Endpoints |
93+
| ---------------------- | ------------------------------------- | ------------- | --------------------------------- |
94+
| show_other_code | Boolean value | False | `/accounts`, `/tags` |
95+
| hide_no_program_code | Boolean value | False | `/accounts`, `/tags` |
96+
| show_inactive_codes | Boolean value | False | `/accounts`, `/balances`, `/tags` |
97+
| priority_codes | Comma-separated list of numeric codes | Empty list | `/accounts`, `/tags` |
98+
| limit | Integer | 0 (unlimited) | `/accounts`, `/tags` |
99+
| target_date | ISO 8601 string | Today | `/balances` |
100+
101+
Boolean values: any value other than "no", "off", or "false" will be interpreted
102+
as true, including the empty string.
86103

87104
A `show_other_code` parameter is available to optionally include an "Other"
88105
entry in the output with a value from the `OtherCode` parameter. Defining any
@@ -105,6 +122,9 @@ prioritized when included. Example value: `123456,654321`.
105122
A `limit` parameter is available to restrict the number of items returned. This
106123
value must be a positive integer, a value of zero disables the parameter.
107124

125+
A `target_date` parameter is available to calculate a balance period from a day
126+
other than today.
127+
108128
### Triggering
109129

110130
The CloudFormation template will output all available endpoint URLs for

‎docs/lambda-mips-api_components.drawio.png‎

-16.8 KB
Loading

‎docs/lambda-mips-api_components.drawio.svg‎

Lines changed: 1 addition & 1 deletion
Loading

‎mip_api/__init__.py‎

Lines changed: 54 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,6 @@
1-
import json
21
import logging
32

4-
from mip_api import chart, s3, ssm, upstream, util
3+
from mip_api import balances, chart, s3, ssm, upstream, util
54

65

76
LOG = logging.getLogger(__name__)
@@ -51,6 +50,7 @@ def lambda_handler(event, context):
5150

5251
api_route_coa = util.get_os_var("ApiChartOfAccounts")
5352
api_route_tags = util.get_os_var("ApiValidTags")
53+
api_route_balances = util.get_os_var("ApiTrialBalances")
5454

5555
_to_omit = util.get_os_var("CodesToOmit")
5656
omit_codes_list = util.parse_codes(_to_omit)
@@ -71,34 +71,65 @@ def lambda_handler(event, context):
7171
s3_path_program_coa += "-full"
7272
s3_path_program_coa += ".json"
7373

74+
s3_path_balances = s3_prefix + "balances"
75+
if not hide_inactive:
76+
s3_path_balances += "-full"
77+
s3_path_balances += ".json"
78+
7479
# get secure parameters
7580
ssm_secrets = ssm.get_secrets(ssm_path)
7681

7782
# parse the path and return appropriate data
7883
if "path" in event:
7984
event_path = event["path"]
8085

81-
# get chart of Program accounts
82-
_raw_program_chart = chart.get_program_chart(
83-
mip_org,
84-
ssm_secrets,
85-
s3_bucket,
86-
s3_path_program_coa,
87-
hide_inactive,
88-
)
89-
LOG.debug(f"Raw chart data: {_raw_program_chart}")
90-
91-
# always process the chart of Program accounts
92-
_program_chart = chart.process_chart(
93-
_raw_program_chart,
94-
omit_codes_list,
95-
code_other,
96-
code_no_program,
97-
params,
98-
)
99-
100-
# always limit the size of the chart
101-
program_chart = chart.limit_chart(_program_chart, params["limit"])
86+
if event_path == api_route_balances:
87+
# get chart of GL accounts
88+
gl_chart = chart.get_gl_chart(
89+
mip_org,
90+
ssm_secrets,
91+
s3_bucket,
92+
s3_path_gl_coa,
93+
hide_inactive,
94+
)
95+
LOG.debug(f"Raw chart data: {gl_chart}")
96+
97+
# get upstream balance data
98+
raw_bal = balances.get_balances(
99+
mip_org,
100+
ssm_secrets,
101+
s3_bucket,
102+
s3_path_balances,
103+
params["date"],
104+
)
105+
106+
# combine them into CSV output
107+
balances_csv = balances.format_csv(raw_bal, gl_chart)
108+
return util.build_return_text(200, balances_csv)
109+
110+
else: # common processing for '/accounts' and '/tags'
111+
112+
# get chart of Program accounts
113+
_raw_program_chart = chart.get_program_chart(
114+
mip_org,
115+
ssm_secrets,
116+
s3_bucket,
117+
s3_path_program_coa,
118+
hide_inactive,
119+
)
120+
LOG.debug(f"Raw chart data: {_raw_program_chart}")
121+
122+
# always process the chart of Program accounts
123+
_program_chart = chart.process_chart(
124+
_raw_program_chart,
125+
omit_codes_list,
126+
code_other,
127+
code_no_program,
128+
params,
129+
)
130+
131+
# always limit the size of the chart
132+
program_chart = chart.limit_chart(_program_chart, params["limit"])
102133

103134
if event_path == api_route_coa:
104135
# no more processing, return chart

‎mip_api/balances.py‎

Lines changed: 162 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,162 @@
1+
import logging
2+
3+
from mip_api import s3, upstream
4+
5+
import csv
6+
import io
7+
8+
LOG = logging.getLogger(__name__)
9+
LOG.setLevel(logging.DEBUG)
10+
11+
12+
def get_balances(org_name, secrets, bucket, path, when=None):
13+
"""
14+
Get trial balances from MIP Cloud and cache a successful response in S3.
15+
16+
Parameters
17+
----------
18+
org_name : str
19+
MIP Cloud organization name
20+
21+
secrets : dict
22+
MIP Cloud authentication credentials
23+
24+
bucket : str
25+
S3 bucket name
26+
27+
path : str
28+
S3 object path
29+
30+
when : str
31+
Activity period target date in ISO 8601 format (YYYY-MM-DD)
32+
"""
33+
LOG.info("Read balances from upstream API")
34+
upstream_dict = upstream.trial_balances(org_name, secrets, when)
35+
LOG.debug(f"Upstream API response: {upstream_dict}")
36+
37+
bal_dict = s3.cache(upstream_dict, bucket, path)
38+
return bal_dict
39+
40+
41+
def process_balance(bal_dict, coa_dict):
42+
"""
43+
Process upstream API response into a list of lists representing
44+
rows in a CSV file.
45+
46+
Parameters
47+
----------
48+
bal_dict : dict
49+
Upstream API response
50+
51+
coa_dict : dict
52+
Chart of accounts
53+
54+
Returns
55+
-------
56+
list
57+
List of lists representing rows in CSV file.
58+
Headers:
59+
'AccountNumber', 'AccountName', 'PeriodStart', 'PeriodEnd',
60+
'StartBalance', 'Activity', 'EndBalance'
61+
62+
"""
63+
64+
# check for success
65+
if "executionResult" not in bal_dict:
66+
LOG.error(f"No execution result found: '{bal_dict}'")
67+
raise KeyError("No 'executionResult' key found")
68+
69+
result = bal_dict["executionResult"]
70+
if result != "SUCCESS":
71+
LOG.error(f"Execution result is not 'SUCCESS': '{result}'")
72+
raise ValueError("Execution result is not 'SUCCESS'")
73+
74+
# collate api response into a dict
75+
_data = {}
76+
77+
_detail = []
78+
for k, v in bal_dict["extraInformation"].items():
79+
if k != "Level1":
80+
LOG.error(f"Unexpected key (not 'Level1'): {k}")
81+
raise KeyError("No 'Level1' key found")
82+
else:
83+
_detail = v
84+
85+
for d in _detail:
86+
account_id = d["DBDETAIL_SUM_SEGMENT_N0"]
87+
if account_id not in _data:
88+
_data[account_id] = {}
89+
90+
if d["DBDETAIL_SUM_TYPE"] == 1:
91+
_data[account_id]["balance_start"] = d["DBDETAIL_SUM_POSTEDAMT"]
92+
elif d["DBDETAIL_SUM_TYPE"] == 2:
93+
_data[account_id]["activity"] = d["DBDETAIL_SUM_POSTEDAMT"]
94+
elif d["DBDETAIL_SUM_TYPE"] == 3:
95+
_data[account_id]["balance_end"] = d["DBDETAIL_SUM_POSTEDAMT"]
96+
else:
97+
LOG.error(f"Unknown balance type: {d['DBDETAIL_SUM_DESC']}")
98+
99+
LOG.debug(f"Raw internal balance dict: {_data}")
100+
101+
# List of rows in CSV
102+
out_rows = []
103+
104+
# Add header row
105+
headers = [
106+
"AccountNumber",
107+
"AccountName",
108+
"PeriodStart",
109+
"PeriodEnd",
110+
"StartBalance",
111+
"Activity",
112+
"EndBalance",
113+
]
114+
out_rows.append(headers)
115+
116+
# Generate rows from input dict
117+
for k, v in _data.items():
118+
if k not in coa_dict:
119+
LOG.error(f"Key {k} not found in chart of accounts")
120+
LOG.debug(f"List of accounts: {coa_dict.keys()}")
121+
continue
122+
name = coa_dict[k]
123+
124+
row = [
125+
k,
126+
name,
127+
bal_dict["period_from"],
128+
bal_dict["period_to"],
129+
v["balance_start"],
130+
v["activity"],
131+
v["balance_end"],
132+
]
133+
out_rows.append(row)
134+
135+
return out_rows
136+
137+
138+
def format_csv(bal_dict, coa_dict):
139+
"""
140+
Process upstream API response into string contents of a CSV file.
141+
142+
Parameters
143+
----------
144+
bal_dict : dict
145+
Upstream API response
146+
147+
coa_dict : dict
148+
Chart of accounts
149+
150+
Returns
151+
-------
152+
str
153+
String contents of CSV file.
154+
"""
155+
csv_out = io.StringIO()
156+
csv_writer = csv.writer(csv_out)
157+
158+
csv_rows = process_balance(bal_dict, coa_dict)
159+
for row in csv_rows:
160+
csv_writer.writerow(row)
161+
162+
return csv_out.getvalue()

0 commit comments

Comments
 (0)