lambda-finops-floqast-sftp

AWS Lambda to periodically fetch trial balances from lambda-mips-api as CSV files and upload them to FloQast via SFTP.

The activity period for the trial balances is one month, or month-to-date for the current month. A configurable number of activity periods will be fetched, starting with the current month and moving backwards. A separate CSV file will be fetched for each activity period, and uploaded to the specified SFTP server with a unique file name.

Parameters

SSM Parameters

User credentials for logging in to the SFTP server are stored as secure parameters in SSM with a configurable prefix. By default, the prefix is /floqast-sftp.

Required SSM Parameters

The user, pass, and host parameters are required for SFTP authentication.

Parameter	Description
user	SFTP username
pass	SFTP password
host	SFTP host name

Optional SSM Parameter

An optional port parameter can be used to configure the host port.

Parameter	Description	Default
port	SFTP host port	22

Template Parameters

The following template parameters are used to configure behavior:

Template Parameter	Type	Default	Description
Schedule	EventBridge Schedule Expression	cron(30 10 2 * ? *)	EventBridge schedule for running the lambda
SsmPrefix	String	/floqast-sftp	Prepend this value to the SSM parameter keys
PeriodCount	Number	2	The number of activity periods (months) to report on, starting at the present and moving backwards

Development

Contributions

Contributions are welcome.

Setup Development Environment

Install the following applications:

• AWS CLI
• AWS SAM CLI
• pre-commit
• pipenv

Install Requirements

Run pipenv sync --dev to install both production and development requirements, and pipenv shell to activate the virtual environment. For more information see the pipenv docs.

After activating the virtual environment, run pre-commit install to install the pre-commit git hook.

Update Requirements

First, make any needed updates to the base requirements in Pipfile, then use pipenv to regenerate both Pipfile.lock and requirements.txt. We use pipenv to control versions in testing, but sam relies on requirements.txt directly for building the container used by the lambda.

$ pipenv update --dev

We use pipenv to control versions in testing, but sam relies on requirements.txt directly for building the lambda artifact, so we dynamically generate requirements.txt from Pipfile.lock before building the artifact. The file must be created in the CodeUri directory specified in template.yaml.

$ pipenv requirements > requirements.txt

Additionally, pre-commit manages its own requirements.

$ pre-commit autoupdate

Create a local build

Use a Lambda-like docker container to build the Lambda artifact

$ sam build --use-container

Run unit tests

Tests are defined in the tests folder in this project, and dependencies are managed with pipenv. Install the development dependencies and run the tests using coverage.

$ pipenv run coverage run -m pytest tests/ -svv

Automated testing will upload coverage results to Coveralls.

Run integration tests

Running integration tests requires docker

$ sam local invoke FloQastSftpFunction --event events/event.json

Deployment

Deploy Lambda to S3

Deployments are sent to the Sage cloudformation repository which requires permissions to upload to Sage bootstrap-awss3cloudformationbucket-19qromfd235z9 and essentials-awss3lambdaartifactsbucket-x29ftznj6pqw buckets.

sam package --template-file .aws-sam/build/template.yaml \
  --s3-bucket essentials-awss3lambdaartifactsbucket-x29ftznj6pqw \
  --output-template-file .aws-sam/build/lambda-finops-floqast-sftp.yaml

aws s3 cp .aws-sam/build/lambda-finops-floqast-sftp.yaml s3://bootstrap-awss3cloudformationbucket-19qromfd235z9/lambda-finops-floqast-sftp/main/

Publish Lambda

Private access

Publishing the lambda makes it available in your AWS account. It will be accessible in the serverless application repository.

sam publish --template .aws-sam/build/lambda-finops-floqast-sftp.yaml

Public access

Making the lambda publicly accessible makes it available in the global AWS serverless application repository

aws serverlessrepo put-application-policy \
  --application-id <lambda ARN> \
  --statements Principals=*,Actions=Deploy

Install Lambda into AWS

Sceptre

Create the following sceptre file config/prod/lambda-finops-floqast-sftp.yaml

template:
  type: http
  url: "https://PUBLISH_BUCKET.s3.amazonaws.com/lambda-finops-floqast-sftp/VERSION/lambda-finops-floqast-sftp.yaml"
stack_name: "lambda-finops-floqast-sftp"
stack_tags:
  Department: "Platform"
  Project: "Infrastructure"
  OwnerEmail: "it@sagebase.org"

Install the lambda using sceptre:

sceptre --var "profile=my-profile" --var "region=us-east-1" launch prod/lambda-finops-floqast-sftp.yaml

AWS Console

Steps to deploy from AWS console.

1. Login to AWS
2. Access the serverless application repository -> Available Applications
3. Select application to install
4. Enter Application settings
5. Click Deploy

Releasing

We have setup our CI to automate a releases. To kick off the process just create a tag (i.e 0.0.1) and push to the repo. The tag must be the same number as the current version in template.yaml. Our CI will do the work of deploying and publishing the lambda.