How to Evaluate statistics for generic "Lambda Error" Alerts posted to Slack

Pre-Requisites

• Access to CMS VPN
• Access to BFD/CMS AWS account(s)
• Installation of AWS CLI, properly configured for access to BFD/CMS AWS account
• Installation of jq

Instructions

Determine which alarms have been tripped within the past N days (24h is default):

Save the following bash script with execute permissions to your local environment after connecting to CMS VPN:

#!/bin/bash

## set Cloudwatch Metrics Query

LAMBDAERRORMETRICSRCH=$(

cat <<EOF

[ { "Id": "e1", "Expression": "SEARCH('{AWS/Lambda,FunctionName} (\"Errors\")', 'Sum', 300)", "Label": "Expression1", "ReturnData": true } ]

EOF

)

DAYSTOREVIEW=1

if [ $# -ne 0 ]

then

newarg=$(expr $1 + 0 2>/dev/null)

if [ ! -z "$newarg" ] && [ $newarg -gt 0 ]

then

DAYSTOREVIEW=$newarg

fi

fi

## set start and end time window for Metrics Query

STARTTS=$(date -u -v-${DAYSTOREVIEW}d "+%Y-%m-%dT%H:%M:%S")

ENDTS=$(date -u "+%Y-%m-%dT%H:%M:%S")

## set temp file for JQ summary of Metrics Query and set trap for exit on failure

JQRYEX=$(mktemp -q /tmp/jqry.XXXXXX.jq || exit 1)

trap 'rm -f -- "${JQRYEX}"' EXIT

## construct JQ expression

cat > ${JQRYEX} <<EOF2

.MetricDataResults | map({ Lambda: (.Label | ltrimstr("Expression1 ")), HitsInUTC: [ .Timestamps as \$ts | .Values as \$vals | [ range(0; \$ts | length) | select (\$vals[.] > 0) | { (\$ts[.]) : \$vals[.]} ] | add ] }) | .[] | select(.HitsInUTC[])

EOF2

## Query Cloudwatch Metrics

aws cloudwatch get-metric-data --metric-data-queries "${LAMBDAERRORMETRICSRCH}" --start-time "${STARTTS}" --end-time "${ENDTS}" --output json | jq -f ${JQRYEX} | jq -s .

## temp file cleanup

rm -f -- "${JQRYEX}"

trap - EXIT

exit

Note: The above script queries the AWS/Lambda Cloudwatch Metrics for the specified DAYSTOREVIEW (date -u -v-${DAYSTOREVIEW}d "+%Y-%m-%dT%H:%M:%S"). That result set is filtered and sorted by the corresponding JQ query (JQRYEX) for output to be evaluated by the engineer.

Investigation

Execute the saved script and proceed with further detailed investigation as warranted; feel free to customize as needed; optionally, add an argument for the number of historical days from current day for which to evaluate alarms.

Example 1: 1 day of metrics

$ scripts/lambda-errors-metrics-pull.sh

Example 2: 3 days of metrics

$ scripts/lambda-errors-metrics-pull.sh 3