How to Evaluate "EC2 Startup Failure" Alerts

Pre-Requisites

• Access to CMS VPN
• Access to BFD/CMS AWS account(s)
• Installation of AWS CLI, properly configured for access to BFD/CMS AWS account
• Installation of jq, sed, awk

Instructions

Determine which instances have failed and whether still available within AWS EC2 environment:

• Run the following command sequence in bash or zsh after connecting to CMS VPN

aws logs filter-log-events --log-group-name /aws/ec2/var/log/cloud-init-output.log --filter-pattern "%failed=[1-9]%" --start-time $(( $(( $(date +%s) - 3600 )) * 1000 )) | jq '[ .events[] | { logStreamName, message }]' | jq '.[].logStreamName' | sed 's/"//g' | awk -F'.' '{ print $1; }' | sed 's/-/./g' | awk -F'.' '{printf "%d.%d.%d.%d\n",$2,$3,$4,$5;}' | while read ipaddr; do echo "$ipaddr"; aws ec2 describe-instances --filters "Name=network-interface.addresses.private-ip-address,Values=$ipaddr"; done;

Note: The above command will output a detailed JSON object of all of the EC2 instances, preceded by the assigned private IPv4 address, that failed any startup scripts within the past hour (adjust '3600' seconds accordingly if necessary); if the JSON object is nothing more than "Reservations[]" then the instance was automatically terminated.

Investigation

1. Review the Cloudwatch log stream associated with each specified Private IPv4 address in order to determine which AMI profile was leveraged for deployment.
2. Evaluate the server profile startup to determine if the failure is general or environment-specific.
3. Proceed with further detailed investigation as warranted.