How to Resolve "EC2 Failing Instances" Alerts

How to Resolve "EC2 Failing Instances" Alerts

Prerequisites

• Access to the CMS VPN
• Access to AWS
• An installation of the AWS CLI that is configured properly for access to the BFD/CMS AWS account
• An installation of jq

Instructions

Determine which Instances are Failing

1. Run the following command in bash or zsh:

    failed_instances="$(aws ec2 describe-instance-status --query 'InstanceStatuses[?SystemStatus.Status==`impaired` || InstanceStatus.Status==`impaired`]')"; [[ -n "${failed_instances#\[\]}" ]] && echo $failed_instances && aws ec2 describe-instances --no-cli-pager --query 'Reservations[].Instances[].[InstanceId,Tags[?Key==`Name`]| [0].Value]' --output table --instance-ids $(jq -n -r '$in.[].InstanceId' --argjson in "$failed_instances")

   This command will output a list, in JSON, of all of the EC2 instances that have failed their EC2 Status Checks. Additionally, a table of Instance IDs to friendly names will be printed at the end.