Skip to content

How to Resolve "EC2 Failing Instances" Alerts

Jump to bottom
Mitchell Alessio edited this page Sep 28, 2023 · 11 revisions

How to Resolve "EC2 Failing Instances" Alerts

Glossary

Term Definition
EC2 Status Checks See AWS documentation

Prerequisites

  • Access to the CMS VPN
  • Access to AWS
  • An installation of the AWS CLI that is configured properly for access to the BFD/CMS AWS account
  • An installation of jq

Instructions

Determine which Instances are Failing

  1. Run the following command in bash or zsh:

     failed_instances="$(aws ec2 describe-instance-status --query 'InstanceStatuses[?SystemStatus.Status==`impaired` || InstanceStatus.Status==`impaired`]')"; [[ -n "${failed_instances#\[\]}" ]] && echo $failed_instances && aws ec2 describe-instances --no-cli-pager --query 'Reservations[].Instances[].[InstanceId,Tags[?Key==`Name`]| [0].Value]' --output table --instance-ids $(jq -n -r '$in.[].InstanceId' --argjson in "$failed_instances")

    This command will output a list, in JSON, of all of the EC2 instances that have failed their EC2 Status Checks. Additionally, a table of Instance IDs to friendly names will be printed at the end.

  2. In all likelihood, the failing instances reported by the CLI command in Step #1 will be some variant of pipeline. If this is the case, and those pipeline instances exist in our established environments, follow the instructions for re-deploying pipeline instances in Resolve Failing bfd-*-pipeline Instances

  3. If the instances are not some variant of pipeline, but are some variant of server or migrator within our established environments, a new deployment should be started specifying that app AMIs should be built

  4. If any instances do not fall in the above cases:

    1. server instances that have been detached from their ASG in an established environment should be destroyed, and a new instance detached
    2. Instances of any variety deployed within an ephemeral environment will require that the corresponding AMIs be re-built and the ephemeral environment re-deployed
    3. Other instances that do not use our typical AMIs or use the Platinum AMI will need to be restarted, or the corresponding AMI re-built and the instance re-deployed

Resolve Failing bfd-*-pipeline Instances

Resolve Failing pipeline Instances by Redeploying

This is the simplest option, and requires no knowledge of Terraform and our various Terraservices.

  1. Re-deploy the latest release using the BFD Deployment Pipeline ensuring that AMIs are built again
For BFD Contributors and Maintainers

Clone this wiki locally