aws-blue-green-toolkit

Utility functions to help with performing blue/green and canary deployments in AWS infrastructure

background

AWS support for blue/green and canary deployments is provided by CodeDeploy. CodeDeploy is great in what it does, but it only really manages swapping versions for web services. Any services that sit behind the public facing API don't get managed by the CodeDeploy flow.

A common pattern for blue/green and canary deployments is to not just use a pair of services for the web component, but use a pair of services for the whole pipeline, eg data ingestion, databases, queues, data manipulation lambdas etc. You can manage starting, hydrating, and using these services as part of the CodeDeploy pipeline through the use of hooks (in the form of lambdas). Using these hook lambdas you can manage blue/green or canary deployments of a whole stack of services using the aws-sdk, unfortunately this normally means writing a lot of code to manage the process.

This module is a set of utility functions that I use to reduce the boilerplate required to set up these deploment hook lambdas.

example architecture

For an example web api, using a node application in docker, with a SQL data store, and data updates being published on an SNS topic, the architecture may looks something like this:

In order to manage blue/green deployments for this service, CodeDeploy will directly handle deploying the new ECS service, attaching target groups to load balancers, flipping traffic over to the new service, and tearing down the old containers.

Unfortunately this really only covers less than half of the architecture. Everything from the data updates topic through to the database is up to you to manage through hooks. When deploying a new version of the service, the steps required will be something like this:

1. Replacement database cluster is started or created
2. Autoscaling minimum capacity on the new db cluster is set to match the current capacity of the active cluster
3. Replacement database is purged and the latest schema is created
4. SNS subscriptions are enabled for the replacement queue
5. SQS subscription lambda is enabled for the replacement stack
6. A new task set is created in the ECS service, it is bound to the replacement stack's database
7. The new task set is placed in a testing target group, this is attached to a testing port on the load balancer
8. Automated tests are carried out against the replacement service via the testing port
9. The replacement stack becomes the active stack, what was the active stack is now the old stack.
10. Autoscaling minimum capacity on the active set is reverted to the normal value, database will scale in when traffic is lower.
11. SNS subscriptions are disabled for the old stack
12. SQS subscription for the old stack's ingest queue is disabled
13. Old database cluster is stopped or deleted
14. The old SQS queues are purged
15. The old ECS taskset is destroyed

CodeDeploy will carry out steps 6, 7, 9, and 15. This module contains tools to help you perform the other steps from your deployment hook lambda functions.

api

Table of Contents

• ClusterState
• EcsTools
  • disableService
    • Parameters
  • enableService
    • Parameters
• SqsTools
  • purgeQueues
    • Parameters
• DynamoTools
  • deleteTable
    • Parameters
• CloudWatchTools
  • disableAlarmsActions
    • Parameters
  • enableAlarmsActions
    • Parameters
• KinesisTools
  • deregisterConsumer
    • Parameters
  • describeConsumer
    • Parameters
  • registerConsumer
    • Parameters
• StackReference
• AuroraTools
  • getClusterState
    • Parameters
  • scaleIn
    • Parameters
  • scaleOut
    • Parameters
  • getReaderCount
    • Parameters
  • startDatabase
    • Parameters
  • stopDatabase
    • Parameters
  • deleteDatabase
    • Parameters
  • applyTags
    • Parameters
  • enablePerformanceInsights
    • Parameters
• LambdaTools
  • createEventSourceMapping
    • Parameters
  • deleteEventMapping
    • Parameters
  • disableEventMapping
    • Parameters
  • disableRule
    • Parameters
  • enableEventMapping
    • Parameters
  • enableRule
    • Parameters
  • getAlias
    • Parameters
  • getLatestMetrics
    • Parameters
  • getVersion
    • Parameters
  • listEventSourceMappings
    • Parameters
• SnsTools
  • disableSubscription
    • Parameters
  • enableSubscription
    • Parameters

ClusterState

src/main/constants.ts:7-12

Enum for describing the state of an RDS cluster

Type: number

EcsTools

src/main/ecs-tools.ts:11-64

Tools for managing pairs of ECS services

disableService

src/main/ecs-tools.ts:32-43

Disables an ECS service by setting the desired task count to zero

Parameters

• reference StackReference The stack to modify

Returns any {Promise}

enableService

src/main/ecs-tools.ts:52-63

Enables an ECS service by setting the desired task count to it's normal value

Parameters

• reference StackReference The stack to modify

Returns any {Promise}

SqsTools

src/main/sqs-tools.ts:11-50

Toolkit for SQS operations

purgeQueues

src/main/sqs-tools.ts:32-45

Purges a queue pair (q and dlq) based on config and queue reference

Parameters

• reference StackReference Reference to a subscription queue stack

Returns Promise<void>

DynamoTools

src/main/dynamo-tools.ts:14-55

Toolkit for Dynamo operations

deleteTable

src/main/dynamo-tools.ts:35-48

Deletes a dynamo table

Parameters

• reference StackReference Reference to a active table

Returns Promise<void>

CloudWatchTools

src/main/cloudwatch-tools.ts:16-80

Toolkit for CloudWatch operations

disableAlarmsActions

src/main/cloudwatch-tools.ts:36-41

Disable all alarm actions

Parameters

• reference StackReference Reference to a subscription queue stack

Returns Promise<void>

enableAlarmsActions

src/main/cloudwatch-tools.ts:49-54

Enable all alarm actions

Parameters

• reference StackReference Reference to a subscription queue stack

Returns Promise<void>

KinesisTools

src/main/kinesis-tools.ts:18-91

Toolkit for Kinesis data stream operations

deregisterConsumer

src/main/kinesis-tools.ts:39-46

Deregisters an existing consumer for a Kinesis data stream

Parameters

• reference StackReference Reference to an active stack

Returns Promise<void>

describeConsumer

src/main/kinesis-tools.ts:55-64

Describes a consumer for a Kinesis data stream

Parameters

• reference StackReference Reference to an active stack

Returns Promise<DescribeStreamConsumerOutput>

registerConsumer

src/main/kinesis-tools.ts:73-82

Registers a new consumer for a Kinesis data stream

Parameters

• reference StackReference Reference to an active stack

Returns Promise<RegisterStreamConsumerOutput>

StackReference

src/main/constants.ts:20-23

Enum for referencing blue or green stacks

Type: number

AuroraTools

src/main/aurora-tools.ts:32-374

Toolkit for Aurora operations

getClusterState

src/main/aurora-tools.ts:58-76

Gets the current state of one of the Aurora clusters

Parameters

• reference StackReference Reference to a db cluster

Returns Promise<ClusterState>

scaleIn

src/main/aurora-tools.ts:85-87

Reverts a cluster's minimum reader count to the configured minimum

Parameters

• reference StackReference Reference to a db cluster

Returns Promise<void>

scaleOut

src/main/aurora-tools.ts:96-101

Scales out a cluster to match it's partner's size

Parameters

• reference StackReference Reference to a db cluster

Returns Promise<void>

getReaderCount

src/main/aurora-tools.ts:110-120

Get a count of the number of active readers for a cluster

Parameters

• reference StackReference Reference to a db cluster

Returns Promise<number> The number of active readers

startDatabase

src/main/aurora-tools.ts:129-132

Starts a stopped db cluster

Parameters

• reference StackReference Reference to a db cluster

Returns Promise<void>

stopDatabase

src/main/aurora-tools.ts:141-144

Stops a running db cluster

Parameters

• reference StackReference Reference to a db cluster

Returns Promise<void>

deleteDatabase

src/main/aurora-tools.ts:153-185

Deletes a running db cluster

Parameters

• reference StackReference Reference to a db cluster

Returns Promise<void>

applyTags

src/main/aurora-tools.ts:195-235

Parses a message from an rds event subscription, if the event was triggered by a scale out operation, the tags defined in config are applied to the newly created reader.

Parameters

• record SNSEventRecord An SNS event record of the type published by rds event streams

Returns Promise<void>

enablePerformanceInsights

src/main/aurora-tools.ts:249-300

Parses a message from an rds event subscription, if the event was triggered by a scale out operation and the new instance does not have performance insights enabled, the instance is updated to enable performance insights.

Parameters

• record SNSEventRecord An SNS event record of the type published by rds event streams
• reEnableIfDisabled boolean Whether or not to automatically re enable insights if they are disabled (optional, default true)
• retryDelay number Time in ms to wait before retrying (optional, default 60e3)
• retryAttempts number Number of retry attempts (optional, default 5)

Returns any {Promise}

LambdaTools

src/main/lambda-tools.ts:38-345

Toolkit for Lambda operations

createEventSourceMapping

src/main/lambda-tools.ts:68-83

Creates a lambda's event source mapping (eg, a Kinesis stream)

Parameters

• reference StackReference Reference to a lambda stack
• eventSourceArn string The ARN of the event source
• sourceSpecificParameters Omit<CreateEventSourceMappingRequest, ("FunctionName" | "EventSourceArn")> (optional, default {})
• sourceSpecificParams (Omit<CreateEventSourceMappingRequest, ("FunctionName" | "EventSourceArn")>) Any params specific to the event source (optional, default {})

Returns any {Promise}

deleteEventMapping

src/main/lambda-tools.ts:94-96

Deletes a lambda's event mapping (eg, a Kinesis stream) You may use the listEventSourceMappings method if you need to retrieve UUIDs of the function event sources

Parameters

• UUID StackReference The identifier of the event source mapping

Returns Promise<void>

disableEventMapping

src/main/lambda-tools.ts:105-107

Disables a lambda's event mappings (eg, an SQS subscription)

Parameters

• reference StackReference Reference to a lambda stack

Returns Promise<void>

disableRule

src/main/lambda-tools.ts:116-118

Disables a lambda's cloudwatch events rule (ie, cron trigger)

Parameters

• reference StackReference Reference to a lambda stack

Returns Promise<void>

enableEventMapping

src/main/lambda-tools.ts:127-129

Enables a lambda's event mappings (eg, an SQS subscription)

Parameters

• reference StackReference Reference to a lambda stack

Returns Promise<void>

enableRule

src/main/lambda-tools.ts:138-140

Enables a lambda's cloudwatch events rule (ie, cron trigger)

Parameters

• reference StackReference Reference to a lambda stack

Returns Promise<void>

getAlias

src/main/lambda-tools.ts:150-156

Returns details about a Lambda function alias.

Parameters

• reference StackReference Reference to a lambda stack
• Name string The name of the alias to return data about

Returns Promise<AliasConfiguration>

getLatestMetrics

src/main/lambda-tools.ts:165-237

Returns the latest metrics about a Lambda function alias.

Parameters

• reference StackReference Reference to a lambda stack

Returns Promise<LatestLambdaMetricsMap>

getVersion

src/main/lambda-tools.ts:246-254

Gets the currently running version of a lambda fn

Parameters

• reference StackReference Reference to a lambda stack

Returns Promise<string> The lambda version

listEventSourceMappings

src/main/lambda-tools.ts:263-282

Lists all event source mappings for the referenced function

Parameters

• reference StackReference -- Reference to a lambda stack

Returns any {Promise<EventSourceMappingConfiguration[]>}

SnsTools

src/main/sns-tools.ts:40-100

Toolkit for SNS operations

disableSubscription

src/main/sns-tools.ts:63-65

Disables an SNS subscription

Parameters

• reference StackReference Reference to a subscription queue stack

Returns Promise<void>

enableSubscription

src/main/sns-tools.ts:74-76

Enables an SNS subscription

Parameters

• reference StackReference Reference to a subscription queue stack

Returns Promise<void>