Merge branch 'develop'

Author: robertpeteuil

CHANGELOG.md

@@ -0,0 +1,76 @@
+# terraform-aws-sns-to-cloudwatch-logs-lambda Changelog
+
+All notable changes to this project will be documented in this file.
+
+This project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).
+
+## [1.0.0] - 2019-03-30
+
+- Moved all Python dependancies to Lambda Layers
+- Python function editable in repository and in Lambda UI
+- Default Python version switched to 3.6
+- Add optional dynamically calculated function name based on topic and Cloudwatch Group/Stream
+- Optionally create custom Lambda Layer zip using [build-lambda-layer-python submodule](https://github.com/robertpeteuil/build-lambda-layer-python)
+  - Enables adding/changing dependancies
+  - Enables compiling for different version of Python
+- Add new variable `lambda_runtime`
+
+## [0.2.6] - 2018-10-14
+
+Add ability to assign tags to created lambda function using new map variable `lambda_tags`
+
+## [0.2.5] - 2018-10-09
+
+Comment Cleanup
+
+## [0.2.4] - 2018-08-20
+
+Update README
+
+## [0.2.3] - 2018-08-01
+
+Update README
+
+## [0.2.2] - 2018-08-01
+
+Update README
+
+## [0.2.1] - 2018-07-29
+
+Added additional outputs:
+
+- `lambda_version` - Latest published version of Lambda Function
+- `lambda_last_modified` - The date the Lambda Function was last modified
+
+## [0.2.0] - 2018-07-28
+
+Update README
+
+## [0.1.3] - 2018-07-28
+
+Add additional outputs
+
+## [0.1.2] - 2018-07-28
+
+Minor Edits
+
+## [0.1.1] - 2018-07-28
+
+Adjust outputs
+
+## [0.1.0] - 2018-07-28
+
+Initial Release
+
+[1.0.0]: https://github.com/robertpeteuil/terraform-aws-sns-to-cloudwatch-logs-lambda/compare/0.2.6...1.0.0
+[0.2.6]: https://github.com/robertpeteuil/terraform-aws-sns-to-cloudwatch-logs-lambda/compare/0.2.5...0.2.6
+[0.2.5]: https://github.com/robertpeteuil/terraform-aws-sns-to-cloudwatch-logs-lambda/compare/0.2.4...0.2.5
+[0.2.4]: https://github.com/robertpeteuil/terraform-aws-sns-to-cloudwatch-logs-lambda/compare/0.2.3...0.2.4
+[0.2.3]: https://github.com/robertpeteuil/terraform-aws-sns-to-cloudwatch-logs-lambda/compare/0.2.2...0.2.3
+[0.2.2]: https://github.com/robertpeteuil/terraform-aws-sns-to-cloudwatch-logs-lambda/compare/0.2.1...0.2.2
+[0.2.1]: https://github.com/robertpeteuil/terraform-aws-sns-to-cloudwatch-logs-lambda/compare/0.2.0...0.2.1
+[0.2.0]: https://github.com/robertpeteuil/terraform-aws-sns-to-cloudwatch-logs-lambda/compare/0.1.3...0.2.0
+[0.1.3]: https://github.com/robertpeteuil/terraform-aws-sns-to-cloudwatch-logs-lambda/compare/0.1.2...0.1.3
+[0.1.2]: https://github.com/robertpeteuil/terraform-aws-sns-to-cloudwatch-logs-lambda/compare/0.1.1...0.1.2
+[0.1.1]: https://github.com/robertpeteuil/terraform-aws-sns-to-cloudwatch-logs-lambda/compare/0.1.0...0.1.1
+[0.1.0]: https://github.com/robertpeteuil/terraform-aws-sns-to-cloudwatch-logs-lambda/tree/0.1.0

README.md

@@ -8,29 +8,34 @@
 
 This Module allows simple and rapid deployment
 
-- Creates Lambda function, IAM Policies, Triggers, and Subscriptions
+- Creates Lambda function, Lambda Layer, IAM Policies, Triggers, and Subscriptions
 - Creates (or use existing) SNS Topic, CloudWatch Log Group and Log Group Stream
 - Options:
   - Create CloudWatch Event to prevent Function hibernation
   - Set Log Group retention period
+- Python function editable in repository and in Lambda UI
+  - Python dependancies packages in Lambda Layers zip
+- Optionally create custom Lambda Layer zip using [build-lambda-layer-python](https://github.com/robertpeteuil/build-lambda-layer-python)
+  - Enables adding/changing dependancies
+  - Enables compiling for different version of Python
 
 ## SNS to CloudWatch Logs Features
 
 This Lambda Function forwards subject & body of SNS messages to CloudWatch Log Group Stream
 
-- Enhances the value of CloudWatch Logs by enabling on-demand entry creation from any service, function and script
+- Enhances the value of CloudWatch Logs by enabling easy entry creation from any service, function and script that can send SNS notifications
 - Enables cloud-init, bootstraps and functions to easily write log entries to a centralized CloudWatch Log
 - Simplifies troubleshooting of solutions with decentralized logic
   - scripts and functions spread across instances, Lambda and services
 - Easily add instrumentation to scripts: `aws sns publish --topic-arn $TOPIC_ARN --message $LOG_ENTRY`
-  - Use with IAM instance policy may require `--region $AWS_REGION` parameter
+  - Use with IAM instance policy requires `--region $AWS_REGION` parameter
 
 ## Usage
 
 ``` ruby
 module "sns_logger" {
   source            = "robertpeteuil/sns-to-cloudwatch-logs-lambda/aws"
-  version           = "0.2.6"
+  version           = "1.0.0"
 
   aws_region        = "us-west-2"
   sns_topic_name    = "projectx-logging"
@@ -58,10 +63,11 @@ module "sns_logger" {
 | create_log_group | Create new log group | string | `true` | no |
 | create_log_stream | Create new log stream | string | `true` | no |
 | log_group_retention_days | Log Group retention (days) | string | `0` (forever) | no |
-| lambda_func_name | Name for Lambda Function | string | `SNStoCloudWatchLogs` | no |
+| lambda_func_name | Name for Lambda Function | string | dynamically calculated | no |
 | lambda_description | Lambda Function Description | string | `Route SNS messages to CloudWatch Logs` | no |
 | lambda_tags | Mapping of Tags to assign to Lambda function | map | `{}` | no |
 | lambda_publish_func | Publish Lambda Function | string | `false` | no |
+| lambda_runtime | Lambda runtime for Function | string | `python3.6` | no |
 | lambda_timeout | Function time-out (seconds) | string | `3` | no |
 | lambda_mem_size | Function RAM assigned (MB) | string | `128` | no |
 | create_warmer_event | Create CloudWatch trigger event to prevent hibernation | string | `false` | no |

base_python3.6.zip

@@ -0,0 +1 @@
+watchtower

function/requirements.txt

@@ -0,0 +1,34 @@
+"""Write log entries to cloudwatch logs."""
+
+from __future__ import print_function
+import logging
+import watchtower
+import os
+
+log_group = os.getenv('log_group')
+log_stream = os.getenv('log_stream')
+logger = logging.getLogger(__name__)
+logger.setLevel(logging.INFO)
+aws_handler = watchtower.CloudWatchLogHandler(log_group=log_group, stream_name=log_stream)
+logger.addHandler(aws_handler)
+
+
+def main(event, context):
+
+    try:
+        message_source = event['Records'][0]['EventSource']
+    except KeyError:
+        return
+
+    if message_source == 'aws:sns':
+        subject = event['Records'][0]['Sns']['Subject']
+        body = event['Records'][0]['Sns']['Message']
+
+        if subject:
+            logger.info("{}\n{}".format(subject, body))
+        else:
+            logger.info(body)
+
+        aws_handler.flush()
+
+    return 'Generated by Lambda'

function/sns_cloudwatch_gw.py

@@ -6,24 +6,54 @@ terraform {
   required_version = "~> 0.11.7"
 }
 
+provider "aws" {
+  region = "${var.aws_region}"
+}
+
+# -----------------------------------------------------------------
+# CREATE LAMBDA BASE LAYER CONTAINING PYTHON LIBRARIES
+# -----------------------------------------------------------------
+
+resource "aws_lambda_layer_version" "logging_base" {
+  filename         = "${path.module}/base_${var.lambda_runtime}.zip"
+  source_code_hash = "${base64sha256(file("${path.module}/base_${var.lambda_runtime}.zip"))}"
+
+  layer_name  = "sns-cloudwatch-base-${replace(var.lambda_runtime,".","")}"
+  description = "python logging and watchtower libraries"
+
+  compatible_runtimes = ["${var.lambda_runtime}"]
+}
+
 # -----------------------------------------------------------------
-# CREATE LAMBDA FUNCTION - SNS TO CLOUDWATCH LOGS GATEWAY 
-#   environment variables used for the 'log_group' and 'log_stream'
-#   function published if 'lambda_publish_func' set
+# CREATE LAMBDA FUNCTION USING ZIP FILE 
 # -----------------------------------------------------------------
 
+# make zip
+data "archive_file" "lambda_function" {
+  type        = "zip"
+  source_file = "${path.module}/function/sns_cloudwatch_gw.py"
+  output_path = "${path.module}/lambda.zip"
+}
+
+locals {
+  dynamic_description = "Routes SNS topic '${var.sns_topic_name}' to CloudWatch group '${var.log_group_name}' and stream '${var.log_stream_name}'"
+}
+
+# create lambda using function only zip on top of base layer
 resource "aws_lambda_function" "sns_cloudwatchlog" {
-  function_name = "${var.lambda_func_name}"
-  description   = "${var.lambda_description}"
+  layers = ["${aws_lambda_layer_version.logging_base.arn}"]
+
+  function_name = "${var.lambda_func_name}-${var.sns_topic_name}"
+  description   = "${length(var.lambda_description) > 0 ? var.lambda_description : local.dynamic_description}"
 
-  filename         = "${path.module}/source.zip"
-  source_code_hash = "${base64sha256(file("${path.module}/source.zip"))}"
+  filename         = "${path.module}/lambda.zip"
+  source_code_hash = "${data.archive_file.lambda_function.output_base64sha256}"
 
   publish = "${var.lambda_publish_func ? 1 : 0}"
   role    = "${aws_iam_role.lambda_cloudwatch_logs.arn}"
 
-  runtime     = "python2.7"
-  handler     = "lambda_function.lambda_handler"
+  runtime     = "${var.lambda_runtime}"
+  handler     = "sns_cloudwatch_gw.main"
   timeout     = "${var.lambda_timeout}"
   memory_size = "${var.lambda_mem_size}"
 
@@ -126,13 +156,13 @@ resource "aws_lambda_permission" "sns_cloudwatchlog" {
 
 # Create IAM role
 resource "aws_iam_role" "lambda_cloudwatch_logs" {
-  name               = "lambda_${lower(var.lambda_func_name)}"
+  name               = "lambda-${lower(var.lambda_func_name)}-${var.sns_topic_name}"
   assume_role_policy = "${data.aws_iam_policy_document.lambda_cloudwatch_logs.json}"
 }
 
 # Add base Lambda Execution policy
 resource "aws_iam_role_policy" "lambda_cloudwatch_logs_polcy" {
-  name   = "lambda_${lower(var.lambda_func_name)}_policy"
+  name   = "lambda-${lower(var.lambda_func_name)}-policy-${var.sns_topic_name}"
   role   = "${aws_iam_role.lambda_cloudwatch_logs.id}"
   policy = "${data.aws_iam_policy_document.lambda_cloudwatch_logs_policy.json}"
 }
@@ -170,7 +200,7 @@ data "aws_iam_policy_document" "lambda_cloudwatch_logs_policy" {
 resource "aws_cloudwatch_event_rule" "warmer" {
   count = "${var.create_warmer_event ? 1 : 0}"
 
-  name                = "sns_logger_warmer"
+  name                = "sns-logger-warmer-${var.sns_topic_name}"
   description         = "Keeps ${var.lambda_func_name} Warm"
   schedule_expression = "rate(15 minutes)"
 }

main.tf

@@ -1,6 +1,6 @@
-# -----------------------------------------------------------------
+# ----------------------------------------------------------------
 # AWS SNS TO CLOUDWATCH LOGS LAMBDA GATEWAY - OUTPUTS
-# -----------------------------------------------------------------
+# ----------------------------------------------------------------
 
 output "lambda_name" {
   description = "Name assigned to Lambda Function."

outputs.tf

@@ -58,7 +58,7 @@ variable lambda_func_name {
 
 variable lambda_description {
   type        = "string"
-  default     = "Route SNS messages to CloudWatch Logs"
+  default     = ""
   description = "Description to assign to Lambda Function."
 }
 
@@ -82,6 +82,12 @@ variable lambda_mem_size {
   description = "Amount of RAM (in MB) assigned to the function. The default (and minimum) is 128MB, and the maximum is 3008MB."
 }
 
+variable lambda_runtime {
+  type        = "string"
+  default     = "python3.6"
+  description = "Lambda runtime to use for the function."
+}
+
 variable "lambda_tags" {
   description = "A mapping of tags to assign to Lambda Function."
   default     = {}