[IT-4821] Integrate cloudwatch with linux docker product

This template provides system-level monitoring for general-purpose Docker
hosts, allowing users to add their own container-specific logging as needed.

Enhanced EC2 Docker product with CloudWatch agent integration.

IAM Permissions: Added CloudWatchAgentServerPolicy to the instance
role for CloudWatch agent access

CloudWatch Log Groups: Created dedicated log group for:
System logs (/aws/ec2/system/{StackName})

CloudWatch Agent Installation:
Added automated installation and configuration of the CloudWatch agent with:

System metrics collection (CPU, memory, disk)
System log file monitoring (messages, secure, cloud-init)
60-second collection intervals

Stack Outputs: Added helpful outputs showing log group names and
CloudWatch console links

Monitoring is enabled for the EC2 instance system-level metrics and logs,
organized by stack name and instance ID for easy identification.
Log group has a 30-day retention period to manage costs effectively.

Author: zaro0508

templates/ec2/README.md

@@ -4,3 +4,26 @@ This reference architecture creates an AWS Service Catalog Portfolio called "Ser
 * EC2 Linux with docker Integration: Provision an AWS Linux EC2 instance with docker and docker compose.
 * Notebook Linux EC2 Instance with Jumpcloud Integration, which builds one EC2 instance using an Ubuntu Bionic-based Rstudio AMI.
 * EC2 Windows with Jumpcloud Integration, which builds one EC2 instance using a Windows Server 2019 AMI.
+
+## Logging
+These templates have been enhanced with AWS CloudWatch agent integration to monitor both system metrics and Docker container logs.
+
+### CloudWatch Log Groups
+The template creates log groups automatically with system Metrics
+
+The CloudWatch agent collects:
+- CPU usage (idle, iowait, user, system)
+- Memory usage percentage
+- Disk usage percentage
+
+### Log Retention
+
+All log groups are configured with a 30-day retention period to manage costs.
+
+### Docker Container Logging
+
+Each Docker container is configured with the `awslogs` logging driver to send logs directly to CloudWatch. The logs are organized by:
+- Instance ID as the log stream prefix
+- Container name as the log stream suffix
+
+This provides clear separation and easy identification of logs from different containers and instances.

templates/ec2/sc-ec2-linux-docker.yaml

@@ -92,6 +92,13 @@ Parameters:
     MinValue: 30
     MaxValue: 5000
 Resources:
+  # CloudWatch Log Groups for system monitoring
+  SystemLogGroup:
+    Type: AWS::Logs::LogGroup
+    Properties:
+      LogGroupName: !Sub '/aws/ec2/system/${AWS::StackName}'
+      RetentionInDays: 30
+
   TgwHubSecurityGroup:
     Type: AWS::EC2::SecurityGroup
     Metadata:
@@ -122,6 +129,7 @@ Resources:
           'Fn::Sub': '${AWS::Region}-get-role-policy-ReadAssumedRoleInformationPolicy'
         - "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
         - "arn:aws:iam::aws:policy/AWSQuickSetupPatchPolicyBaselineAccess" #For SSM patching
+        - 'arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy' #For CloudWatch agent
       AssumeRolePolicyDocument:
         Version: '2012-10-17'
         Statement:
@@ -153,6 +161,8 @@ Resources:
             - cfn_hup_service
           SetEnv:
             - set_env_vars
+            - install_cloudwatch_agent
+            - configure_cloudwatch_agent
         cfn_hup_service:
           files:
             /etc/cfn/cfn-hup.conf:
@@ -208,6 +218,82 @@ Resources:
                 AWS_REGION: !Ref AWS::Region
                 STACK_NAME: !Ref AWS::StackName
                 STACK_ID: !Ref AWS::StackId
+        install_cloudwatch_agent:
+          packages:
+            yum:
+              amazon-cloudwatch-agent: []
+        configure_cloudwatch_agent:
+          files:
+            /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json:
+              content: !Sub |
+                {
+                  "agent": {
+                    "metrics_collection_interval": 60,
+                    "run_as_user": "cwagent"
+                  },
+                  "logs": {
+                    "logs_collected": {
+                      "files": {
+                        "collect_list": [
+                          {
+                            "file_path": "/var/log/messages",
+                            "log_group_name": "/aws/ec2/system/${AWS::StackName}",
+                            "log_stream_name": "{instance_id}/messages",
+                            "timezone": "UTC"
+                          },
+                          {
+                            "file_path": "/var/log/secure",
+                            "log_group_name": "/aws/ec2/system/${AWS::StackName}",
+                            "log_stream_name": "{instance_id}/secure",
+                            "timezone": "UTC"
+                          },
+                          {
+                            "file_path": "/var/log/cloud-init-output.log",
+                            "log_group_name": "/aws/ec2/system/${AWS::StackName}",
+                            "log_stream_name": "{instance_id}/cloud-init",
+                            "timezone": "UTC"
+                          }
+                        ]
+                      }
+                    }
+                  },
+                  "metrics": {
+                    "namespace": "CWAgent",
+                    "metrics_collected": {
+                      "cpu": {
+                        "measurement": [
+                          "cpu_usage_idle",
+                          "cpu_usage_iowait",
+                          "cpu_usage_user",
+                          "cpu_usage_system"
+                        ],
+                        "metrics_collection_interval": 60,
+                        "totalcpu": false
+                      },
+                      "disk": {
+                        "measurement": [
+                          "used_percent"
+                        ],
+                        "metrics_collection_interval": 60,
+                        "resources": [
+                          "*"
+                        ]
+                      },
+                      "mem": {
+                        "measurement": [
+                          "mem_used_percent"
+                        ],
+                        "metrics_collection_interval": 60
+                      }
+                    }
+                  }
+                }
+              mode: "000644"
+              owner: "root"
+              group: "root"
+          commands:
+            start_cloudwatch_agent:
+              command: "/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json"
     Properties:
       ImageId: !FindInMap [AMIs, !Ref AMI, AmiId]
       InstanceType: !Ref 'EC2InstanceType'
@@ -268,3 +354,10 @@ Outputs:
   Documentation:
     Description: 'Service Catalog Documentation'
     Value: "https://help.sc.sageit.org/sc/Service-Catalog-Provisioning.938836322.html"
+  CloudWatchLogGroups:
+    Description: 'CloudWatch Log Groups for monitoring'
+    Value: !Sub |
+      System Logs: /aws/ec2/system/${AWS::StackName}
+  CloudWatchConsoleURI:
+    Description: 'CloudWatch Logs Console'
+    Value: !Sub "https://${AWS::Region}.console.aws.amazon.com/cloudwatch/home?region=${AWS::Region}#logsV2:log-groups"