Merge pull request #35 from awslabs/1.5.0

1.5.0

Author: hackersifu

CHANGELOG.md

@@ -171,4 +171,14 @@
 
 ### Changed
 * References to Team DragonCat are now referred to Customer Incident Response Team (CIRT).
-* Various argparse help message to be more descriptive.
+* Various argparse help message to be more descriptive.
+
+## [1.5.0] - 2021-12-01
+
+###  Added
+* Ability to turn on Amazon S3 Server Access logs within single and multi-account modes.
+    * Added Dry Run capabilities for Amazon S3 Server Access logs within single and multi-account modes.
+    * Added Cleanup capabilities for Amazon S3 Server Access logs created by Assisted Log Enabler within single account mode.
+* Updated help (-h) message example within the README.
+* Permissions examples for enabling Amazon S3 Server Access logs within the permissions directory.
+* Diagram for Amazon S3 Server Access Logs within the README.

README.md

@@ -6,6 +6,7 @@ With Assisted Log Enabler for AWS, logging is turned on automatically for the va
 * AWS CloudTrail (Single Account Only)
 * Amazon Elastic Kubernetes Service (EKS) Audit and Authenticator Logs (Single Account and Multi-Account using AWS Organizations)
 * Amazon Route 53 Resolver Query Logs (Single Account and Multi-Account using AWS Organizations)
+* NEW! Amazon S3 Server Access Logs (Single Account and Multi-Account using AWS Organizations)
 
 Link to related AWS Open Source Blog Post: [Introducing Assisted Log Enabler for AWS](https://aws.amazon.com/blogs/opensource/introducing-assisted-log-enabler-for-aws/)
 
@@ -21,6 +22,10 @@ The following is a simple diagram on how Assisted Log Enabler for AWS works in a
 
 ![Alt text](diagrams/assisted_log_enabler.png)
 
+NEW! The following is a simple diagram on how Assisted Log Enabler for AWS works with turning on Amazon S3 Server Access Logging in a single account:
+
+![Alt text](diagrams/assisted_log_enabler_s3.png)
+
 ## Prerequisites
 ### Permissions
 The following permissions are needed within AWS IAM for Assisted Log Enabler for AWS to run. Please see each section for a breakdown per AWS Service and functionality:
@@ -48,6 +53,15 @@ The following permissions are needed within AWS IAM for Assisted Log Enabler for
 "route53resolver:ListTagsForResource",
 "route53resolver:DisassociateResolverQueryLogConfig",
 "route53resolver:DeleteResolverQueryLogConfig"
+"s3:PutBucketLogging",
+"s3:GetBucketLogging",
+"s3:ListBucket",
+"s3:ListAllMyBuckets",
+"s3:GetBucketLocation",
+"s3:GetBucketAcl",
+"s3:PutBucketAcl",
+"s3:PutBucketPublicAccessBlock",
+"s3:PutBucketLifecycleConfiguration"
 
 # For adding AWS CloudTrail logs:
 "s3:GetBucketPolicy",
@@ -86,7 +100,18 @@ The following permissions are needed within AWS IAM for Assisted Log Enabler for
 "route53resolver:AssociateResolverQueryLogConfig",
 "iam:CreateServiceLinkRole" # This is used to create the AWSServiceRoleForRoute53 Resolver, which is used for creating the Amazon Route 53 Query Logging Configurations.
 
-# NEW! For cleanup of Amazon Route 53 Resolver Query Logs created by Assisted Log Enabler for AWS:
+# NEW! For adding Amazon S3 Server Access Logs:
+"s3:PutBucketLogging",
+"s3:GetBucketLogging",
+"s3:ListBucket",
+"s3:ListAllMyBuckets",
+"s3:GetBucketLocation",
+"s3:GetBucketAcl",
+"s3:PutBucketAcl",
+"s3:PutBucketPublicAccessBlock",
+"s3:PutBucketLifecycleConfiguration"
+
+# For cleanup of Amazon Route 53 Resolver Query Logs created by Assisted Log Enabler for AWS:
 "route53resolver:ListResolverQueryLogConfigs",
 "route53resolver:ListTagsForResource",
 "route53resolver:ListResolverQueryLogConfigAssociations",
@@ -110,6 +135,9 @@ The following are the details of what happens within the Assisted Log Enabler fo
 * If no trail is configured, one is created and configured to log to the bucket created. (Single Account only as of this release)
 * If Amazon EKS Clusters exist, audit & authenticator logs are turned on.
 * Amazon Route 53 Query Logging is turned on for VPCs that do not have it turned on already.
+* NEW! Amazon S3 Server Access Logs are created for buckets that do not have it turned on already.
+   * This does not include for S3 buckets created by Assisted Log Enabler for AWS
+   * Amazon S3 Server Access Logs require buckets that reside in the same account & region, so additional buckets for Amazon S3 Server Access logs are created for this.
 
 
 ## Running the Code
@@ -149,27 +177,58 @@ No valid option selected. Please run with -h to display valid options.
 ```
 python3 assisted_log_enabler.py -h
 usage: assisted_log_enabler.py [-h] [--mode MODE] [--all] [--eks] [--vpcflow]
-                               [--r53querylogs] [--cloudtrail]
+                               [--r53querylogs] [--s3logs] [--cloudtrail]
+                               [--single_r53querylogs] [--single_cloudtrail]
+                               [--single_vpcflow] [--single_all]
+                               [--single_s3logs] [--single_account]
+                               [--multi_account]
 
 Assisted Log Enabler - Find resources that are not logging, and turn them on.
 
 optional arguments:
-  -h, --help      show this help message and exit
-  --mode MODE     Choose the mode that you want to run Assisted Log Enabler
-                  in. Available modes: single_account, multi_account. WARNING:
-                  For multi_account, You must have the associated
-                  CloudFormation template deployed as a StackSet. See the
-                  README file for more details.
-
-Service Options:
+  -h, --help            show this help message and exit
+  --mode MODE           Choose the mode that you want to run Assisted Log
+                        Enabler in. Available modes: single_account,
+                        multi_account, cleanup, dryrun. WARNING: For
+                        multi_account, You must have the associated
+                        CloudFormation template deployed as a StackSet. See
+                        the README file for more details.
+
+Single & Multi Account Options:
   Use these flags to choose which services you want to turn logging on for.
 
-  --all           Turns on all of the log types within the Assisted Log
-                  Enabler for AWS.
-  --eks           Turns on Amazon EKS audit & authenticator logs.
-  --vpcflow       Turns on Amazon VPC Flow Logs.
-  --r53querylogs  Turns on Amazon Route 53 Resolver Query Logs.
-  --cloudtrail    Turns on AWS CloudTrail.
+  --all                 Turns on all of the log types within the Assisted Log
+                        Enabler for AWS.
+  --eks                 Turns on Amazon EKS audit & authenticator logs.
+  --vpcflow             Turns on Amazon VPC Flow Logs.
+  --r53querylogs        Turns on Amazon Route 53 Resolver Query Logs.
+  --s3logs              Turns on Amazon Bucket Logs.
+  --cloudtrail          Turns on AWS CloudTrail.
+
+Cleanup Options:
+  Use these flags to choose which resources you want to turn logging off
+  for.
+
+  --single_r53querylogs
+                        Removes Amazon Route 53 Resolver Query Log resources
+                        created by Assisted Log Enabler for AWS.
+  --single_cloudtrail   Removes AWS CloudTrail trails created by Assisted Log
+                        Enabler for AWS.
+  --single_vpcflow      Removes Amazon VPC Flow Log resources created by
+                        Assisted Log Enabler for AWS.
+  --single_all          Turns off all of the log types within the Assisted Log
+                        Enabler for AWS.
+  --single_s3logs       Removes Amazon Bucket Log resources created by
+                        Assisted Log Enabler for AWS.
+
+Dry Run Options:
+  Use these flags to run Assisted Log Enabler for AWS in Dry Run mode.
+
+  --single_account      Runs Assisted Log Enabler for AWS in Dry Run mode for
+                        a single AWS account.
+  --multi_account       Runs Assisted Log Enabler for AWS in Dry Run mode for
+                        a multi-account AWS environment, using AWS
+                        Organizations.
 ```
 
 ### Step-by-Step Instructions (for running in AWS CloudShell, single account mode)
@@ -197,6 +256,8 @@ python3 assisted_log_enabler.py --mode single_account --vpcflow
 python3 assisted_log_enabler.py --mode single_account --r53querylogs
 # For AWS CloudTrail:
 python3 assisted_log_enabler.py --mode single_account --cloudtrail
+# NEW! For Amazon S3 Server Access Logs:
+python3 assisted_log_enabler.py --mode single_account --s3logs
 ```
 
 ### Step-by-Step Instructions (for running in AWS CloudShell, multi account mode)
@@ -254,6 +315,8 @@ python3 assisted_log_enabler.py --mode multi_account --eks
 python3 assisted_log_enabler.py --mode multi_account --vpcflow
 # For Amazon Route 53 Resolver Query Logs:
 python3 assisted_log_enabler.py --mode multi_account --r53querylogs
+# NEW! For Amazon S3 Server Access Logs:
+python3 assisted_log_enabler.py --mode multi_account --s3logs
 
 ```
 
@@ -286,7 +349,7 @@ To run Assisted Log Enabler for AWS in Dry Run mode, you can use the commands be
 # Single Account Dry Run
 python3 assisted_log_enabler.py --mode dryrun --single_account
 # Multi-Account Dry Run
-python3 assisted_log_enabler.py --mode dryrun --single_account
+python3 assisted_log_enabler.py --mode dryrun --multi_account
 ```
 
 ## Cleaning Up
@@ -303,6 +366,8 @@ python3 assisted_log_enabler.py --mode cleanup --single_r53querylogs
 python3 assisted_log_enabler.py --mode cleanup --single_vpcflow
 # To remove AWS CloudTrail trails created by Assisted Log Enabler for AWS (single account):
 python3 assisted_log_enabler.py --mode cleanup --single_cloudtrail
+# NEW! To remove Amazon S3 Server Access logging created by Assisted Log Enabler for AWS (single account):
+python3 assisted_log_enabler.py --mode cleanup --single_s3logs
 ```
 
 ## Shared Responsibility Model

assisted_log_enabler.py

@@ -13,7 +13,6 @@
 from botocore.exceptions import ClientError
 from datetime import timezone
 
-
 from subfunctions import ALE_multi_account
 from subfunctions import ALE_single_account
 from subfunctions import ALE_cleanup_single
@@ -82,13 +81,15 @@ def assisted_log_enabler():
     function_parser_group.add_argument('--eks', action='store_true', help=' Turns on Amazon EKS audit & authenticator logs.')
     function_parser_group.add_argument('--vpcflow', action='store_true', help=' Turns on Amazon VPC Flow Logs.')
     function_parser_group.add_argument('--r53querylogs', action='store_true', help=' Turns on Amazon Route 53 Resolver Query Logs.')
+    function_parser_group.add_argument('--s3logs', action='store_true', help=' Turns on Amazon Bucket Logs.')
     function_parser_group.add_argument('--cloudtrail', action='store_true', help=' Turns on AWS CloudTrail. Only available in Single Account version.')
 
     cleanup_parser_group = parser.add_argument_group('Cleanup Options', 'Use these flags to choose which resources you want to turn logging off for.')
     cleanup_parser_group.add_argument('--single_r53querylogs', action='store_true', help=' Removes Amazon Route 53 Resolver Query Log resources created by Assisted Log Enabler for AWS.')
     cleanup_parser_group.add_argument('--single_cloudtrail', action='store_true', help=' Removes AWS CloudTrail trails created by Assisted Log Enabler for AWS.')
     cleanup_parser_group.add_argument('--single_vpcflow', action='store_true', help=' Removes Amazon VPC Flow Log resources created by Assisted Log Enabler for AWS.')
     cleanup_parser_group.add_argument('--single_all', action='store_true', help=' Turns off all of the log types within the Assisted Log Enabler for AWS.')
+    cleanup_parser_group.add_argument('--single_s3logs', action='store_true', help=' Removes Amazon Bucket Log resources created by Assisted Log Enabler for AWS.')
 
     dryrun_parser_group = parser.add_argument_group('Dry Run Options', 'Use these flags to run Assisted Log Enabler for AWS in Dry Run mode.')
     dryrun_parser_group.add_argument('--single_account', action='store_true', help=' Runs Assisted Log Enabler for AWS in Dry Run mode for a single AWS account.')
@@ -106,6 +107,8 @@ def assisted_log_enabler():
             ALE_single_account.run_vpc_flow_logs()
         elif args.r53querylogs:
             ALE_single_account.run_r53_query_logs()
+        elif args.s3logs:
+            ALE_single_account.run_s3_logs()
         elif args.cloudtrail:
             ALE_single_account.run_cloudtrail()
         elif args.all:
@@ -119,13 +122,17 @@ def assisted_log_enabler():
             ALE_multi_account.run_vpc_flow_logs()
         elif args.r53querylogs:
             ALE_multi_account.run_r53_query_logs()
+        elif args.s3logs:
+            ALE_multi_account.run_s3_logs()
         elif args.all:
             ALE_multi_account.lambda_handler(event, context)
         else:
             logging.info("No valid option selected. Please run with -h to display valid options.")
     elif args.mode == 'cleanup':
         if args.single_r53querylogs:
             ALE_cleanup_single.run_r53_cleanup()
+        elif args.single_s3logs:
+            ALE_cleanup_single.run_s3_cleanup()
         elif args.single_cloudtrail:
             ALE_cleanup_single.run_cloudtrail_cleanup()
         elif args.single_vpcflow:

diagrams/assisted_log_enabler_s3.png

@@ -50,6 +50,15 @@ Resources:
               - route53resolver:ListResolverQueryLogConfigAssociations
               - route53resolver:CreateResolverQueryLogConfig
               - route53resolver:AssociateResolverQueryLogConfig
+              - s3:PutBucketLogging
+              - s3:GetBucketLogging
+              - s3:ListBucket
+              - s3:ListAllMyBuckets
+              - s3:GetBucketLocation
+              - s3:GetBucketAcl
+              - s3:PutBucketAcl
+              - s3:PutBucketPublicAccessBlock
+              - s3:PutBucketLifecycleConfiguration
             Resource: '*'
             Condition:
               StringEquals:

permissions/ALE_child_account_role.yaml

@@ -25,8 +25,16 @@
                 "eks:ListClusters",
                 "route53resolver:ListResolverQueryLogConfigAssociations",
                 "route53resolver:CreateResolverQueryLogConfig",
-                "route53resolver:AssociateResolverQueryLogConfig"
-
+                "route53resolver:AssociateResolverQueryLogConfig",
+                "s3:PutBucketLogging",
+                "s3:GetBucketLogging",
+                "s3:ListBucket",
+                "s3:ListAllMyBuckets",
+                "s3:GetBucketLocation",
+                "s3:GetBucketAcl",
+                "s3:PutBucketAcl",
+                "s3:PutBucketPublicAccessBlock",
+                "s3:PutBucketLifecycleConfiguration"
             ],
             "Resource": "*",
             "Condition": {

permissions/ALE_permissions_example_multi_account.json

@@ -26,7 +26,16 @@
                 "eks:ListClusters",
                 "route53resolver:ListResolverQueryLogConfigAssociations",
                 "route53resolver:CreateResolverQueryLogConfig",
-                "route53resolver:AssociateResolverQueryLogConfig"
+                "route53resolver:AssociateResolverQueryLogConfig",
+                "s3:PutBucketLogging",
+                "s3:GetBucketLogging",
+                "s3:ListBucket",
+                "s3:ListAllMyBuckets",
+                "s3:GetBucketLocation",
+                "s3:GetBucketAcl",
+                "s3:PutBucketAcl",
+                "s3:PutBucketPublicAccessBlock",
+                "s3:PutBucketLifecycleConfiguration"
             ],
             "Resource": "*",
             "Condition": {

permissions/ALE_permissions_example_single_account.json

@@ -178,6 +178,52 @@ def vpcflow_cleanup():
         except Exception as exception_handle:
             logging.error(exception_handle)
 
+# 4. Remove the S3 Logging Resources created by Assisted Log Enabler
+def s3_cleanup():
+    """Function to clean up Bucket Logs"""
+    logging.info("Cleaning up Bucket Logs created by Assisted Log Enabler for AWS.")
+    for aws_region in region_list:
+        s3 = boto3.client('s3', region_name=aws_region)
+        try:
+            logging.info("---- LINE BREAK BETWEEN REGIONS ----")
+            logging.info("Cleaning up Bucket Logs created by Assisted Log Enabler for AWS in region " + aws_region + ".")
+            removal_list: list = []
+            logging.info("ListBuckets API Call")
+            buckets = s3.list_buckets()
+            for bucket in buckets['Buckets']:
+                s3region=s3.get_bucket_location(Bucket=bucket["Name"])['LocationConstraint']
+                if s3region == aws_region:
+                    if 'aws-s3-log-collection-' not in str(bucket["Name"]):
+                        logging.info("Parsed out buckets created by Assisted Log Enabler for AWS in " + aws_region)
+                        logging.info("Checking remaining buckets to see if logs were enabled by Assisted Log Enabler for AWS in " + aws_region)
+                        logging.info("GetBucketLogging API Call for " + bucket["Name"])
+                        s3temp=s3.get_bucket_logging(Bucket=bucket["Name"])
+                        if 'aws-s3-log-collection-' in str(s3temp):
+                            removal_list.append(bucket["Name"])
+                elif s3region is None and aws_region == 'us-east-1':
+                    if 'aws-s3-log-collection-' not in str(bucket["Name"]):
+                        logging.info("Parsed out buckets created by Assisted Log Enabler for AWS in " + aws_region)
+                        logging.info("Checking remaining buckets to see if logs were enabled by Assisted Log Enabler for AWS in " + aws_region)
+                        logging.info("GetBucketLogging API Call for " + bucket["Name"])
+                        s3temp=s3.get_bucket_logging(Bucket=bucket["Name"])
+                        if 'aws-s3-log-collection-' in str(s3temp):
+                            removal_list.append(bucket["Name"])
+            if removal_list != []:
+                logging.info("List S3 Buckets with Logging enabled by by Assisted Log Enabler for AWS in " + aws_region)
+                print(removal_list)
+                for bucket in removal_list:
+                    logging.info("Removing S3 Bucket Logging for " + bucket)
+                    logging.info("PutBucketLogging API Call")
+                    delete_s3_log = s3.put_bucket_logging(
+                        Bucket=bucket,
+                        BucketLoggingStatus={}
+                    )
+                logging.info("Removed S3 Bucket Logging created by Assisted Log Enabler for AWS.")
+                time.sleep(1)
+            else:
+                logging.info("There are no S3 Bucket set by Log Enabler in " + aws_region)
+        except Exception as exception_handle:
+            logging.error(exception_handle)
 
 def run_vpcflow_cleanup():
     """Function to run the vpcflow_cleanup function"""
@@ -196,6 +242,10 @@ def run_r53_cleanup():
     r53_cleanup()
     logging.info("This is the end of the script. Please feel free to validate that logging resources have been cleaned up.")
 
+def run_s3_cleanup():
+    """Function to run the s3_cleanup function"""
+    s3_cleanup()
+    logging.info("This is the end of the script. Please feel free to validate that logging resources have been cleaned up.")
 
 def lambda_handler(event, context):
     """Function that runs all of the previously defined functions"""