Make public bucket optional (#495)

Co-authored-by: Michael Graeb <graebm@amazon.com>

Author: TingDaoK

tests/test_helper/README.md

@@ -24,36 +24,38 @@ python3 test_helper.py clean
 ### `init` action
 
 * Create `<BUCKET_NAME>` in us-west-2.
-* Add the lifecycle to automatic clean up the `upload/` after one day
-* Upload files:
-  + `pre-existing-10MB-aes256-c` [SSE-C](https://docs.aws.amazon.com/AmazonS3/latest/userguide/ServerSideEncryptionCustomerKeys.html#sse-c-highlights) encrypted fille
-  + `pre-existing-10MB-aes256` [SSE-S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/specifying-s3-encryption.html) encrypted fille
-  + `pre-existing-10MB-kms` [SSE-KMS](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingKMSEncryption.html) encrypted fille
-  + `pre-existing-10MB`
-  + `pre-existing-1MB`
-  + `pre-existing-empty`
-  + `pre-existing-error-xml`
-  + with `--large_objects` enabled, several large objects will also be uploaded. Currently, only aws-c-s3's tests require these files, the aws-crt-*** repos do not:
-    - `pre-existing-256MB`
-    - `pre-existing-256MB-@`
-    - `pre-existing-2GB`
-    - `pre-existing-2GB-@`
-
-* Create `<BUCKET_NAME>-public` in us-west-2
-* Upload files:
-  + `pre-existing-1MB` 1MB file with public read access.
+  + Add the lifecycle to automatic clean up the `upload/` and clean up incomplete multipart uploads after one day.
+  + Upload files:
+    - `pre-existing-10MB-aes256-c` [SSE-C](https://docs.aws.amazon.com/AmazonS3/latest/userguide/ServerSideEncryptionCustomerKeys.html#sse-c-highlights) encrypted fille
+    - `pre-existing-10MB-aes256` [SSE-S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/specifying-s3-encryption.html) encrypted fille
+    - `pre-existing-10MB-kms` [SSE-KMS](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingKMSEncryption.html) encrypted fille
+    - `pre-existing-10MB`
+    - `pre-existing-1MB`
+    - `pre-existing-empty`
+    - `pre-existing-error-xml`
+    - with `--large_objects` enabled, several large objects will also be uploaded. Currently, only aws-c-s3's tests require these files, the aws-crt-*** repos do not:
+      - `pre-existing-256MB`
+      - `pre-existing-256MB-@`
+      - `pre-existing-2GB`
+      - `pre-existing-2GB-@`
+
+* with `--create_public_bucket` enabled, create `<BUCKET_NAME>-public` in us-west-2
+  + Upload files:
+    - `pre-existing-1MB` 1MB file with public read access.
 
 * Create directory bucket `<BUCKET_NAME>--usw2-az1--x-s3` in us-west-2
-* Upload files:
-  + `pre-existing-10MB` 10MB file.
-  + with `--large_objects` enabled
-    - `pre-existing-2GB`
+  + Add the lifecycle to automatic clean up the `upload/` and clean up incomplete multipart uploads after one day.
+  + Upload files:
+    - `pre-existing-10MB` 10MB file.
+    - with `--large_objects` enabled
+      - `pre-existing-2GB`
 
 * Create directory bucket `<BUCKET_NAME>--use1-az4--x-s3` in us-east-1
-* Upload files:
-  + `pre-existing-10MB` 10MB file.
-  + with `--large_objects` enabled
-    - `pre-existing-2GB`
+  + Add the lifecycle to automatic clean up the `upload/` and clean up incomplete multipart uploads after one day.
+  + Upload files:
+    - `pre-existing-10MB` 10MB file.
+    - with `--large_objects` enabled
+      - `pre-existing-2GB`
 
 ### `clean` action
 

tests/test_helper/test_helper.py

@@ -28,12 +28,17 @@
 parser.add_argument(
     'bucket_name',
     nargs='?',
-    help='The bucket name base to use for the test buckets. If not specified, the $CRT_S3_TEST_BUCKET_NAME will be used, if set. Otherwise, a random name will be generated.')
+    help='The bucket name base to use for the test buckets. If not specified, the $CRT_S3_TEST_BUCKET_NAME will be used, if set. \
+        Otherwise, a random name will be generated.')
 parser.add_argument(
     '--large_objects',
     action='store_true',
     help='enable helper to create pre-existing large objects.')
-
+parser.add_argument(
+    '--create_public_bucket',
+    action='store_true',
+    help='Allow script to create a public bucket. If not specified, the script will not attempt to create a public bucket. \
+        Note: Some aws-c-s3 tests will fail without public bucket.')
 
 args = parser.parse_args()
 
@@ -256,7 +261,11 @@ def cleanup(bucket_name, availability_zone=None, client=s3_client):
     create_bucket_with_lifecycle("use1-az4", s3_client_east1, "us-east-1")
     create_bucket_with_lifecycle("usw2-az1")
     create_bucket_with_lifecycle()
-    create_bucket_with_public_object()
+    if args.create_public_bucket:
+        create_bucket_with_public_object()
+    else:
+        print("Skipping public bucket, run with --create_public_bucket if you need these.")
+
     if os.environ.get('CRT_S3_TEST_BUCKET_NAME') != BUCKET_NAME_BASE:
         print(
             f"*** Set the environment variable $CRT_S3_TEST_BUCKET_NAME to {BUCKET_NAME_BASE} before running the tests ***")