Make setup.yml cfn template support multiple stacks in a single account, fix bug in petctl setp where None was being passed to cfn param, pump docker logs to cloudwatch

Summary:
1. Uses docker log-driver == awslogs to make docker output go to cloud watch (see screenshots below)
2. #1 creates a log group called `torchelastic/$USER` in CW and creates log streams (one per worker) called `$job_name/$instance_id`
3. Fixes a bug in `petctl setup` where if no efs and s3 buckets are specified the `NoneType` is passed to the cfn template param which throws a validation error because it expects a string
4. Fixes an issue with cfn template where the CloudWatch IAM managed policy was being created with a specific name hence preventing multiple stacks from being created in the same account.

#thanks Vinicius Reis for testing `petctl` and reporting bugs #3 and #4.

{F223965947}
{F223965943}

Reviewed By: vreis

Differential Revision: D18826855

fbshipit-source-id: 2d75f607734135ab6d5301fc636501a38cfee9d9

Author: Kiuk Chung

aws/README.md

@@ -133,8 +133,13 @@ Auto Scaling Groups
 1. etcd server 
 2. workers
 
-#### SSH onto worker nodes
-To SSH onto the worker nodes to inspect the worker process we use AWS 
+#### Inspect the logs
+Log into the AWS CloudWatch Logs console. You should see a log group called
+`torchelastic/$USER`. Under it there will be a log stream per instance with the 
+name `$job_name/$instance_id` (e.g. `my_job/i0b938EXAMPLE`).
+
+#### Troubleshooting
+To SSH onto the worker nodes to debug/inspect the worker process use AWS 
 Session Manager instead of the ec2 key pair. [Install](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-working-with-install-plugin.html)
  the Session Manager plugin and run
 
@@ -160,18 +165,18 @@ docker ps
 docker logs -f <container id>
 ```
 
+> Note since we have configured the log driver to be `awslogs` tailing
+the docker logs will not work. For more information see: https://docs.docker.com/config/containers/logging/awslogs/
+
 You can also manually stop and start the workers by running
 ``` bash
 systemctl stop torchelastic_worker
 systemctl start torchelastic_worker
 ```
 
-> **EXCERCISE:** Open up two terminals and SSH onto each worker. Tail the docker logs
-on each worker. Now stop worker 1 and observe the worker 2 re-rendezvous and
-since `--min_size=1` it continues training by itself. Now restart worker 1 and
-observe that worker 2 notices that worker 1 is waiting to join and re-rendezvous,
-the `state` object in worker 2 is `sync()`'ed to worker 1 and both resume training
-without loss of progress.
+> **EXCERCISE:** Try stopping or adding worker(s) to see elasticity in action!
+To add workers, simply increase the `desired` size of the worker autoscaling group.
+
 
 > **Note**: by design, `petctl` tries to use the least number of AWS services. This
 was done intentionally to allow non-AWS users to easily transfer the functionality

aws/cfn/setup.yml

@@ -252,7 +252,6 @@ Resources:
     Type: AWS::IAM::ManagedPolicy
     Properties:
       Description: "Allows container instances to use CloudWatch APIs"
-      ManagedPolicyName: "ContainerCloudWatchLogsPolicy"
       Path: "/"
       PolicyDocument:
         Version: "2012-10-17"

aws/cloudformation.py

@@ -10,6 +10,8 @@
 import getpass
 import logging
 import os
+import random
+import string
 
 from jinja2 import Template
 from util import wait_for
@@ -25,18 +27,23 @@ def __init__(self, session):
 
     def create_specs_file(self, specs_file, s3_bucket_name, efs_id):
         username = getpass.getuser()
-        stack_name = f"torchelastic-{username}"
+        rand = "".join(random.choices(string.ascii_uppercase + string.digits, k=5))
+        hash = f"{username}-{rand}"
+        stack_name = f"torchelastic-{hash}"
         this_dir = os.path.dirname(__file__)
         cfn_template = os.path.join(this_dir, "cfn/setup.yml")
         sample_specs = os.path.join(this_dir, "config/sample_specs.json")
 
         params = {
-            "S3BucketName": s3_bucket_name,
-            "EFSFileSystemId": efs_id,
-            "WorkerRoleName": f"torchelastic_worker_role-{username}",
-            "RendezvousRoleName": f"torchelastic_rendezvous_role-{username}",
+            "WorkerRoleName": f"torchelastic_worker_role-{hash}",
+            "RendezvousRoleName": f"torchelastic_rendezvous_role-{hash}",
         }
 
+        if s3_bucket_name:
+            params["S3BucketName"] = s3_bucket_name
+        if efs_id:
+            params["EFSFileSystemId"] = efs_id
+
         self.create_stack(stack_name, cfn_template, **params)
 
         for _ in wait_for(

aws/config/user_data_worker

@@ -47,6 +47,9 @@ cat > /var/torchelastic/run_worker <<\EOL
 container_name=$1
 shift
 
+region=$(curl -s http://169.254.169.254/latest/dynamic/instance-identity/document | jq .region -r)
+instance_id=$(curl -s http://169.254.169.254/latest/meta-data/instance-id)
+
 docker run \
   --init \
   --net=host \
@@ -55,6 +58,11 @@ docker run \
   --env-file /var/torchelastic/worker.env \
   -v /mnt/efs/fs1:/mnt/efs/fs1 \
   --name ${container_name} \
+  --log-driver=awslogs \
+  --log-opt awslogs-region=${region} \
+  --log-opt awslogs-group=torchelastic/{{ user }} \
+  --log-opt awslogs-create-group=true \
+  --log-opt awslogs-stream=${container_name}/${instance_id} \
   {{ docker_image }} $*
 EOL
 

aws/petctl.py

@@ -139,7 +139,7 @@ def parse_arguments(args, **default_args):
         help="s3 bucket to use for running petctl (if empty, one is created)",
     )
     parser_setup.add_argument(
-        "--efs_id", default="", help="efs id to use, if empty, one is created"
+        "--efs_id", help="efs id to use, if empty, one is created"
     )
 
     petctl_args, script_args = split_args(args[1:])
@@ -190,6 +190,7 @@ def run_job(session, specs_json, args):
     worker_specs["job_name"] = job_name
     worker_specs["script"] = script
     worker_specs["args"] = " ".join(script_args)
+    worker_specs["user"] = getpass.getuser()
 
     instance_type = worker_specs["instance_type"]
     script_args_str = worker_specs["args"]