Merge pull request #59 from hallieswan/MG-182

MG-182: update agora infra based on improvements to model-ad infra

Author: hallieswan

.env.example

@@ -48,7 +48,7 @@ jobs:
           role-session-name: ${{ inputs.role-session-name }}
           role-duration-seconds: ${{ inputs.role-duration-seconds }}
       - name: CDK deploy
-        run: cdk deploy --all --concurrency 5 --require-approval never
+        run: cdk deploy --all --debug --concurrency 5 --require-approval never
         env:
           ENV: ${{ inputs.environment }}
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

.github/workflows/aws-deploy.yaml

@@ -29,4 +29,4 @@ jobs:
         env:
           ENV: ${{ inputs.environment }}
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-        run: cdk synth --output ./cdk.out
+        run: cdk synth --debug --output ./cdk.out

.github/workflows/test.yaml

@@ -1,14 +1,17 @@
 
 # AWS CDK app
 
-AWS CDK app for deploying Agora.
+A Github template using the AWS CDK to create an ECS infrastructure project for deploying Agora.
 
 # Prerequisites
 
 AWS CDK projects require some bootstrapping before synthesis or deployment.
 Please review the [bootstapping documentation](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html#getting_started_bootstrap)
 before development.
 
+> [!Note]
+> Sage IT deploys this CDK bootstrap upon creation of every AWS account in our AWS Organization.
+
 # Dev Container
 
 This repository provides a [dev container](https://containers.dev/) that includes all the tools
@@ -80,7 +83,12 @@ Please install pre-commit, once installed the file validations will
 automatically run on every commit.  Alternatively you can manually
 execute the validations by running `pre-commit run --all-files`.
 
-Create a [GitHub classic PAT](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#creating-a-personal-access-token-classic) with `read:packages` access, then create a .env file using .env.example as a template.
+Create a [GitHub classic personal access token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#creating-a-personal-access-token-classic) with `read:packages` access. The token is used to authenticate with the GitHub API and access the packages endpoint to look up the latest image version for each service. Create an `.env` file with the following variables:
+
+```
+GITHUB_TOKEN="your-token-here"
+ENV="dev"
+```
 
 Verify CDK to Cloudformation conversion by running [cdk synth]:
 
@@ -99,7 +107,7 @@ python -m pytest tests/ -s -v
 ```
 
 
-# Environments
+## Environments
 
 An `ENV` environment variable must be set when running the `cdk` command tell the
 CDK which environment's variables to use when synthesising or deploying the stacks.
@@ -110,7 +118,7 @@ Set environment variables for each environment in the [app.py](./app.py) file:
 environment_variables = {
     "VPC_CIDR": "10.254.192.0/24",
     "FQDN": "dev.app.io",
-    "CERTIFICATE_ARN": "arn:aws:acm:us-east-1:XXXXXXXXXXX:certificate/0e9682f6-3ffa-46fb-9671-b6349f5164d6",
+    "CERTIFICATE_ID": "0e9682f6-3ffa-46fb-9671-b6349f5164d6",
     "TAGS": {"CostCenter": "NO PROGRAM / 000000"},
 }
 ```
@@ -121,15 +129,20 @@ For example, synthesis with the `prod` environment variables:
 ENV=prod cdk synth
 ```
 
-# Certificates
+> [!NOTE]
+> The `VPC_CIDR` must be a unique value within our AWS organization. Check our
+> [wiki](https://sagebionetworks.jira.com/wiki/spaces/IT/pages/2850586648/Setup+AWS+VPC)
+> for information on how to obtain a unique CIDR
+
+## Certificates
 
 Certificates to set up HTTPS connections should be created manually in AWS certificate manager.
 This is not automated due to AWS requiring manual verification of the domain ownership.
 Once created take the ARN of the certificate and set that ARN in environment_variables.
 
 ![ACM certificate](docs/acm-certificate.png)
 
-# Secrets
+## Secrets
 
 Secrets can be manually created in the
 [AWS Secrets Manager](https://docs.aws.amazon.com/secretsmanager/latest/userguide/create_secret.html).
@@ -185,11 +198,61 @@ import os
 
 my_secret = os.environ.get("SINGLE_VALUE_SECRET", None)
 ```
-
+![Secrets Manager secret](docs/secrets-manager-secret.png)
 
 > [!NOTE]
 > Retrieving secrets requires access to the AWS Secrets Manager
 
+## DNS
+
+A DNS CNAME must be created in org-formation after the initial
+deployment of the application to make the application available at the desired
+URL. The CDK application exports the DNS name of the Application Load Balancer
+to be consumed in org-formation. [An example PR setting up a CNAME](https://github.com/Sage-Bionetworks-IT/organizations-infra/pull/1299).
+
+Login to the AWS cloudformation console and navigate to the deployed stack `app-load-balancer`
+and click on the `Outputs` tab.  On the row whose key is `LoadBalancerDNS` look for the
+value in the `Export Name` column, e.g., `app-dev-load-balancer-dns`.
+![Cloudformation Load Balancer](docs/cloudformation-load-balancer.png)
+
+Now use the name in the `TargetHostName` definition, for example:
+
+```
+TargetHostName: !CopyValue [!Sub 'app-dev-load-balancer-dns', !Ref DnTDevAccount]
+```
+
+(You would also replace `DnTDevAccount` with the name of the account in which the application is deployed.)
+
+> [!NOTE]
+> Setting up the DNS cname should be done at the very end of this infra setup
+
+
+## Debugging
+
+Generally CDK deployments will create cloudformation events during a CDK deploy.
+The events can be viewed in the AWS console under the cloudformation service page.
+Viewing those events will help with errors during a deployment.  Below are cases
+where it might be difficult to debug due to misleading or insufficient error
+messages from AWS
+
+### Missing Secrets
+
+Each new environment (dev/staging/prod/etc..) may require adding secrets.  If a
+secret is not created for the environment you may get an error with the following
+stack trace..
+```
+Resource handler returned message: "Error occurred during operation 'ECS Deployment Circuit Breaker was triggered'." (RequestToken: d180e115-ba94-d8a2-acf9-abe17a3aaed9, HandlerErrorCode: GeneralServiceException)
+    new BaseService (/private/var/folders/qr/ztb40vmn2pncyh8jpsgfnrt40000gp/T/jsii-kernel-4PEWmj/node_modules/aws-cdk-lib/aws-ecs/lib/base/base-service.js:1:3583)
+    \_ new FargateService (/private/var/folders/qr/ztb40vmn2pncyh8jpsgfnrt40000gp/T/jsii-kernel-4PEWmj/node_modules/aws-cdk-lib/aws-ecs/lib/fargate/fargate-service.js:1:967)
+    \_ new ApplicationLoadBalancedFargateService (/private/var/folders/qr/ztb40vmn2pncyh8jpsgfnrt40000gp/T/jsii-kernel-4PEWmj/node_modules/aws-cdk-lib/aws-ecs-patterns/lib/fargate/application-load-balanced-fargate-service.js:1:2300)
+    \_ Kernel._create (/private/var/folders/qr/ztb40vmn2pncyh8jpsgfnrt40000gp/T/tmpqkmckdm2/lib/program.js:9964:29)
+    \_ Kernel.create (/private/var/folders/qr/ztb40vmn2pncyh8jpsgfnrt40000gp/T/tmpqkmckdm2/lib/program.js:9693:29)
+    \_ KernelHost.processRequest (/private/var/folders/qr/ztb40vmn2pncyh8jpsgfnrt40000gp/T/tmpqkmckdm2/lib/program.js:11544:36)
+    \_ KernelHost.run (/private/var/folders/qr/ztb40vmn2pncyh8jpsgfnrt40000gp/T/tmpqkmckdm2/lib/program.js:11504:22)
+    \_ Immediate._onImmediate (/private/var/folders/qr/ztb40vmn2pncyh8jpsgfnrt40000gp/T/tmpqkmckdm2/lib/program.js:11505:46)
+    \_ processImmediate (node:internal/timers:464:21)
+```
+
 # Deployment
 
 ## Bootstrap
@@ -269,7 +332,6 @@ AWS_PROFILE=itsandbox-dev AWS_DEFAULT_REGION=us-east-1 aws ecs execute-command \
   --command "/bin/sh" --interactive
 ```
 
-
 # CI Workflow
 
 This repo has been set up to use Github Actions CI to continuously deploy the application.
@@ -284,3 +346,66 @@ The workflow for continuous integration:
 * CI deploys changes to the staging environment (stage.app.io) in the AWS prod account.
 * Changes are promoted (or merged) to the git prod branch.
 * CI deploys changes to the prod environment (prod.app.io) in the AWS prod account.
+
+![CI deployment workflow](docs/ci-deployment-workflow.png)
+
+# Deployment Process
+
+## Overview
+
+The source code for the application lives in the [sage-monorepo](https://github.com/Sage-Bionetworks/sage-monorepo). When a new git tag is added in the monorepo, images are published to [GHCR](https://github.com/orgs/Sage-Bionetworks/packages?tab=packages&q=agora). These images are deployed with GHA to AWS Fargate using infrastructure code in this repo.
+
+### Dev
+
+| ![Tags on latest image published by sage-monorepo](docs/dev-package-tags.png) |
+| :---: |
+| Tags on latest image published by sage-monorepo |
+
+The development environment was created so that we can redeploy easily. Whenever code in the sage-monorepo is merged after PR was reviewed and all checks approved, a GHA job publishes new images tagged with `edge` and the commit SHA for each service in the stack that changed (e.g. see screenshot of [agora-app package](https://github.com/sage-bionetworks/sage-monorepo/pkgs/container/agora-app) above). The dev infra stack points at the `edge` image tag. So we can rerun the latest `deploy-dev` GHA job to redeploy the latest app code, by looking up the SHA of the most recent image using the `edge` tag and the GitHub API.
+
+So, the footer of the dev site will only show the image tag of the `agora-app` package, which in this case is the commit SHA.
+
+### Stage/Prod
+
+| ![git tag on image published by sage-monorepo](docs/stageprod-package-tags.png) |
+| :---: |
+| git tag on image published by sage-monorepo |
+
+Stage and prod environments point at a specific git tag that is manually added. As described below, when a new git tag is manually created for this app in the sage-monorepo, a GHA job publishes new images tagged with that git tag (see screenshot above). Then, the stage or prod environment variavles in the infra repo can be updated to point at that tag. When the changes are merged to the `stage` or `prod` branch, a GHA job will run the `deploy-stage` or `deploy-prod` job accordingly, which will deploy the images tagged with the git tag.
+
+## Dev Deployment
+
+1. When a PR that affects this app is merged to `main` in the sage-monorepo, wait for the [ci job](https://github.com/Sage-Bionetworks/sage-monorepo/actions/workflows/ci.yml) to finish building and publishing images for the affected projects to GHCR.
+2. Rerun the last [deploy-dev job](https://github.com/Sage-Bionetworks-IT/agora-infra-v3/actions/workflows/deploy-dev.yaml) and wait for the job to successfully update dev deployment. Deployment can be monitored in AWS console in AWS ECS.
+3. Confirm that [dev site](https://agora-dev.adknowledgeportal.org/) shows changes from last merged PR.
+
+## Staging Deployment
+
+1. Review the list of existing tags [here](https://github.com/Sage-Bionetworks/sage-monorepo/tags). Identify the next `agora` tag. For example, if the last `agora` tag is `agora/v4.0.0-rc2`, then the next tag will be `agora/v4.0.0-rc3`. If the last `agora` tag doesn’t have a release candidate suffix (e.g. `agora/v4.0.0`), then the next tag will be the first release candidate of a new version (e.g. `agora/v4.0.1-rc1`). This follows the convention found at [semver.org](https://semver.org/).
+2. Create a new git tag in the sage-monorepo:
+   - Open devcontainer
+   - Checkout the main branch: `git checkout main`
+   - Fetch latest changes: `git fetch upstream`
+   - Rebase: `git rebase upstream/main`
+   - Tag the commit: `git tag agora/v4.0.0-rc3`
+   - Push the tag: `git push upstream tag agora/v4.0.0-rc3`
+3. Wait for sage-monorepo [release GHA job](https://github.com/Sage-Bionetworks/sage-monorepo/actions/workflows/release.yml) to successfully build, tag, and push images to GHCR.
+4. Create PR in this repo **to the dev branch** that sets `GHCR_PACKAGE_VERSION` for stage and prod environments to the new version number in `app.py`, since the images are only tagged with the version number (e.g. `4.0.0-rc3`) rather than the full tag name (e.g. `agora/v4.0.0-rc3` ).
+5. Merge PR. Wait for [deploy-dev job](https://github.com/Sage-Bionetworks-IT/agora-infra-v3/actions/workflows/deploy-dev.yaml) to successfully update dev deployment. Deployment can be monitored in AWS console in AWS ECS.
+6. Create PR in this repo **to merge dev into the stage branch**.
+7. Merge PR. Wait for [deploy-stage GHA job](https://github.com/Sage-Bionetworks-IT/modeladexplorer-v3/actions/workflows/deploy-stage.yaml) to successfully update staging deployment. Deployment can be monitored in AWS console in AWS ECS.
+8. Confirm that [staging site](https://agora-stage.adknowledgeportal.org/) shows new version’s tag in the app footer.
+
+## Production Deployment
+
+1. Go to the staging site and note the tag in the app footer. Identify the `agora` release tag. For example, if the staging site tag is `agora/v4.0.0-rc3`, then the release version will be `agora/release/v4.0.0`.
+2. Create a new git tag in the sage-monorepo:
+   - Open devcontainer
+   - Checkout the main branch: `git checkout main`
+   - Fetch latest changes: `git fetch upstream`
+   - Rebase: `git rebase upstream/main`
+   - Tag the commit: `git tag agora/release/v4.0.0`
+   - Push the tag: `git push upstream tag agora/release/v4.0.0`
+3. Create a PR in this repo **to merge the stage branch into the prod branch**.
+4. Merge PR. Wait for the [deploy-prod GHA job](https://github.com/Sage-Bionetworks-IT/modeladexplorer-v3/actions/workflows/deploy-prod.yaml) to successfully update prod deployment. Deployment can be monitored in AWS console in AWS ECS.
+5. Confirm that [production site](https://agora.adknowledgeportal.org/) shows the same tag in the app footer as the staging site.

README.md

@@ -25,6 +25,7 @@
             "CERTIFICATE_ID": "69b3ba97-b382-4648-8f94-a250b77b4994",
             "TAGS": {"CostCenter": "AMP-AD DCC / 101500", "Environment": "prod"},
             "AUTO_SCALE_CAPACITY": {"min": 2, "max": 4},
+            "GHCR_PACKAGE_VERSION": "4.0.0-rc4",
         }
     case "stage":
         environment_variables = {
@@ -33,6 +34,7 @@
             "CERTIFICATE_ID": "69b3ba97-b382-4648-8f94-a250b77b4994",
             "TAGS": {"CostCenter": "AMP-AD DCC / 101500", "Environment": "stage"},
             "AUTO_SCALE_CAPACITY": {"min": 2, "max": 4},
+            "GHCR_PACKAGE_VERSION": "4.0.0-rc4",
         }
     case "dev":
         environment_variables = {
@@ -41,6 +43,7 @@
             "CERTIFICATE_ID": "e8093404-7db1-4042-90d0-01eb5bde1ffc",
             "TAGS": {"CostCenter": "AMP-AD DCC / 101500", "Environment": "dev"},
             "AUTO_SCALE_CAPACITY": {"min": 1, "max": 2},
+            "GHCR_PACKAGE_VERSION": "edge",
         }
     case _:
         valid_envs_str = ",".join(VALID_ENVIRONMENTS)
@@ -51,13 +54,13 @@
 stack_name_prefix = f"agora-{environment}"
 fully_qualified_domain_name = environment_variables["FQDN"]
 environment_tags = environment_variables["TAGS"]
-agora_version = "edge"
+ghcr_package_version = environment_variables["GHCR_PACKAGE_VERSION"]
 docdb_master_username = "master"
 mongodb_port = 27017
 vpn_cidr = "10.1.0.0/16"
 
 # Get image versions
-if agora_version == "edge":
+if ghcr_package_version == "edge":
     app_version = get_alternate_tag_for_edge_package_version(
         "Sage-Bionetworks", "agora-app"
     )
@@ -68,7 +71,7 @@
         "Sage-Bionetworks", "agora-apex"
     )
 else:
-    app_version = api_version = apex_version = agora_version
+    app_version = api_version = apex_version = ghcr_package_version
 
 print(
     f"Using images: agora-app:{app_version}, agora-api:{api_version}, agora-apex:{apex_version}"