feat: bloom-dev open tofu config

Author: Avritt Rohwer

docker-compose.yml

@@ -107,10 +107,10 @@ services:
       DATABASE_URL: "postgres://postgres:example@db:5432/bloom_prisma"
     healthcheck:
       test: ["CMD", "curl", "--fail", "http://127.0.0.1:3100/"]
-      interval: "2s"
-      timeout: "1s"
-      retries: 20
-      start_period: "1s"
+      interval: "5s"
+      timeout: "2s"
+      retries: 10
+      start_period: "5s"
     depends_on:
       db:
         condition: service_healthy

infra/README.md

@@ -10,47 +10,53 @@ Cloud Native Computing Foundation.
   is a set of resources that are all managed together. Each root module has a state file that
   records the results of the latest apply operation.
 
+   - [bloom_dev](./tofu_root_modules/bloom_dev/README.md): Configures the bloom-dev AWS account.
    - [bloom_dev_deployer_permission_set](./tofu_root_modules/bloom_dev_deployer_permission_set/README.md):
      Configures the bloom-dev-deployer permission set that is assigned on the bloom-dev account.
 
+- [tofu_importable_modules](./tofu_importable_modules): Contains all the Open Tofu importable
+  modules. An importable module is a reusable set of resources configured through input
+  parameters. Root modules import importable modules.
+
+   - [bloom_deployment](./tofu_importable_modules/bloom_deployment/): Configures all the resources
+     needed for a Bloom deployment in a single AWS account.
+
+
+
 ## Infrastructure-as-code mental model
 
 Let's say that you need to deploy Bloom to an AWS account. A straight-forward way of achieving this
 would be to log into the AWS web console and create all the required resources. The downside of this
-approach is that unless you take really good notes, it is a difficult process to replicate. Even if
-you take really good notes, the process might have a lot of steps which are all opportunities for
-making mistakes. Additionally, it is not possible to automate such a process (well, maybe if you
-have one of those neat AIs that can control your browser. But the AI could also make mistakes just
-like a human).
+approach is that unless you take really good notes, it is a difficult process to replicate.
 
 Another approach could be to write a bash script that calls a bunch of AWS CLI commands that create
 the resources. This improves on the web-based approach because all the steps are explicitly written
 down. However, the script only works on a fresh AWS account - if you run it again there will be
 a bunch of errors because the resources will have already been created. If you need to change how
-the account is configured, you need to write more scripts. That reminds me too much of database
-evolutions to seem like an good idea...
-
-Enter infrastructure-as-code tools like Open Tofu. I like to think of them as CLI scripting with
-a bunch of functionality already built in. We have `.tf` files that contain [resource
-descriptions](https://opentofu.org/docs/language/resources/) for everything we want to configure. We
-can run the 'script' by running [`tofu apply`](https://opentofu.org/docs/cli/commands/apply/). If
-the AWS account already matches our desired configuration, Tofu gives us a nice message that the
-infrastructure matches the configuration. Otherwise, Tofu presents us with a list of planned changes
-it thinks it needs to make and asks if it should go forward with the plan.
-
-These tools are not magic, however. It is still possible to misconfigure resources and get errors
-from the AWS API. These cases are not always handled gracefully and sometimes require deleting or
-configuring things manually to unbork the tool. Using a infrastructure-as-code still requires manual
-testing and knowledge of the underlying systems you are configuring. It is a heck of a lot better
-than shell scripting, however, at least in my experience :)
-
-_Monologue by Avritt Rohwer_
+the account is configured, you need to write more scripts.
+
+Enter infrastructure-as-code tools like Terraform and Open Tofu. I like to think of them as CLI
+scripting with a bunch of functionality already built in. `.tf` files contain [resource
+descriptions](https://opentofu.org/docs/language/resources/). Run the 'script' by running [`tofu
+apply`](https://opentofu.org/docs/cli/commands/apply/). If the AWS account already matches the
+desired configuration, Tofu prints a nice message that the infrastructure matches the
+configuration. Otherwise, Tofu presents us with a list of planned changes it thinks it needs to make
+and asks if it should go forward with the plan.
+
+It is still possible to misconfigure resources and get errors from the AWS API. These cases are not
+always handled gracefully and sometimes require deleting or configuring things manually to unblock
+the tool. Using a infrastructure-as-code still requires manual testing and knowledge of the
+underlying systems you are configuring. It is a heck of a lot better than shell scripting, however,
+at least in my experience :)
 
 ## Developer setup
 
-1. Install required tools:
-   1. Open Tofu: https://opentofu.org/docs/intro/install/
-   2. AWS CLI: https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html
+1. Install required CLI tools:
+   1. bash: `which bash`
+   2. openssl: `which openssl`
+   3. tr: `which tr`
+   4. Open Tofu: https://opentofu.org/docs/intro/install/
+   5. AWS CLI: https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html
 
       After installing, edit your `~/.aws/config` file for SSO authentication:
 
@@ -100,8 +106,7 @@ unless you add a provider. In that case, run:
 tofu init
 ```
 
-To update a required version for a provider, change the version in the relevant `main.tf` file then
-run:
+To update a required version for a provider, change the version then run:
 
 ```bash
 tofu init -upgrade
@@ -113,15 +118,27 @@ directory.
 ### Applying changes
 
 1. Open a shell and change directory to the root module.
-2. Run `aws sso login` to authenticate to AWS. After 1 hour, you will need to re-authenticate using
-   the same command.
-3. Edit the `main.tf` file to update the desired configuration.
+2. Run `aws sso login` to authenticate to AWS.
+3. Edit the tofu files for the desired configuration.
 4. Run `tofu apply` and review the planned changes. If there are unexpected planned changes, go back
    to step 1. If all the changes are expected, approve the apply.
 5. Inspect the relevant AWS resources via the CLI or the AWS web console
    (Log in via https://d-9067ac8222.awsapps.com/start). If there are unexpected results, go back to
-   step 1. In some cases you may have to manually modify or delete resources directly to 'unstick'
+   step 3. In some cases you may have to manually modify or delete resources directly to 'unstick'
    Open Tofu.
+6. To delete only the resources provisioned by the bloom_deployment module, run `tofu destroy
+   -target=module.bloom_deployment`.
+
+#### Forcing resource recreation
+
+To force Tofu to replace a resource, run `tofu apply -recreate=ADDRESS`. For example:
+
+```
+tofu apply -recreate=module.bloom_deployment.aws_secretsmanager_secret.api_jwt_signing_key
+```
+
+This is helpful when testing the local-exec provisioner because the provisioner only runs on
+resource creation.
 
 ## AWS setup done manually
 

infra/tofu_importable_modules/bloom_deployment/db.tf

@@ -0,0 +1,51 @@
+# Create a database.
+locals {
+  # Pricing: https://aws.amazon.com/rds/postgresql/pricing/?pg=pr&loc=3
+  # Machine specs: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.DBInstanceClass.Summary.html#hardware-specifications.burstable-inst-classes
+  db_instance_class = local.is_prod ? "db.t4g.medium" : "db.t4g.micro"
+  db_multi_az       = local.is_prod ? true : false
+
+  # https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Storage.html#gp2-storage
+  # Unit: GiB
+  db_start_storage = local.is_prod ? 10 : 5
+  db_max_storage   = local.is_prod ? 50 : 10
+
+  # Unit: days
+  db_backup_retention = local.is_prod ? 30 : 7
+}
+resource "aws_db_subnet_group" "bloom" {
+  region     = var.aws_region
+  name       = "bloom"
+  subnet_ids = [for s in aws_subnet.private : s.id]
+}
+resource "aws_db_instance" "bloom" {
+  identifier                      = "bloom"
+  deletion_protection             = local.is_prod
+  engine                          = "postgres"
+  engine_version                  = "17"
+  instance_class                  = local.db_instance_class
+  multi_az                        = local.db_multi_az
+  enabled_cloudwatch_logs_exports = ["postgresql", "upgrade", "iam-db-auth-error"]
+  username                        = "master"
+  manage_master_user_password     = true
+
+  # Networking
+  vpc_security_group_ids              = [aws_security_group.db.id]
+  iam_database_authentication_enabled = true
+  db_subnet_group_name                = aws_db_subnet_group.bloom.id
+
+  # Updates
+  apply_immediately           = true # If false, any changes are applied in the next maintenance window instead of when tofu apply runs.
+  engine_lifecycle_support    = "open-source-rds-extended-support"
+  allow_major_version_upgrade = false
+  auto_minor_version_upgrade  = true
+
+  # Storage
+  storage_encrypted         = true
+  storage_type              = "gp2"
+  allocated_storage         = local.db_start_storage
+  max_allocated_storage     = local.db_max_storage
+  backup_retention_period   = local.db_backup_retention
+  final_snapshot_identifier = "bloom-db-finalsnapshot"
+  skip_final_snapshot       = !local.is_prod
+}

infra/tofu_importable_modules/bloom_deployment/ecs.tf

@@ -0,0 +1,189 @@
+# Create an ECS cluster and services for each Bloom binary.
+resource "aws_iam_service_linked_role" "ecs" {
+  aws_service_name = "ecs.amazonaws.com"
+}
+resource "aws_ecs_cluster" "bloom" {
+  region     = var.aws_region
+  name       = "bloom"
+  depends_on = [aws_iam_service_linked_role.ecs]
+  setting {
+    name  = "containerInsights"
+    value = "enabled"
+  }
+}
+
+# Set up service discovery for the sites to talk to the api.
+resource "aws_service_discovery_http_namespace" "bloom" {
+  region      = var.aws_region
+  name        = "bloom"
+  description = "Service namespace the bloom services use."
+}
+
+# Create logs groups.
+locals {
+  logs_retention = local.is_prod ? 30 : 7
+}
+resource "aws_cloudwatch_log_group" "task_logs" {
+  for_each = toset([
+    "bloom-api",
+    "bloom-site-partners",
+    "bloom-site-public"
+  ])
+  region            = var.aws_region
+  name              = each.value
+  log_group_class   = "STANDARD"
+  retention_in_days = local.logs_retention
+}
+
+# Create a secret key used by the API to sign JWTs.
+resource "aws_secretsmanager_secret" "api_jwt_signing_key" {
+  region                  = var.aws_region
+  description             = "Key used by the Bloom API to sign JWTs"
+  name_prefix             = "bloom-api-jwt-signing-key" # avoids 'you can't create this secret because a secret with this name is already scheduled for deletion' issue when re-deploying an account.
+  recovery_window_in_days = 7                           # minimum
+
+  # The provisioner block runs after the resource has been created, and never again.
+  provisioner "local-exec" {
+    interpreter = ["/usr/bin/env", "bash", "-c"]
+    # We need to be very careful that any errors result in a non-zero exit code. Otherwise tofu will
+    # think this block succeeded and not error.
+    command = <<-EOT
+    if ! type -P aws &>/dev/null; then
+      echo 'ERROR: aws required'
+      exit 1
+    fi
+    if ! type -P openssl &>/dev/null; then
+      echo 'ERROR: openssl required'
+      exit 1
+    fi
+    if ! type -P tr &>/dev/null; then
+      echo 'ERROR: tr required'
+      exit 1
+    fi
+
+    if ! s=$(openssl rand -base64 256 | tr -d '\n'); then
+        echo 'ERROR: failed to generate random value'
+        exit 1
+    fi
+
+    if ! aws secretsmanager put-secret-value \
+         --profile ${var.aws_profile} \
+         --region ${var.aws_region} \
+         --secret-id ${self.id} \
+         --secret-string "$s"
+    then
+        echo 'ERROR: failed to put secret value'
+        exit 1
+    fi
+    EOT
+  }
+}
+
+locals {
+  roles = {
+    "api" = {
+      task_execution_policy_extra_statements = [
+        {
+          Action   = "secretsmanager:GetSecretValue"
+          Effect   = "Allow"
+          Resource = aws_db_instance.bloom.master_user_secret[0].secret_arn
+        },
+        {
+          Action   = "secretsmanager:GetSecretValue"
+          Effect   = "Allow"
+          Resource = aws_secretsmanager_secret.api_jwt_signing_key.arn
+        }
+      ]
+    }
+    "site-partners" = {
+      task_execution_policy_extra_statements = []
+    }
+    "site-public" = {
+      task_execution_policy_extra_statements = []
+    }
+  }
+}
+
+# Create roles for the ECS task executor and the tasks.
+resource "aws_iam_role" "bloom_ecs" {
+  for_each    = local.roles
+  name        = "bloom-${each.key}-ecs"
+  description = "Role the ECS service uses when launching Bloom ${each.key} tasks."
+  # https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html#create_task_iam_policy_and_role
+  assume_role_policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [{
+      Action = "sts:AssumeRole"
+      Effect = "Allow"
+      Principal = {
+        Service = "ecs-tasks.amazonaws.com"
+      }
+      Condition = {
+        ArnLike = {
+          "aws:SourceArn" = "arn:aws:ecs:${var.aws_region}:${var.aws_account_number}:*"
+        }
+        StringEquals = {
+          "aws:SourceAccount" = var.aws_account_number
+        }
+      }
+    }]
+  })
+}
+resource "aws_iam_role_policy" "bloom_ecs" {
+  for_each = local.roles
+  name     = "bloom-${each.key}-ecs"
+  role     = aws_iam_role.bloom_ecs[each.key].id
+  policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = concat(
+      [
+        {
+          Action = [
+            "logs:CreateLogStream",
+            "logs:PutLogEvents",
+          ]
+          Effect   = "Allow"
+          Resource = "${aws_cloudwatch_log_group.task_logs["bloom-${each.key}"].arn}:log-stream:*"
+        },
+      ],
+      each.value.task_execution_policy_extra_statements
+    )
+  })
+}
+resource "aws_iam_role" "bloom_container" {
+  for_each    = local.roles
+  name        = "bloom-${each.key}-container"
+  description = "Role the Bloom ${each.key} container runs as."
+  # https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html#create_task_iam_policy_and_role
+  assume_role_policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [{
+      Action = "sts:AssumeRole"
+      Effect = "Allow"
+      Principal = {
+        Service = "ecs-tasks.amazonaws.com"
+      }
+      Condition = {
+        ArnLike = {
+          "aws:SourceArn" = "arn:aws:ecs:${var.aws_region}:${var.aws_account_number}:*"
+        }
+        StringEquals = {
+          "aws:SourceAccount" = var.aws_account_number
+        }
+      }
+    }]
+  })
+}
+resource "aws_iam_role_policy" "bloom_container" {
+  for_each = local.roles
+  name     = "bloom-${each.key}-container"
+  role     = aws_iam_role.bloom_container[each.key].id
+  policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [{
+      Action   = "*"
+      Effect   = "Deny"
+      Resource = "*"
+    }]
+  })
+}