feat: bloom-dev open tofu config (bloom-housing#5580)

Author: Avritt Rohwer

docker-compose.yml

@@ -107,10 +107,10 @@ services:
       DATABASE_URL: "postgres://postgres:example@db:5432/bloom_prisma"
     healthcheck:
       test: ["CMD", "curl", "--fail", "http://127.0.0.1:3100/"]
-      interval: "2s"
-      timeout: "1s"
-      retries: 20
-      start_period: "1s"
+      interval: "5s"
+      timeout: "2s"
+      retries: 10
+      start_period: "5s"
     depends_on:
       db:
         condition: service_healthy

infra/README.md

@@ -10,47 +10,53 @@ Cloud Native Computing Foundation.
   is a set of resources that are all managed together. Each root module has a state file that
   records the results of the latest apply operation.
 
+   - [bloom_dev](./tofu_root_modules/bloom_dev/README.md): Configures the bloom-dev AWS account.
    - [bloom_dev_deployer_permission_set](./tofu_root_modules/bloom_dev_deployer_permission_set/README.md):
      Configures the bloom-dev-deployer permission set that is assigned on the bloom-dev account.
 
+- [tofu_importable_modules](./tofu_importable_modules): Contains all the Open Tofu importable
+  modules. An importable module is a reusable set of resources configured through input
+  parameters. Root modules import importable modules.
+
+   - [bloom_deployment](./tofu_importable_modules/bloom_deployment/): Configures all the resources
+     needed for a Bloom deployment in a single AWS account.
+
+
+
 ## Infrastructure-as-code mental model
 
 Let's say that you need to deploy Bloom to an AWS account. A straight-forward way of achieving this
 would be to log into the AWS web console and create all the required resources. The downside of this
-approach is that unless you take really good notes, it is a difficult process to replicate. Even if
-you take really good notes, the process might have a lot of steps which are all opportunities for
-making mistakes. Additionally, it is not possible to automate such a process (well, maybe if you
-have one of those neat AIs that can control your browser. But the AI could also make mistakes just
-like a human).
+approach is that unless you take really good notes, it is a difficult process to replicate.
 
 Another approach could be to write a bash script that calls a bunch of AWS CLI commands that create
 the resources. This improves on the web-based approach because all the steps are explicitly written
 down. However, the script only works on a fresh AWS account - if you run it again there will be
 a bunch of errors because the resources will have already been created. If you need to change how
-the account is configured, you need to write more scripts. That reminds me too much of database
-evolutions to seem like an good idea...
-
-Enter infrastructure-as-code tools like Open Tofu. I like to think of them as CLI scripting with
-a bunch of functionality already built in. We have `.tf` files that contain [resource
-descriptions](https://opentofu.org/docs/language/resources/) for everything we want to configure. We
-can run the 'script' by running [`tofu apply`](https://opentofu.org/docs/cli/commands/apply/). If
-the AWS account already matches our desired configuration, Tofu gives us a nice message that the
-infrastructure matches the configuration. Otherwise, Tofu presents us with a list of planned changes
-it thinks it needs to make and asks if it should go forward with the plan.
-
-These tools are not magic, however. It is still possible to misconfigure resources and get errors
-from the AWS API. These cases are not always handled gracefully and sometimes require deleting or
-configuring things manually to unbork the tool. Using a infrastructure-as-code still requires manual
-testing and knowledge of the underlying systems you are configuring. It is a heck of a lot better
-than shell scripting, however, at least in my experience :)
-
-_Monologue by Avritt Rohwer_
+the account is configured, you need to write more scripts.
+
+Enter infrastructure-as-code tools like Terraform and Open Tofu. I like to think of them as CLI
+scripting with a bunch of functionality already built in. `.tf` files contain [resource
+descriptions](https://opentofu.org/docs/language/resources/). Run the 'script' by running [`tofu
+apply`](https://opentofu.org/docs/cli/commands/apply/). If the AWS account already matches the
+desired configuration, Tofu prints a nice message that the infrastructure matches the
+configuration. Otherwise, Tofu presents a list of planned changes it wants to make and asks if it
+should go forward with the plan.
+
+It is still possible to misconfigure resources and get errors from the AWS API. These cases are not
+always handled gracefully and sometimes require deleting or configuring things manually to unblock
+the tool. Using a infrastructure-as-code still requires manual testing and knowledge of the
+underlying systems you are configuring. It is a heck of a lot better than shell scripting, however,
+at least in my experience :)
 
 ## Developer setup
 
-1. Install required tools:
-   1. Open Tofu: https://opentofu.org/docs/intro/install/
-   2. AWS CLI: https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html
+1. Install required CLI tools:
+   1. bash: `which bash`
+   2. openssl: `which openssl`
+   3. tr: `which tr`
+   4. Open Tofu: https://opentofu.org/docs/intro/install/
+   5. AWS CLI: https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html
 
       After installing, edit your `~/.aws/config` file for SSO authentication:
 
@@ -94,34 +100,41 @@ tofu init
 ```
 
 Once the provider dependences have been downloaded, you will not have to run `tofu init` again
-unless you add a provider. In that case, run:
+unless you add a provider.
 
-```bash
-tofu init
-```
-
-To update a required version for a provider, change the version in the relevant `main.tf` file then
-run:
+To update a required version for a provider, change the version then run:
 
 ```bash
 tofu init -upgrade
 ```
 
-Both of these command will download the new dependencies and update the `.terraform.lock.hcl` file in the root module
-directory.
+Both of these commands will download the new dependencies and update the `.terraform.lock.hcl` file
+in the root module directory.
 
 ### Applying changes
 
 1. Open a shell and change directory to the root module.
-2. Run `aws sso login` to authenticate to AWS. After 1 hour, you will need to re-authenticate using
-   the same command.
-3. Edit the `main.tf` file to update the desired configuration.
+2. Run `aws sso login --profile bloom-dev-deployer` to authenticate to AWS.
+3. Edit the `.tf` files.
 4. Run `tofu apply` and review the planned changes. If there are unexpected planned changes, go back
-   to step 1. If all the changes are expected, approve the apply.
+   to step 3. If all the changes are expected, approve the apply.
 5. Inspect the relevant AWS resources via the CLI or the AWS web console
    (Log in via https://d-9067ac8222.awsapps.com/start). If there are unexpected results, go back to
-   step 1. In some cases you may have to manually modify or delete resources directly to 'unstick'
+   step 3. In some cases you may have to manually modify or delete resources directly to 'unstick'
    Open Tofu.
+6. To delete only the resources provisioned by the bloom_deployment module, run `tofu destroy
+   -target=module.bloom_deployment`.
+
+#### Forcing resource recreation
+
+To force Tofu to replace a resource, run `tofu apply -replace=ADDRESS`. For example:
+
+```
+tofu apply -replace=module.bloom_deployment.aws_secretsmanager_secret.api_jwt_signing_key
+```
+
+This is helpful when testing the local-exec provisioner because the provisioner only runs on
+resource creation.
 
 ## AWS setup done manually
 

infra/tofu_importable_modules/bloom_deployment/db.tf

@@ -0,0 +1,42 @@
+# Create a database.
+resource "aws_db_subnet_group" "bloom" {
+  region     = var.aws_region
+  name       = "bloom"
+  subnet_ids = [for s in aws_subnet.private : s.id]
+}
+resource "aws_db_instance" "bloom" {
+  identifier                      = "bloom"
+  deletion_protection             = local.is_prod
+  engine                          = "postgres"
+  engine_version                  = "17"
+  instance_class                  = local.database_config.instance_class
+  multi_az                        = var.high_availability
+  enabled_cloudwatch_logs_exports = ["postgresql", "upgrade", "iam-db-auth-error"]
+  username                        = "master"
+  manage_master_user_password     = true
+
+  # Monitoring
+  performance_insights_enabled          = "true"
+  performance_insights_retention_period = 7 # minimum
+  database_insights_mode                = "standard"
+
+  # Networking
+  vpc_security_group_ids              = [aws_security_group.db.id]
+  iam_database_authentication_enabled = true
+  db_subnet_group_name                = aws_db_subnet_group.bloom.id
+
+  # Updates
+  apply_immediately           = true # If false, any changes are applied in the next maintenance window instead of when tofu apply runs.
+  engine_lifecycle_support    = "open-source-rds-extended-support"
+  allow_major_version_upgrade = false
+  auto_minor_version_upgrade  = true
+
+  # Storage
+  storage_encrypted         = true
+  storage_type              = "gp2"
+  allocated_storage         = local.database_config.starting_storage_gb
+  max_allocated_storage     = local.database_config.max_storage_gb
+  backup_retention_period   = local.database_config.backup_retention_days
+  final_snapshot_identifier = "bloom-db-finalsnapshot"
+  skip_final_snapshot       = !local.is_prod
+}

infra/tofu_importable_modules/bloom_deployment/ecs.tf

@@ -0,0 +1,189 @@
+# Create an ECS cluster and services for each Bloom binary.
+resource "aws_iam_service_linked_role" "ecs" {
+  aws_service_name = "ecs.amazonaws.com"
+}
+resource "aws_ecs_cluster" "bloom" {
+  region     = var.aws_region
+  name       = "bloom"
+  depends_on = [aws_iam_service_linked_role.ecs]
+  setting {
+    name  = "containerInsights"
+    value = "enhanced"
+  }
+}
+
+# Set up service discovery for the sites to talk to the api.
+resource "aws_service_discovery_http_namespace" "bloom" {
+  region      = var.aws_region
+  name        = "bloom"
+  description = "Service namespace the bloom services use."
+}
+
+# Create logs groups.
+resource "aws_cloudwatch_log_group" "task_logs" {
+  for_each = toset([
+    "bloom-api",
+    "bloom-site-partners",
+    "bloom-site-public"
+  ])
+  region            = var.aws_region
+  name              = each.value
+  log_group_class   = "STANDARD"
+  retention_in_days = local.ecs_logs_retention_days
+}
+
+# Create a secret key used by the API to sign JWTs.
+resource "aws_secretsmanager_secret" "api_jwt_signing_key" {
+  region                  = var.aws_region
+  description             = "Key used by the Bloom API to sign JWTs"
+  name_prefix             = "bloom-api-jwt-signing-key" # avoids 'you can't create this secret because a secret with this name is already scheduled for deletion' issue when re-deploying an account.
+  recovery_window_in_days = 7                           # minimum
+
+  # TODO: use an ephemeral resource instead of local-exec:
+  # https://github.com/bloom-housing/bloom/issues/5637.
+  #
+  # The provisioner block runs after the resource has been created, and never again.
+  provisioner "local-exec" {
+    interpreter = ["/usr/bin/env", "bash", "-c"]
+    # We need to be very careful that any errors result in a non-zero exit code. Otherwise tofu will
+    # think this block succeeded and not error.
+    command = <<-EOT
+    if ! type -P aws &>/dev/null; then
+      echo 'ERROR: aws required'
+      exit 1
+    fi
+    if ! type -P openssl &>/dev/null; then
+      echo 'ERROR: openssl required'
+      exit 1
+    fi
+    if ! type -P tr &>/dev/null; then
+      echo 'ERROR: tr required'
+      exit 1
+    fi
+
+    if ! s=$(openssl rand -base64 256 | tr -d '\n'); then
+        echo 'ERROR: failed to generate random value'
+        exit 1
+    fi
+
+    if ! aws secretsmanager put-secret-value \
+         --profile ${var.aws_profile} \
+         --region ${var.aws_region} \
+         --secret-id ${self.id} \
+         --secret-string "$s"
+    then
+        echo 'ERROR: failed to put secret value'
+        exit 1
+    fi
+    EOT
+  }
+}
+
+locals {
+  roles = {
+    "api" = {
+      task_execution_policy_extra_statements = [
+        {
+          Action   = "secretsmanager:GetSecretValue"
+          Effect   = "Allow"
+          Resource = aws_db_instance.bloom.master_user_secret[0].secret_arn
+        },
+        {
+          Action   = "secretsmanager:GetSecretValue"
+          Effect   = "Allow"
+          Resource = aws_secretsmanager_secret.api_jwt_signing_key.arn
+        }
+      ]
+    }
+    "site-partners" = {
+      task_execution_policy_extra_statements = []
+    }
+    "site-public" = {
+      task_execution_policy_extra_statements = []
+    }
+  }
+}
+
+# Create roles for the ECS task executor and the tasks.
+resource "aws_iam_role" "bloom_ecs" {
+  for_each    = local.roles
+  name        = "bloom-${each.key}-ecs"
+  description = "Role the ECS service uses when launching Bloom ${each.key} tasks."
+  # https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html#create_task_iam_policy_and_role
+  assume_role_policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [{
+      Action = "sts:AssumeRole"
+      Effect = "Allow"
+      Principal = {
+        Service = "ecs-tasks.amazonaws.com"
+      }
+      Condition = {
+        ArnLike = {
+          "aws:SourceArn" = "arn:aws:ecs:${var.aws_region}:${var.aws_account_number}:*"
+        }
+        StringEquals = {
+          "aws:SourceAccount" = var.aws_account_number
+        }
+      }
+    }]
+  })
+}
+resource "aws_iam_role_policy" "bloom_ecs" {
+  for_each = local.roles
+  name     = "bloom-${each.key}-ecs"
+  role     = aws_iam_role.bloom_ecs[each.key].id
+  policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = concat(
+      [
+        {
+          Action = [
+            "logs:CreateLogStream",
+            "logs:PutLogEvents",
+          ]
+          Effect   = "Allow"
+          Resource = "${aws_cloudwatch_log_group.task_logs["bloom-${each.key}"].arn}:log-stream:*"
+        },
+      ],
+      each.value.task_execution_policy_extra_statements
+    )
+  })
+}
+resource "aws_iam_role" "bloom_container" {
+  for_each    = local.roles
+  name        = "bloom-${each.key}-container"
+  description = "Role the Bloom ${each.key} container runs as."
+  # https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html#create_task_iam_policy_and_role
+  assume_role_policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [{
+      Action = "sts:AssumeRole"
+      Effect = "Allow"
+      Principal = {
+        Service = "ecs-tasks.amazonaws.com"
+      }
+      Condition = {
+        ArnLike = {
+          "aws:SourceArn" = "arn:aws:ecs:${var.aws_region}:${var.aws_account_number}:*"
+        }
+        StringEquals = {
+          "aws:SourceAccount" = var.aws_account_number
+        }
+      }
+    }]
+  })
+}
+resource "aws_iam_role_policy" "bloom_container" {
+  for_each = local.roles
+  name     = "bloom-${each.key}-container"
+  role     = aws_iam_role.bloom_container[each.key].id
+  policy = jsonencode({
+    Version = "2012-10-17"
+    Statement = [{
+      Action   = "*"
+      Effect   = "Deny"
+      Resource = "*"
+    }]
+  })
+}