AWS: Having the provider & state depend on the output of a previous module (for assume_role)

[ckeenan34]

I started out writing a question and ended up answering it for myself, but still wanted to share incase others find it helpful or have handle this in different ways

I have an illustrative setup showing what I'd like to do

Directory structure:

iam/
    terragrunt.hcl
vpc/
    terragurnt.hcl
root.hcl

Where iam/terragrunt.hcl uses a module that outputs a role with key "infra_admin_role_arn"

vpc/terragrunt.hcl depends on iam:

...
dependency "iam" {
   config_path = "../iam"
}
...

I want root.hcl to set up the state & provider configuration, such that when I'm running the iam module it will not assume any role, which means I rely on the default provider chain to determine the aws account that will create the infra admin role that I want. This would mean I'm running as a user with the correct iam permissions.

However, after iam is run, I'd like to have the vpc module use the newly create role for permissions. The primary reason for this is security (separation of concerns), I want to create all my iam roles first (with an existing role that only has iam:*), then I can be sure my downstream modules will work without any additional iam permissions.

I've been able to get this to work with the following root.hcl:

generate "provider" {
  path      = "provider.tf"
  if_exists = "overwrite_terragrunt"
  contents  = <<EOF
provider "aws" {
  region = "${local.aws_region}"
  allowed_account_ids = ["${local.account_id}"]

  %{ if try(dependency.iam.outputs.infra_admin_role_arn, "") != "" }
    assume_role {
      role_arn = "${dependency.iam.outputs.infra_admin_role_arn}"
    }
  %{ endif }
}
EOF
}

remote_state {
  backend = "s3"
  generate = {
    path      = "backend.tf"
    if_exists = "overwrite_terragrunt"
  }
  config = {
    bucket         = "..."
    key            = "..."
    region         = local.aws_region
    encrypt        = true
    use_lockfile   = true
    assume_role = try(dependency.iam.outputs.infra_admin_role_arn, "") != "" ? {
      role_arn = dependency.iam.outputs.infra_admin_role_arn
    } : null
  }
}

What this will do is assume an iam role if the unit depends on the iam unit, otherwise it will attempt to use the default provider chain.

Initially I thought this was producing a problem where if I tried to run terragrunt apply --all in one shot that it would attempt to plan both in parallel and the vpc would fail (since it can't assume the role to init). After messing with this more, this actually works! There is one small change where I needed to add this errors block to the root.hcl:

errors {
  # Retry if role assumption fails due to iam role creation being eventually consistent
  retry "retry_iam_not_ready" {
      retryable_errors = [".*Error: Cannot assume IAM Role.*"] 
      max_attempts = 3
      sleep_interval_sec = 20
  }
}

Otherwise downstream modules may attempt to assume the role before it exists, but a simple retry fixes it (although not certain if 60 sec total is enough)

The only issue with this approach is running terragrunt plan --all. This makes sense because I can't generate a plan for vpc until the iam module is applied. My workaround this once I add this to my CI/CD is to have 2 manual approval steps, (ie: plan iam -> review & approve -> apply iam -> plan vpc -> review & approve -> apply vpc). The typical workaround to this problem would be using mocked outputs, but that doesn't apply here since the role doesn't just get passed to the module, it gets used during the plan.

Curious if anyone else does something similar