S3 remote_state with assume_role_with_web_identity fails

[Tedlar]

Describe the bug

S3 remote_state block with assume_role_with_web_identity configured is failing while it's generate counterpart is working file.

Reproducing bugs

We are currently trying to switch from IRSA authentication to OIDC.
I modified remote_state block following the documentation but terragrunt is failing to authenticate.

Steps To Reproduce

Failing configuration

remote_state {
  backend = "s3"
  generate = {
    path      = "backend.tf"
    if_exists = "overwrite"
  }
  config = {
    bucket         = "xxx"
    key            = "xxx"
    region         = "xxx"
    encrypt        = true
    use_lockfile   = true

    assume_role_with_web_identity = {
      role_arn                = "arn:aws:iam::${local.aws_account_id}:role/terragrunt"
      web_identity_token_file = "/tmp/web-identity-token"
    }
  }
}

Error

ERROR  error checking access to S3 bucket xxx: Forbidden: Forbidden
	status code: 403

Working configuration

generate "backend" {
  path      = "backend.tf"
  if_exists = "overwrite"
  contents  = <<-EOF
    terraform {
      backend "s3" {
        bucket = "xxx"
        key    = "xxx"
        region = "xxx"
        encrypt        = true
        use_lockfile   = true

        assume_role_with_web_identity = {
          role_arn                = "arn:aws:iam::${local.aws_account_id}:role/terragrunt"
          web_identity_token_file = "/tmp/web-identity-token"
        }
      }
    }
  EOF
}

Also IRSA configuration for remote_state is working fine

remote_state {
  backend = "s3"
  generate = {
    path      = "backend.tf"
    if_exists = "overwrite"
  }
  config = {
    bucket         = "xxx"
    key            = "xxx"
    region         = "xxx"
    encrypt        = true
    use_lockfile   = true

    assume_role = {
      role_arn = "arn:aws:iam::${local.aws_account_id}:role/terragrunt"
    }
  }
}

Expected behavior

remote_state block should work same way as it's generate counterpart.

Versions

• Terragrunt version: 0.84.1
• OpenTofu/Terraform version: 1.10.0

Additional context

Was this issue solved in newer versions of terragrunt?
We did not update terragrunt version yet because our automation is using arguments removed in 0.85.0

[Tedlar]

I jumped into source code of terragrunt and started debugging like crazy.
What I found out is that S3 remote_state while using assume_role_with_web_identity is failing on function NeedsBootstrap.
What's interesting is that same block but with assume_role pointing to exactly same AWS IAM Role is working fine.

After setting up environment variable TG_DISABLE_BUCKET_UPDATE to true terragrunt started working correctly.

I think there may be issue with implementation of NeedsBootstrap logic. It does not use OIDC credentials correctly.

[platform-skyfed]

#5105

I think this might be the same issue I'm facing, trying to assume the role via ENV vars throws a 403 if going cross account. If the role is in the same account it's working correctly but might just default back to the Access keys without an error.

Good find on the issue!

[hp197]

We faced the same issue with the regular assume_role (not web identity).
The env variable didn't fix it (the env var in combination with run --all didn't fix it), what fixed it for us is moving from run --all plan to just plain plan (w/o the run all).

What it looks like to us, is that the terragrunt dependency state bootstrap doesn't assume role properly, as we provide our automation a role with only a permission to assume another role. Lets say github runner has RoleA and we assume RoleB.

* operation error S3: GetObject, https response error StatusCode: 403, RequestID: XYZ, HostID: x/xxx/xxx=, api error AccessDenied: User: arn:aws:sts::1234567890:assumed-role/RoleA/githubplan is not authorized to perform: s3:GetObject on resource: "arn:aws:s3:::bucket/infrastructure/cloud/01-state" because no identity-based policy allows the s3:GetObject action

So it tries to fetch our states from the bucket with RoleA, previously (and with self generated .tf config) it first assumes role to RoleB and then fetches the states from the bucket.

[nijave]

I think this is caused by #4834