aws-tf/parallelcluster/aws//modules/pcluster_api: ParallelClusterLambdaRole is not authorized to perform: iam:AttachRolePolicy on resource RoleHeadNode - AccessDenied

[abazzi-neogenomics]

Hello there, I'm trying to create clusters with additional iam custom policies attached.
I'm using tf 1.6.6, aws-tf/aws-parallelcluster 1.1.0 and pcluster api 3.8.0 (also tried 3.11.1, same behaviour).

For the API I'm using aws-tf/parallelcluster/aws//modules/pcluster_api 1.1.0

module "pcluster_api" {
  source  = "aws-tf/parallelcluster/aws//modules/pcluster_api"
  version = "1.1.0"

  region         = var.region
  api_stack_name = "${local.resource_name_prefix}-${var.name}-API-${replace(var.pcluster_api_version, ".", "-")}-${random_id.suffix.hex}"
  api_version    = var.pcluster_api_version

  parameters = {
    EnableIamAdminAccess   = "true"
    CreateApiUserRole      = "false"
    IAMRoleAndPolicyPrefix = "${local.resource_name_prefix}-"
  }
}

Upon applying the API configuration, the lambda roles get created successfully, however when I create the clusters I get the following error:

Resource handler returned message: "User: arn:aws:sts::xxx:assumed-role/ParallelClusterLambdaRole-eaa5bf80/ParallelClusterTF-API-3-8--ParallelClusterFunction-VcLkj2hKwIOu is not authorized to perform: iam:AttachRolePolicy on resource: role test-tfpcluster3-slurm01-RoleHeadNode-kIxMITQF7ugm because no identity-based policy allows the iam:AttachRolePolicy action (Service: Iam, Status Code: 403, Request ID: 3cdde460-b7e2-46c5-ac99-a5aee19cc300) (SDK Attempt Count: 1)" (RequestToken: 4c788761-5c87-b34d-7660-37a85d54341e, HandlerErrorCode: AccessDenied)

This is the provider i'm using the the pcluster cluster creation

provider "aws-parallelcluster" {
  region = var.region

  endpoint = module.api-3-8-0.stack_outputs.ParallelClusterApiInvokeUrl # ignore the ref, I'm wrapping my own module around aws-tf/parallelcluster/aws//modules/pcluster_api
  role_arn = var.assume_role_arn # this is admin in the account
}

Upon closer inspection I noticed that aws-tf/parallelcluster/aws//modules/pcluster_api correctly creates the lambda admin role, but only allows attach for policies named parallelcluster*, SSM managed istance and a few others.
My workaround so far is naming my custom policy parallelcluster--something and it works.

Am I missing something? Is there a better way to pass inputs to the api module or is it a feature request?
I have naming convetions in these accounts for prefix, so all resources have the same prefixes, and having to add 'parallelcluster' for this makes it a bit messy

tagging @gmarciani since he's developed the module :)

Thank you!

[gmarciani]

Hi @abazzi-neogenomics ,

the failure you're observing is expected. By default to guarantee a better security posture, the admin permissions created by ParallelCluster only allow to attach/detach policies with the parallelcluster prefix.
See https://docs.aws.amazon.com/parallelcluster/latest/ug/iam-roles-in-parallelcluster-v3.html#iam-roles-in-parallelcluster-v3-user-policy-manage-iam for further details about it.

In ParallelCluster 3.12.0 we added a new parameter to the ParallelCluster API to overcome this limitation.

To unblock you need to:

1. Create a policy with permissions to attach/detach policies having your prefix.
2. Use the Terraform module to deploy ParallelCluster 3.12.0+ with the parameter ParallelClusterFunctionAdditionalPolicies set to the ARN of that policy.

I hope this helps.
Let me know if further assistance is required.