Skip to content

Expiring assumed role token after 6 hours when using FargateCluster with max duration set to 12 hours #422

Open
@JimHBeam

Description

@JimHBeam

Describe the issue:
I was advised on SO to post here, not sure its a bug, I may just not know something...

Im running dask locally and spinning up fargate cluster in AWS.
Its assuming a role correctly and starting up the cluster.
After six hours its permissions fail, I have set the maximum session duration on the role to 12 hours but it seems to be getting a defailt 6 hour token anyway and not asking for a 12 hour token.

Is there a way to instruct it to get a 12 hour token or do I need to refresh the session token somehow?

Minimal Complete Verifiable Example:
Cant post a full Minimal Complete Verifiable Example, would need something that goes over 6 hrs.
But our cluster is setup like:

cluster = FargateCluster(
        cloudwatch_logs_group=f"{project}_log_group",
        cluster_name_template=f"{project}-{uuid}",
        environment={"DEFAULT_GSF_VERSION": gsf_version},
        image=f"{aws_account_id}.dkr.ecr.eu-west-2.amazonaws.com/{project}:latest",
        n_workers=9,
        scheduler_cpu=4096,
        scheduler_mem=30720,
        scheduler_timeout="240 minutes",
        security_groups=[aws_security_group_id],
        task_role_arn=f"arn:aws:iam::{aws_account_id}:role/{project}ECSTaskRole",
        worker_cpu=4096,
        worker_mem=30720,
        worker_extra_args=algorithm_extra_cluster_args,
        fargate_use_private_ip=use_private_ip,
        vpc=vpc,
        subnets=subnets,
        skip_cleanup=True,
        platform_version="LATEST",
        shutdown_on_close=True,
        mount_points=mount_points,
        volumes=volumes,
    )
    cluster.adapt(minimum=9, maximum=80)
    logger.info(f"dashboard_link is: {cluster.dashboard_link}")

Anything else we need to know?:

I have set the maximum session duration on my role to 43200(12 hours), seems it still only gets a 6 hour token.
I tried adding duration_seconds=43200 to AWS config files hoping to override the default 6 hour token, still lost perms after 6 hours

Environment:

  • Dask version:022.12.0
  • Python version:3.8.3
  • Operating System:linux
  • Install method (conda, pip, source):poetry

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions