Skip to content

Excessive sts:AssumeRoleWithWebIdentity calls from Jenkins controller IRSA role after enabling artifact-manager-s3 #788

@saheerb

Description

@saheerb

Jenkins and plugins versions report

Environment
OS: Linux - 6.1.159
Java: 17.0.12 - Red Hat, Inc. (OpenJDK 64-Bit Server VM)
---
PrioritySorter:5.1.0
ace-editor:1.1
analysis-model-api:12.1.0
antisamy-markup-formatter:162.v0e6ec0fcfcf6
apache-httpcomponents-client-4-api:4.5.14-208.v438351942757
apache-httpcomponents-client-5-api:5.2.1-1.0
artifact-manager-s3:871.v72f7f642a_245
asm-api:9.7-33.v4d23ef79fcc8
audit-trail:361.v82cde86c784e
authentication-tokens:1.119.v50285141b_7e1
aws-credentials:231.v08a_59f17d742
aws-global-configuration:130.v35b_7b_96f53c3
aws-java-sdk-ec2:1.12.767-467.vb_e93f0c614b_6
aws-java-sdk-minimal:1.12.767-467.vb_e93f0c614b_6
aws-java-sdk-secretsmanager:1.12.767-467.vb_e93f0c614b_6
aws-secrets-manager-credentials-provider:1.214.va_0a_d8268d068
aws-secrets-manager-secret-source:1.72.v61781b_35c542
badge:1.13
basic-branch-build-strategies:81.v05e333931c7d
blueocean:1.27.14
blueocean-bitbucket-pipeline:1.27.14
blueocean-commons:1.27.14
blueocean-config:1.27.14
blueocean-core-js:1.27.14
blueocean-dashboard:1.27.14
blueocean-display-url:2.4.3
blueocean-events:1.27.14
blueocean-git-pipeline:1.27.14
blueocean-github-pipeline:1.27.14
blueocean-i18n:1.27.14
blueocean-jwt:1.27.14
blueocean-personalization:1.27.14
blueocean-pipeline-api-impl:1.27.14
blueocean-pipeline-editor:1.27.14
blueocean-pipeline-scm-api:1.27.14
blueocean-rest:1.27.14
blueocean-rest-impl:1.27.14
blueocean-web:1.27.14
bootstrap5-api:5.3.3-1
bouncycastle-api:2.30.1.78.1-248.ve27176eb_46cb_
branch-api:2.1178.v969d9eb_c728e
build-name-setter:2.4.0
build-timeout:1.31
built-on-column:1.4
caffeine-api:3.1.8-133.v17b_1ff2e0599
checks-api:2.2.0
cloud-stats:336.v788e4055508b_
cloudbees-administrative-monitors:156
cloudbees-analytics:1.61
cloudbees-assurance:2.1141
cloudbees-bitbucket-branch-source:888.v8e6d479a_1730
cloudbees-blueocean-default-theme:0.8
cloudbees-casc-client:2.63
cloudbees-casc-items-api:2.68
cloudbees-casc-items-commons:2.68
cloudbees-casc-items-controller:2.68
cloudbees-casc-shared:1.6
cloudbees-disk-usage-simple:205.v47f4ee8803d1
cloudbees-folder:6.942.vb_43318a_156b_2
cloudbees-folders-plus:3.32
cloudbees-license:1265
cloudbees-platform-common:1.390
cloudbees-platform-data:753
cloudbees-plugin-usage:2.26
cloudbees-support:3.33
cloudbees-uc-data-api:4.59
cloudbees-unified-ui:1.808
command-launcher:115.vd8b_301cc15d0
commons-lang3-api:3.16.0-82.ve2b_07d659d95
commons-text-api:1.12.0-129.v99a_50df237f7
conditional-buildstep:1.4.3
config-file-provider:973.vb_a_80ecb_9a_4d0
configuration-as-code:1836.vccda_4a_122a_a_e
copyartifact:749.vfb_dca_a_9b_6549
coverage:1.16.1
credentials:1371.vfee6b_095f0a_3
credentials-binding:681.vf91669a_32e45
data-tables-api:2.1.4-1
display-url-api:2.204.vf6fddd8a_8b_e9
docker-commons:443.v921729d5611d
docker-java-api:3.3.6-90.ve7c5c7535ddd
docker-plugin:1.5
docker-slaves:1.0.7
durable-task:568.v8fb_5c57e8417
ec2:1688.v8c07e01d657f
echarts-api:5.5.0-1
eddsa-api:0.3.0-4.v84c6f0f4969e
email-ext:1814.v404722f34263
envinject:2.908.v66a_774b_31d93
envinject-api:1.199.v3ce31253ed13
extended-read-permission:53.v6499940139e5
favorite:2.221.v19ca_666b_62f5
flexible-publish:0.16.1
font-awesome-api:6.6.0-1
forensics-api:2.5.0
generic-webhook-trigger:1.88.2
gerrit-code-review:0.5.0
gerrit-trigger:3.1969.v65d614ec771a_
git:5.4.1
git-client:5.0.0
git-parameter:0.9.19
git-server:126.v0d945d8d2b_39
github:1.39.0
github-api:1.321-468.v6a_9f5f2d5a_7e
github-branch-source:1793.v1831e9c68d77
github-checks:554.vb_ee03a_000f65
github-oauth:597.ve0c3480fcb_d0
groovy-postbuild:228.vcdb_cf7265066
gson-api:2.11.0-41.v019fcf6125dc
handy-uri-templates-2-api:2.1.8-30.v7e777411b_148
htmlpublisher:1.36
icon-shim:3.0.0
infradna-backup:1038
instance-identity:185.v303dc7c645f9
ionicons-api:74.v93d5eb_813d5f
jackson2-api:2.17.0-379.v02de8ec9f64c
jakarta-activation-api:2.1.3-1
jakarta-mail-api:2.1.3-1
javadoc:280.v050b_5c849f69
javax-activation-api:1.2.0-7
javax-mail-api:1.6.2-10
jaxb:2.3.9-1
jdk-tool:80.v8a_dee33ed6f0
jenkins-design-language:1.27.14
jenkins-multijob-plugin:623.v03401733c9a_9
jjwt-api:0.11.5-112.ve82dfb_224b_a_d
jobConfigHistory:1229.v3039470161a_d
joda-time-api:2.12.7-29.v5a_b_e3a_82269a_
jquery:1.12.4-3
jquery3-api:3.7.1-2
jsch:0.2.16-86.v42e010d9484b_
json-api:20240303-41.v94e11e6de726
json-path-api:2.9.0-58.v62e3e85b_a_655
junit:1296.vb_f538b_c88630
kube-agent-management:803
kubernetes:4285.v50ed5f624918
kubernetes-client-api:6.10.0-240.v57880ce8b_0b_2
kubernetes-credentials:174.va_36e093562d9
ldap:725.v3cb_b_711b_1a_ef
leastload:3.0.0
lockable-resources:1218.va_3dd45e2b_fa_7
mailer:472.vf7c289a_4b_420
mapdb-api:1.0.9-40.v58107308b_7a_7
matrix-auth:3.2.2
matrix-project:832.va_66e270d2946
maven-plugin:3.23
metrics:4.2.21-451.vd51df8df52ec
mina-sshd-api-common:2.13.2-125.v200281b_61d59
mina-sshd-api-core:2.13.2-125.v200281b_61d59
mina-sshd-api-sftp:2.13.2-125.v200281b_61d59
monitoring:1.95.0
multiple-scms:0.8
nectar-license:8.44
nectar-rbac:1617
node-iterator-api:55.v3b_77d4032326
oauth-credentials:0.653.v14cf2088e950
okhttp-api:4.11.0-172.vda_da_1feeb_c6e
operations-center-agent:3.27163.27168
operations-center-client:3.27163.27168
operations-center-context:3.27163.27168
parameterized-scheduler:255.v73827fcdf618
parameterized-trigger:806.vf6fff3e28c3e
pipeline-build-step:540.vb_e8849e1a_b_d8
pipeline-github:2.8-159.09e4403bc62f
pipeline-github-lib:61.v629f2cc41d83
pipeline-githubnotify-step:49.vf37bf92d2bc8
pipeline-graph-analysis:216.vfd8b_ece330ca_
pipeline-graph-view:340.v28cecee8b_25f
pipeline-groovy-lib:730.ve57b_34648c63
pipeline-input-step:495.ve9c153f6067b_
pipeline-milestone-step:119.vdfdc43fc3b_9a_
pipeline-model-api:2.2205.vc9522a_9d5711
pipeline-model-definition:2.2205.vc9522a_9d5711
pipeline-model-extensions:2.2205.vc9522a_9d5711
pipeline-rest-api:2.34
pipeline-stage-step:312.v8cd10304c27a_
pipeline-stage-tags-metadata:2.2205.vc9522a_9d5711
pipeline-utility-steps:2.16.0
plain-credentials:183.va_de8f1dd5a_2b_
plugin-util-api:4.1.0
popper2-api:2.11.6-5
postbuildscript:3.2.0-550.v88192b_d3e922
prism-api:1.29.0-16
prometheus:795.v995762102f28
pubsub-light:1.18
rebuild:332.va_1ee476d8f6d
resource-disposer:0.23
role-strategy:743.v142ea_b_d5f1d3
run-condition:1.7
s3:483.vcb_db_3dcee68f
scm-api:696.v778d637b_a_762
script-security:1354.va_70a_fe478c7f
slack:684.v833089650554
snakeyaml-api:2.2-121.v5a_68b_9300b_d4
sse-gateway:1.27
ssh-agent:376.v8933585c69d3
ssh-credentials:343.v884f71d78167
ssh-slaves:2.973.v0fa_8c0dea_f9f
sshd:3.330.vc866a_8389b_58
structs:338.v848422169819
support-core:1475.va_b_510dc46b_3d
throttle-concurrents:2.14
timestamper:1.27
token-macro:400.v35420b_922dcb_
trilead-api:2.147.vb_73cc728a_32e
unique-id:2.107.v3fa_e48679298
user-activity-monitoring:2.421
variant:60.v7290fc0eb_b_cd
warnings-ng:11.3.0
windows-cloud:1.0.1
workflow-aggregator:600.vb_57cdd26fdd7
workflow-api:1336.vee415d95c521
workflow-basic-steps:1058.vcb_fc1e3a_21a_9
workflow-cps:3943.v3519a_3260660
workflow-cps-global-lib-http:2.54.0
workflow-durable-task-step:1364.v2fd76fb_6fd41
workflow-job:1436.vfa_244484591f
workflow-multibranch:795.ve0cb_1f45ca_9a_
workflow-scm-step:427.v4ca_6512e7df1
workflow-step-api:678.v3ee58b_469476
workflow-support:920.v59f71ce16f04
ws-cleanup:0.45
yet-another-docker-plugin:0.2.0

What Operating System are you using (both controller, and any agents involved in the problem)?

Linux

Reproduction steps

  1. Deploy Jenkins controller on EKS with a service account annotated for IRSA.
  2. Attach an IAM role to the controller service account with permissions for:
  • S3 artifact bucket access
  • KMS if using SSE-KMS
  1. Enable CloudTrail management-event logging for the account.
  2. Run a baseline Jenkins workload without artifact-manager-s3.
  3. Use a pipeline that archives artifacts repeatedly, for example many short builds each archiving multiple small files.
  4. Record CloudTrail counts for:
  • sts.amazonaws.com / AssumeRoleWithWebIdentity
  • grouped by the Jenkins controller role ARN
  1. Install and configure artifact-manager-s3 on the same Jenkins controller.
  2. Configure the plugin to use the controller’s IAM/IRSA-based credentials and the same S3 bucket.
  3. Re-run the same Jenkins workload with the same build volume and artifact pattern.
  4. Compare CloudTrail counts again for:
  • sts.amazonaws.com / AssumeRoleWithWebIdentity
  • grouped by the Jenkins controller role ARN

Expected Results

The plugin should reuse AWS credentials efficiently enough that enabling S3 artifact storage does not cause a dramatic increase in AssumeRoleWithWebIdentity calls from the Jenkins controller IRSA role under the same workload.

Actual Results

After enabling artifact-manager-s3, the Jenkins controller IRSA role shows a very large increase in AssumeRoleWithWebIdentity events in CloudTrail during artifact operations, suggesting excessive credential reacquisition or AWS client/session churn on the controller path.

Anything else?

No response

Are you interested in contributing a fix?

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No fields configured for Bug.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions