Pipeline succeeded but some steps are still running

[guolunwei]

Jenkins and plugins versions report

Environment

Jenkins: 2.555.1
OS: Linux - 3.10.0-1160.el7.x86_64
Java: 25.0.1 - Oracle Corporation (Java HotSpot(TM) 64-Bit Server VM)
---
ansicolor:536.v13fa_b_860c267
antisamy-markup-formatter:173.v680e3a_b_69ff3
apache-httpcomponents-client-4-api:4.5.14-269.vfa_2321039a_83
asm-api:9.9.1-189.vb_5ef2964da_91
authentication-tokens:1.144.v5ff4a_5ec5c33
authorize-project:2.0.0
bootstrap5-api:5.3.8-1024.v127320880c60
bouncycastle-api:2.30.1.84-291.v9f17b_21896e2
branch-api:2.1280.v0d4e5b_b_460ef
build-timeout:1.40
caffeine-api:3.2.3-194.v31a_b_f7a_b_5a_81
checks-api:402.vca_263b_f200e3
cloudbees-folder:6.1100.ve9eed61d16c4
commons-compress-api:1.28.0-3
commons-lang3-api:3.20.0-109.ve43756e2d2b_4
commons-text-api:1.15.0-218.va_61573470393
config-file-provider:1006.vc7366c201f57
credentials:1502.v5c95e620ddfe
credentials-binding:719.v80e905ef14eb_
data-tables-api:2.3.7-1558.vf82fa_133ce1c
display-url-api:2.217.va_6b_de84cc74b_
docker-commons:472.vee120e23d3a_c
docker-workflow:634.vedc7242b_eda_7
durable-task:664.v2b_e7a_dfff66c
echarts-api:6.0.0-1281.vd3d21a_1ca_cb_4
eddsa-api:0.3.0.1-29.v67e9a_1c969b_b_
font-awesome-api:7.2.0-983.v3f63c34eddb_9
git:5.10.1
git-client:6.6.0
git-parameter:462.vdcf3df2ed2ca_
gitlab-plugin:1.9.13
gson-api:2.14.0-201.v8eefe5515533
instance-identity:203.v15e81a_1b_7a_38
ionicons-api:94.vcc3065403257
jackson-annotations2-api:2.21-7.v4777a_f3a_a_d47
jackson2-api:2.21.2-436.v29efdb_7418ff
jackson3-api:3.1.2-73.v3e5485d8b_148
jakarta-activation-api:2.1.4-1
jakarta-mail-api:2.1.5-1
jakarta-xml-bind-api:4.0.6-12.vb_1833c1231d3
javax-activation-api:1.2.0-8
jaxb:2.3.9-143.v5979df3304e6
jersey3-api:3.1.11-4.v77818819c2e1
job-dsl:3654.vdf58f53e2d15
joda-time-api:2.14.1-187.vdf2def02b_8a_1
jquery3-api:3.7.1-682.vfa_cdce169929
jsch:0.2.16-95.v3eecb_55fa_b_78
json-api:20251224-185.v0cc18490c62c
json-path-api:3.0.0-218.vcd4dd1355de2
junit:1403.vd9d1413fd205
lark-notice:2.1.7
lockable-resources:1509.va_6b_5b_5cb_0b_40
mailer:534.v1b_36f5864073
mapdb-api:1.0.9-44.va_1e1310c9118
matrix-project:870.v9db_fcfc2f45b_
metrics:4.2.37-494.v06f9a_939d33a_
mina-sshd-api-common:2.16.0-167.va_269f38cc024
mina-sshd-api-core:2.16.0-167.va_269f38cc024
nodejs:1.6.6
pipeline-build-step:584.vdb_a_2cc3a_d07a_
pipeline-graph-analysis:254.v0f63a_a_447dca_
pipeline-graph-view:867.vece2ec789435
pipeline-groovy-lib:797.v90ea_a_9b_e45a_0
pipeline-input-step:551.vdff487c5998c
pipeline-milestone-step:152.v6e22b_8cfc66c
pipeline-model-api:2.2277.v00573e73ddf1
pipeline-model-definition:2.2277.v00573e73ddf1
pipeline-model-extensions:2.2277.v00573e73ddf1
pipeline-rest-api:2.41
pipeline-stage-step:345.va_96187909426
pipeline-stage-tags-metadata:2.2277.v00573e73ddf1
pipeline-stage-view:2.41
pipeline-utility-steps:2.20.0
plain-credentials:199.v9f8e1f741799
plugin-util-api:7.1330.v47b_46ee2047a_
prism-api:1.30.0-723.v97277866cece
publish-over:238.v1db_583004c9b_
publish-over-ssh:390.vb_f56e7405751
resource-disposer:0.25
scm-api:728.vc30dcf7a_0df5
script-security:1399.ve6a_66547f6e1
snakeyaml-api:2.5-149.v72471e9c6371
snakeyaml-engine-api:3.0.1-5.vd98ea_ff3b_92e
ssh-agent:396.vcc7d84e622ec
ssh-credentials:372.va_250881b_08cd
ssh-slaves:3.1097.v868116049892
structs:362.va_b_695ef4fdf9
timestamper:1.30
token-macro:477.vd4f0dc3cb_cf1
trilead-api:2.284.v1974ea_324382
uno-choice:2.8.9
variant:70.va_d9f17f859e0
woodstox-core-api:7.1.1-1.v4d297985f397
workflow-aggregator:608.v67378e9d3db_1
workflow-api:1413.v2ff1a_5e720fa_
workflow-basic-steps:1098.v808b_fd7f8cf4
workflow-cps:4303.v8fa_2a_581f0a_6
workflow-durable-task-step:1475.ved562f6ec8b_3
workflow-job:1571.vb_423c255d6d9
workflow-multibranch:821.vc3b_4ea_780798
workflow-scm-step:466.va_d69e602552b_
workflow-step-api:724.v538c2362b_dfb_
workflow-support:1015.v785e5a_b_b_8b_22
ws-cleanup:0.49

What Operating System are you using (both controller, and any agents involved in the problem)?

CentOS Linux release 7.9.2009 (Core)

Reproduction steps

Example pipeline

pipeline {
  agent {
      label 'backend'
  }

  options {
      buildDiscarder(logRotator(numToKeepStr: '10'))
      timestamps()
      disableConcurrentBuilds()
  }

  environment {
      xxx
  }

  parameters {
      gitParameter(
          xxx
      )
  }

  stages {
      stage('Init config') {
          steps {
              script {
                  xxx
              }
          }
      }

      stage('Checkout') {
          steps {
              git branch: xxx
          }
      }

      stage('Package & Build & Deploy') {
          tools {
              xxx
          }

          steps {
              script {
                  xxx
              }
          }
      }
  }

  post {
      always {
          cleanWs()
      }
  }
}

Expected Results

All steps should be marked as Successful after completion.

Actual Results

Some steps remain in the Running state even after the post-action phase has completed, and the timer keeps increasing during this state. This behavior occurs intermittently and is not consistently reproducible.

example 1:

example 2:

Anything else?

This issue may have started occurring after the plugin upgrade. I have not encountered this behavior before.

Are you interested in contributing a fix?

No response

[timja]

Is the first version since last week that you updated to 867.vece2ec789435?

There was a hardening done in #1249 which stopped that happening for me

[timja]

#1253 will invalidate any existing ones caught in this state, can you let me know if you see the issue again if so we may need to add a delay to writing the cache or something.

[guolunwei]

Yes, we upgraded to version 867.vece2ec789435 last week.

I’ll verify again after #1253 and let you know if the issue still occurs.

[dawid-remitly]

I experienced similar effects on 867.vece2ec789435, now upgraded to 871.vd450b_6eb_d1e1 and still seeing problems:

• stages waiting for input disapear from left tree panel, but are visible in top graph
• finished stages still appear to be running, time counters are still ticking

Below example showing problem 1 & 2:

[guolunwei]

Same here — still seeing the issue after upgrading to 871.vd450b_6eb_d1e1.

[timja]

Reports of it still happening after the fix we tried: #1255 (comment)

[timja]

I created a runner pipeline:

// Wrapper for https://github.com/jenkinsci/pipeline-graph-view-plugin/issues/1252
// Triggers the downstream reproducer job many times so the intermittent
// "still running" cache bug has more chances to surface.
pipeline {
    agent any

    parameters {
        string(name: 'DOWNSTREAM_JOB', defaultValue: 'issue-1252', description: 'Name of the job running Jenkinsfile.issue-1252')
        string(name: 'RUN_COUNT', defaultValue: '30', description: 'How many times to trigger the downstream job')
        booleanParam(name: 'WAIT_FOR_DOWNSTREAM', defaultValue: false, description: 'If true, block on each downstream build; if false, queue them all and return')
    }

    stages {
        stage('Trigger downstream builds') {
            steps {
                script {
                    int count = params.RUN_COUNT.toInteger()
                    for (int i = 1; i <= count; i++) {
                        echo "Triggering ${params.DOWNSTREAM_JOB} run ${i}/${count}"
                        build job: params.DOWNSTREAM_JOB,
                              wait: params.WAIT_FOR_DOWNSTREAM,
                              propagate: false,
                              quietPeriod: 0
                    }
                }
            }
        }
    }
}

and can reproduce it with that.

[panicking]

@timja I have still as bug

hudson.AbortException: script returned exit code 1
[Pipeline] withCredentials
Masking supported pattern matches of $GERRIT_RESTAPI_USER or $GERRIT_RESTAPI_PASS
[Pipeline] {
[Pipeline] echo
GerritClient: setLabel {
	change: axelera/meta-axelera/45216/1 (I10c96c64a10a0d5ee7ba5a47e9207a5ecbf707b4)
	label: Verified
	verified: -1
}
[Pipeline] echo
GerritClient: gerritPost {
	url: https://gerrit-review.amarulasolutions.com/
	resource: /a/changes/axelera%2Fmeta-axelera~wrynose~I10c96c64a10a0d5ee7ba5a47e9207a5ecbf707b4/revisions/3ae75fe598cc34c5d75d06c53e5c55db9c31c2d5/review
	body: [message:Build Failed https://jenkins.amarulasolutions.com/job/amarula/job/axelera/job/axeleraBsp-verification/1134/: FAILURE, labels:[Verified:-1]]
}
[Pipeline] echo
GerritClient: gerritPostJson {
	url: https://gerrit-review.amarulasolutions.com/
	resource: /a/changes/axelera%2Fmeta-axelera~wrynose~I10c96c64a10a0d5ee7ba5a47e9207a5ecbf707b4/revisions/3ae75fe598cc34c5d75d06c53e5c55db9c31c2d5/review
	body: {"message":"Build Failed https://jenkins.amarulasolutions.com/job/amarula/job/axelera/job/axeleraBsp-verification/1134/: FAILURE","labels":{"Verified":"-1"}}
}
[Pipeline] }
[Pipeline] // withCredentials
[Pipeline] dir
Running in /opt/workers/worker02/workspace/amarula/axelera/axeleraBsp-verification/meta-axelera/build/tmp-glibc
[Pipeline] {
[Pipeline] deleteDir
[Pipeline] }
[Pipeline] // dir
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // throttle
[Pipeline] End of Pipeline
ERROR: script returned exit code 1
Notified JIRA that a build has completed.
Finished: FAILURE
Deleted 0 stash(es) from https://*******/1134/stashes/

[timja]

Is there a reproducer for that or does it just happen randomly?