Frequent exception "java.io.StreamCorruptedException: invalid stream header" reported by the pipeline using GCE plugin

[mkjkec2005]

Jenkins and plugins versions report

Environment

Jenkins: 2.492.1
OS: Linux - 5.14.0-503.21.1.el9_5.x86_64
Java: 17.0.14 - Red Hat, Inc. (OpenJDK 64-Bit Server VM)

ace-editor:1.1
active-directory:2.39
analysis-model-api:12.9.1
ansicolor:1.0.6
ant:511.v0a_a_1a_334f41b_
antisamy-markup-formatter:162.v0e6ec0fcfcf6
apache-httpcomponents-client-4-api:4.5.14-269.vfa_2321039a_83
apache-httpcomponents-client-5-api:5.4-136.v5a_21779c63f8
artifactory:4.0.8
asm-api:9.7.1-97.v4cc844130d97
authentication-tokens:1.119.v50285141b_7e1
aws-credentials:243.v41c19a_fb_5dcf
aws-java-sdk-ec2:1.12.772-477.v650d756dcf6d
aws-java-sdk-minimal:1.12.772-477.v650d756dcf6d
aws-java-sdk2-core:2.30.10-22.v1339d66b_e0f4
aws-java-sdk2-ec2:2.30.10-22.v1339d66b_e0f4
blueocean:1.27.16
blueocean-autofavorite:1.2.5
blueocean-bitbucket-pipeline:1.27.16
blueocean-commons:1.27.16
blueocean-config:1.27.16
blueocean-core-js:1.27.16
blueocean-dashboard:1.27.16
blueocean-display-url:2.4.3
blueocean-events:1.27.16
blueocean-git-pipeline:1.27.16
blueocean-github-pipeline:1.27.16
blueocean-i18n:1.27.16
blueocean-jira:1.27.16
blueocean-jwt:1.27.16
blueocean-personalization:1.27.16
blueocean-pipeline-api-impl:1.27.16
blueocean-pipeline-editor:1.27.16
blueocean-pipeline-scm-api:1.27.16
blueocean-rest:1.27.16
blueocean-rest-impl:1.27.16
blueocean-web:1.27.16
bootstrap4-api:4.6.0-6
bootstrap5-api:5.3.3-1
bouncycastle-api:2.30.1.80-256.vf98926042a_9b_
branch-api:2.1214.v3f652804588d
build-name-setter:2.5.0
build-timeout:1.35
build-user-vars-plugin:195.v8c35f9d5c3dc
caffeine-api:3.1.8-133.v17b_1ff2e0599
checks-api:2.2.2
cloud-stats:377.vd8a_6c953e98e
cloudbees-bitbucket-branch-source:934.4.2
cloudbees-folder:6.982.vf165a_16c9507
command-launcher:118.v72741845c17a_
commons-compress-api:1.27.1-2
commons-lang3-api:3.17.0-84.vb_b_938040b_078
commons-text-api:1.13.0-153.v91dcd89e2a_22
conditional-buildstep:1.5.0
config-driven-pipeline:1.3
config-file-provider:982.vb_a_e458a_37021
configuration-as-code:1932.v75cb_b_f1b_698d
copyartifact:765.v0357cc6e6eb_3
credentials:1408.va_622a_b_f5b_1b_1
credentials-binding:687.v619cb_15e923f
dashboard-view:2.528.v3470c02b_d7c9
data-tables-api:2.1.8-1
display-url-api:2.209.v582ed814ff2f
docker-build-step:2.12
docker-commons:445.v6b_646c962a_94
docker-java-api:3.4.1-96.v77147a_de67f8
docker-plugin:1.9.0
docker-workflow:595.v60cec912059b_
durable-task:581.v299a_5609d767
echarts-api:5.5.1-5
eddsa-api:0.3.0-4.v84c6f0f4969e
email-ext:1876.v28d8d38315b_d
envinject:2.926.v69c9b_3896a_96
envinject-api:1.235.va_14c74f8f487
extended-choice-parameter:382.v5697b_32134e8
external-monitor-job:221.v35059272565b_
favorite:2.225.v68765b_b_a_1fa_3
font-awesome-api:6.6.0-2
forensics-api:2.7.0
generic-webhook-trigger:2.3.1
git:5.7.0
git-client:6.1.2
git-parameter:0.11.0
git-server:126.v0d945d8d2b_39
github:1.41.0
github-api:1.321-478.vc9ce627ce001
github-branch-source:1810.v913311241fa_9
github-pullrequest:0.7.2
gitlab-plugin:1.9.7
google-compute-engine:4.683.v0ce26579a_ee7
google-oauth-plugin:1.330.vf5e86021cb_ec
gradle:2.14
groovy:497.v7b_061a_a_de65d
gson-api:2.12.1-113.v347686d6729f
h2-api:11.1.4.199-30.v1c64e772f3a_c
handlebars:3.0.8
handy-uri-templates-2-api:2.1.8-30.v7e777411b_148
htmlpublisher:1.37
http_request:1.20
icon-shim:3.0.0
instance-identity:201.vd2a_b_5a_468a_a_6
ionicons-api:74.v93d5eb_813d5f
ivy:2.9
jackson2-api:2.17.0-386.vcb_b_037da_0d62
jakarta-activation-api:2.1.3-1
jakarta-mail-api:2.1.3-1
javadoc:310.v032f3f16b_0f8
javax-activation-api:1.2.0-7
javax-mail-api:1.6.2-10
jaxb:2.3.9-1
jdk-tool:83.v417146707a_3d
jenkins-design-language:1.27.16
jersey2-api:2.45-154.v4ded3dc34f81
jira:3.13
jjwt-api:0.11.5-112.ve82dfb_224b_a_d
jnr-posix-api:3.1.20-125.vb_6ec4b_21b_15e
job-dsl:1.90
job-import-plugin:3.6
jobConfigHistory:1302.va_65b_ccb_fde71
joda-time-api:2.13.1-115.va_6b_5f8efb_1d8
jquery:1.12.4-3
jquery-detached:1.2.1
jquery3-api:3.7.1-2
jsch:0.2.16-86.v42e010d9484b_
json-api:20250107-125.v28b_a_ffa_eb_f01
json-path-api:2.9.0-148.v22a_7ffe323ce
junit:1314.vd966e9a_88895
kubernetes:4306.vc91e951ea_eb_d
kubernetes-client-api:6.10.0-240.v57880ce8b_0b_2
kubernetes-credentials:190.v03c305394deb_
ldap:776.vddf3e325103b_
localization-support:1.2
localization-zh-cn:371.v23851f835d6b_
lockable-resources:1349.v8b_ccb_c5487f7
mailer:489.vd4b_25144138f
mapdb-api:1.0.9-40.v58107308b_7a_7
matrix-auth:3.2.4
matrix-project:845.vffd7fa_f27555
maven-plugin:3.25
mercurial:1309.v6802b_f0efb_b_9
metrics:4.2.21-461.v881e35d8fa_b_a_
mina-sshd-api-common:2.14.0-143.v2b_362fc39576
mina-sshd-api-core:2.14.0-143.v2b_362fc39576
momentjs:1.1.1
nodejs:1.6.3
oauth-credentials:0.653.v14cf2088e950
okhttp-api:4.11.0-183.va_87fc7a_89810
oss-symbols-api:296.v4981240eeb_1a_
pam-auth:1.11
parameterized-trigger:840.v3c7d4a_a_5e6c7
pipeline-build-step:555.v589d5c24a_3d6
pipeline-github-lib:61.v629f2cc41d83
pipeline-graph-analysis:216.vfd8b_ece330ca_
pipeline-groovy-lib:752.vdddedf804e72
pipeline-input-step:513.v7eb_fe7e2c0fc
pipeline-milestone-step:119.vdfdc43fc3b_9a_
pipeline-model-api:2.2221.vc657003fb_d93
pipeline-model-declarative-agent:1.1.1
pipeline-model-definition:2.2221.vc657003fb_d93
pipeline-model-extensions:2.2221.vc657003fb_d93
pipeline-rest-api:2.37
pipeline-stage-step:312.v8cd10304c27a_
pipeline-stage-tags-metadata:2.2221.vc657003fb_d93
pipeline-stage-view:2.37
pipeline-utility-steps:2.18.0
plain-credentials:183.va_de8f1dd5a_2b_
plugin-util-api:5.1.0
popper-api:1.16.1-3
popper2-api:2.11.6-5
prism-api:1.29.0-18
publish-over:0.22
publish-over-ssh:387.vec3df0f668cd
pubsub-light:1.18
rebuild:338.va_0a_b_50e29397
resource-disposer:0.25
role-strategy:756.v978cb_392eb_d3
run-condition:243.v3c3f94e46a_8b_
scm-api:703.v72ff4b_259600
script-security:1373.vb_b_4a_a_c26fa_00
simple-queue:1.4.6
slack:761.v2a_8770f0d169
snakeyaml-api:2.3-123.v13484c65210a_
sonar:2.18
sse-gateway:1.27
ssh:158.ve2a_e90fb_7319
ssh-agent:376.v8933585c69d3
ssh-credentials:349.vb_8b_6b_9709f5b_
ssh-slaves:3.1031.v72c6b_883b_869
sshd:3.353.v2b_d33c46e970
structs:338.v848422169819
subversion:1287.vd2d507146906
thinBackup:2.1.2
throttle-concurrents:2.16
timestamper:1.28
token-macro:444.v52de7e9c573d
trilead-api:2.147.vb_73cc728a_32e
uno-choice:2.8.6
variant:70.va_d9f17f859e0
windows-slaves:1.8.1
workflow-aggregator:600.vb_57cdd26fdd7
workflow-api:1363.v03f731255494
workflow-basic-steps:1079.vce64b_a_929c5a_
workflow-cps:4018.vf02e01888da_f
workflow-cps-global-lib:612.v55f2f80781ef
workflow-durable-task-step:1405.v1fcd4a_d00096
workflow-job:1505.vea_4b_20a_4a_495
workflow-multibranch:800.v5f0a_a_660950e
workflow-scm-step:427.v4ca_6512e7df1
workflow-step-api:686.v603d058a_e148
workflow-support:946.v2a_79d8a_4b_e14
ws-cleanup:0.48

What Operating System are you using (both controller, and any agents involved in the problem)?

Controller - Red Hat Enterprise Linux release 9.5
Slave - GCP VM based on Debian 12.

Reproduction steps

Trigger multiple pipelines in parallel. Each of these pipelines provision a dynamic Vm and run pipeline stages. Some stages are on the host and other run inside the docker container.

Expected Results

No abnormal termination of the pipelines.

Actual Results

Pipeline are terminated randomly with the below exception.

---

java.io.StreamCorruptedException: invalid stream header: 636F7272
at java.base/java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:958)
at java.base/java.io.ObjectInputStream.(ObjectInputStream.java:392)
at hudson.remoting.ObjectInputStreamEx.(ObjectInputStreamEx.java:50)
at hudson.remoting.Command.readFrom(Command.java:141)
at hudson.remoting.Command.readFrom(Command.java:127)
at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:62)
Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to gx-dcc-pool-80nyd4
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1923)
at hudson.remoting.Request.call(Request.java:204)
at hudson.remoting.Channel.call(Channel.java:1111)
at hudson.Launcher$RemoteLauncher.launch(Launcher.java:1121)
at hudson.Launcher$ProcStarter.start(Launcher.java:507)
at PluginClassLoader for durable-task//org.jenkinsci.plugins.durabletask.BourneShellScript.launchWithCookie(BourneShellScript.java:180)
at PluginClassLoader for durable-task//org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:134)
at PluginClassLoader for workflow-durable-task-step//org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:330)
at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:323)
at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:196)
at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:124)
at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:47)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.sandbox.DefaultInvoker.methodCall(DefaultInvoker.java:20)
at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.LoggingInvoker.methodCall(LoggingInvoker.java:117)
at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:90)
at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:116)
at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:85)
at jdk.internal.reflect.GeneratedMethodAccessor406.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.impl.LocalVariableBlock$LocalVariable.get(LocalVariableBlock.java:39)
at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.LValueBlock$GetAdapter.receive(LValueBlock.java:30)
at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.impl.LocalVariableBlock.evalLValue(LocalVariableBlock.java:28)
at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.LValueBlock$BlockImpl.eval(LValueBlock.java:55)
at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.LValueBlock.eval(LValueBlock.java:16)
at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.Next.step(Next.java:83)
at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:147)
at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:17)
at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:49)
at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:180)
at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:422)
at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:330)
at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:294)
at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService.lambda$wrap$4(CpsVmExecutorService.java:140)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
at jenkins.util.ErrorLoggingExecutorService.lambda$wrap$0(ErrorLoggingExecutorService.java:51)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.call(CpsVmExecutorService.java:53)
at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.call(CpsVmExecutorService.java:50)
at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:136)
at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:275)
at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService.lambda$categoryThreadFactory$0(CpsVmExecutorService.java:50)
at java.base/java.lang.Thread.run(Thread.java:840)
Also: org.jenkinsci.plugins.workflow.actions.ErrorAction$ErrorId: 135b1982-c3ac-4b43-8172-9f679598a3df
Caused: hudson.remoting.RequestAbortedException
at hudson.remoting.Request.abort(Request.java:358)
at hudson.remoting.Channel.terminate(Channel.java:1196)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:95)
Finished: FAILURE

Anything else?

No response

Are you interested in contributing a fix?

No response

[gbhat618]

@mkjkec2005 ,

This issue might be due to an incompatibility between the agent and controller, possibly related to the Java version.

Relevant references:

• Stack Overflow discussion
• CloudBees KB article

I see you're running the latest controller with Java 17 (correct setup). Are all agents also on Java 17? If a project requires an older version, the agent should have multiple Java versions installed, with the agent process itself running on Java 17.

This doesn't seem directly related to the GCE plugin but might be influenced by pipeline design, especially with multiple agents and parallel stages. If you can provide a minimal reproducible pipeline, I'd be interested to take a closer look.

[gbhat618]

Also, the pipeline uses all dynamic agents or are there any static agents involved? if there are static agents - the version of the agent should be compatible with the controller (i.e. download the agent.jar from the controller itself), to avoid any remoting version compatibility issues.

[mkjkec2005]

@gbhat618 Thank you for your response, and apologies for the delayed reply as I was out of the office. We are using a simple pipeline that runs a few shell scripts, which takes around 1 to 1.5 hours to complete. Please note that it fails roughly once out of every 10 runs. For your reference, I’ve included the pipeline snippet below. We rely heavily on this plugin, and this issue is significantly affecting the reliability of our automation.

`pipeline {
agent { label 'my-gce-pool' }

options {
    timestamps()
    buildDiscarder(logRotator(numToKeepStr: '20'))
    timeout(time: 3, unit: 'HOURS')
    skipDefaultCheckout()
}

environment {

    BUILD_WORKSPACE = "<some workspace>"
    
}

stages {

    stage('Setup workspace') {
	
        steps {
		   
            script {
              
               // some git clone commands 
				
                
            }
        }
    }
	stage('Model Validation') {
		
        
         steps {
            script {
                
                
			   sh '''
			    // export some shell variables
                 export CHANGE_ID=${PULL_REQUEST_ID}
                 export CHANGE_TARGET=${PULL_REQUEST_TO_BRANCH}
                 echo "CHANGE_ID is $CHANGE_ID"
                 echo "CHANGE_TARGET is $CHANGE_TARGET"
				 BIN_DIR=${BUILD_WORKSPACE}/bin
				 YANG_DIR=${BUILD_WORKSPACE}/yang
				 PYANG_DIR="../update_pyang"
                 //execute some shell commands that take totally 1-1.5 hr to complete. 
			 '''


	     }
			echo "stage completed!!!"
       }          
	}
  }
}

post {
        always {
               echo 'Cleanup workspace!!!'
			   sh 'sudo rm -rf ${BUILD_WORKSPACE}'
            }
     }

`

[smithdm]

I suffered 'random' java.io.StreamCorruptedException: invalid stream header: 636F7272 errors for a couple of months.

I was using GCE plugin with an instance template based on debian.

• Created in early Jan'25
• Test ran fine until mid to late Jan, what I thought was random ....invalid stream header: 636F7272
• JDK was being updated.
  ** Checked JDK updates on debian, and there was a minor update rolled out!

My issue issue was entirely down to unattended-upgrades being enabled on the instance I was using. The default JDK was getting updated and pulling the rug from under the Jenkins agent, BANG that exception every time!

Since disabling unattended-upgrades I've had no issues.

• If you are concerned about disabling because of security, then simply recreate a new template or instance that the GCE plugin references created with latest versions. Update when you need security patches, but ensure the floor under you is solid and not changing.

It may not be your issue but, it was impacting our environment.

Anyway, hope it may help.

[mkjkec2005]

Thank you @smithdm. I have made the change to disable unattended-upgrades. Let me monitor it for couple of days and share the observation.

[mkjkec2005]

@smithdm Was the "invalid stream header" completely eliminated after disabling unattended upgrades in your case? In the last 24 hours, I observed that the frequency of this random error is less, but I just hit the same one again.

java.io.StreamCorruptedException: invalid stream header: 636F7272 at java.base/java.io.ObjectInputStream.readStreamHeader(Unknown Source) at java.base/java.io.ObjectInputStream.<init>(Unknown Source) at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:50) at hudson.remoting.Command.readFrom(Command.java:142) at hudson.remoting.Command.readFrom(Command.java:128) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:61) Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to gx-dcc-pool-7z0k8l at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1787) at hudson.remoting.Request.call(Request.java:199) at hudson.remoting.Channel.call(Channel.java:1002) at hudson.Launcher$RemoteLauncher.launch(Launcher.java:1121) at hudson.Launcher$ProcStarter.start(Launcher.java:506) at org.jenkinsci.plugins.docker.workflow.client.DockerClient.launch(DockerClient.java:314) at org.jenkinsci.plugins.docker.workflow.client.DockerClient.launch(DockerClient.java:295) at org.jenkinsci.plugins.docker.workflow.client.DockerClient.launch(DockerClient.java:292) at org.jenkinsci.plugins.docker.workflow.client.DockerClient.stop(DockerClient.java:185) at org.jenkinsci.plugins.docker.workflow.WithContainerStep.destroy(WithContainerStep.java:111)

This mostly happens when a pipeline stage running under a docker container ends and the Docker plugin tries to stop the container.

[Pipeline] }
07:13:50 $ docker stop --time=1 ae53a585ebd5f38636d041905e867a1097ce4232708b92a8fd7047993518e5b5 --> Seen after this.
[Pipeline] // withDockerContainer
[Pipeline] }
[Pipeline] // stage

[mkjkec2005]

@smithdm , @gbhat618 Any clue on this issue? Any pointers will be really helpful. Thank you.
java.lang.InterruptedException
  at java.base/java.lang.Object.wait0(Native Method)
  at java.base/java.lang.Object.wait(Unknown Source)
  at hudson.remoting.Request.call(Request.java:177)
  at hudson.remoting.Channel.call(Channel.java:1002)
  at hudson.Launcher$RemoteLauncher.launch(Launcher.java:1121)
Also: org.jenkinsci.plugins.workflow.actions.ErrorAction$ErrorId: 16ed9dae-d5e9-4b14-908a-9fd6c9a64b94
Caused: java.io.InterruptedIOException
This mostly happens when a pipeline stage running under a docker container ends and the Docker plugin tries to stop the container. We are using reuseNode in the pipeline agent.

[gbhat618]

The above stacktrace is not sufficient to say where the error occurred.

We are using reuseNode in the pipeline agent.

The pipeline sample you shared last week doesn't include docker-plugin or the reuseNode part, so it's unclear which one might be causing the issue.

This discussion has limited visibility. I suggest posting on Jenkins Community, where more experts on pipeline design can help identify the issue.

[gbhat618]

By any chance, do you have two Jenkins instances pointing to the same docker infra (i.e. same config in docker cloud in jenkins)? (perhaps you may have cloned the Jenkins instance for testing purposes or something..)

There is an orphan docker container cleanup in the docker-plugin, which runs every 5min -- it will cleanup the container if those container are not attached to the current Jenkins instance. If that is the case - a way to prevent that kind of cleanup happening is by passing a system property during jenkins startup (you would need to do that for both jenkins if they are pointing to same docker infra)

-Dcom.nirima.jenkins.plugins.docker.DockerContainerWatchdog.enabled=false

[em-simone]

I have the same issue, I'm not sure it depends on docker commands though.
This is an example of stacktrace, even though the details may differ from time to time.
The key line here is Unable to create live FilePath for jenkins-yil83x; jenkins-yil83x was marked offline: Connection was broken and I think this is a network issue.

hudson.remoting.ProxyException: java.io.StreamCorruptedException: invalid stream header: 636F7272
	at java.base/java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:958)
	at java.base/java.io.ObjectInputStream.<init>(ObjectInputStream.java:392)
	at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:50)
	at hudson.remoting.Command.readFrom(Command.java:141)
	at hudson.remoting.Command.readFrom(Command.java:127)
	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:62)
Also:   hudson.remoting.ProxyException: hudson.remoting.Channel$CallSiteStackTrace: Remote call to jenkins-yil83x
		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1916)
		at hudson.remoting.Request.call(Request.java:204)
		at hudson.remoting.Channel.call(Channel.java:1107)
		at hudson.Launcher$RemoteLauncher.launch(Launcher.java:1136)
		at hudson.Launcher$ProcStarter.start(Launcher.java:507)
		at PluginClassLoader for durable-task//org.jenkinsci.plugins.durabletask.BourneShellScript.launchWithCookie(BourneShellScript.java:180)
		at PluginClassLoader for durable-task//org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:134)
		at PluginClassLoader for workflow-durable-task-step//org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:330)
		at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:323)
		at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:196)
		at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:124)
		at groovy.lang.MetaClassImpl.invokeMethodOnGroovyObject(MetaClassImpl.java:1295)
		at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1184)
		at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034)
		at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:41)
		at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
		at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
		at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.sandbox.DefaultInvoker.methodCall(DefaultInvoker.java:20)
		at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.LoggingInvoker.methodCall(LoggingInvoker.java:118)
		at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:90)
		at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:114)
		at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:83)
		at jdk.internal.reflect.GeneratedMethodAccessor178.invoke(Unknown Source)
		at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		at java.base/java.lang.reflect.Method.invoke(Method.java:569)
		at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
		at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:108)
		at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:83)
		at jdk.internal.reflect.GeneratedMethodAccessor178.invoke(Unknown Source)
		at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		at java.base/java.lang.reflect.Method.invoke(Method.java:569)
		at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
		at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.impl.CastBlock$ContinuationImpl.cast(CastBlock.java:47)
		at jdk.internal.reflect.GeneratedMethodAccessor209.invoke(Unknown Source)
		at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		at java.base/java.lang.reflect.Method.invoke(Method.java:569)
		at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
		at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.impl.CollectionLiteralBlock$ContinuationImpl.dispatch(CollectionLiteralBlock.java:55)
		at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.impl.CollectionLiteralBlock$ContinuationImpl.item(CollectionLiteralBlock.java:45)
		at jdk.internal.reflect.GeneratedMethodAccessor213.invoke(Unknown Source)
		at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		at java.base/java.lang.reflect.Method.invoke(Method.java:569)
		at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
		at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
		at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.Next.step(Next.java:83)
		at PluginClassLoader for workflow-cps//com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:147)
		at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:17)
		at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:49)
		at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:180)
		at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:419)
		at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:327)
		at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:292)
		at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService.lambda$wrap$4(CpsVmExecutorService.java:140)
		at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
		at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139)
		at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
		at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
		at jenkins.util.ErrorLoggingExecutorService.lambda$wrap$0(ErrorLoggingExecutorService.java:51)
		at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
		at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
		at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
		at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
		at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.call(CpsVmExecutorService.java:53)
		at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.call(CpsVmExecutorService.java:50)
		at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:136)
		at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:275)
		at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService.lambda$categoryThreadFactory$0(CpsVmExecutorService.java:50)
		at java.base/java.lang.Thread.run(Thread.java:840)
Also:   hudson.remoting.ProxyException: org.jenkinsci.plugins.workflow.actions.ErrorAction$ErrorId: 53dad5e0-1b4a-4cbd-8ac1-839f4bc6819c
Also:   hudson.remoting.ProxyException: org.jenkinsci.plugins.workflow.support.steps.AgentOfflineException: Unable to create live FilePath for jenkins-yil83x; jenkins-yil83x was marked offline: Connection was broken
		at PluginClassLoader for workflow-durable-task-step//org.jenkinsci.plugins.workflow.support.steps.ExecutorStepDynamicContext$FilePathTranslator.get(ExecutorStepDynamicContext.java:188)
		at PluginClassLoader for workflow-durable-task-step//org.jenkinsci.plugins.workflow.support.steps.ExecutorStepDynamicContext$FilePathTranslator.get(ExecutorStepDynamicContext.java:160)
		at PluginClassLoader for workflow-durable-task-step//org.jenkinsci.plugins.workflow.support.steps.ExecutorStepDynamicContext$Translator.get(ExecutorStepDynamicContext.java:153)
		at PluginClassLoader for workflow-durable-task-step//org.jenkinsci.plugins.workflow.support.steps.ExecutorStepDynamicContext$FilePathTranslator.get(ExecutorStepDynamicContext.java:170)
		at PluginClassLoader for workflow-durable-task-step//org.jenkinsci.plugins.workflow.support.steps.ExecutorStepDynamicContext$FilePathTranslator.get(ExecutorStepDynamicContext.java:160)
		at PluginClassLoader for workflow-step-api//org.jenkinsci.plugins.workflow.steps.DynamicContext$Typed.get(DynamicContext.java:95)
		at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.ContextVariableSet.get(ContextVariableSet.java:139)
		at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsThread.getContextVariable(CpsThread.java:135)
		at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsStepContext.doGet(CpsStepContext.java:297)
		at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsBodySubContext.doGet(CpsBodySubContext.java:88)
		at PluginClassLoader for workflow-support//org.jenkinsci.plugins.workflow.support.DefaultStepContext.get(DefaultStepContext.java:97)
		at PluginClassLoader for credentials-binding//org.jenkinsci.plugins.credentialsbinding.impl.BindingStep$Callback.finished(BindingStep.java:247)
		at PluginClassLoader for credentials-binding//org.jenkinsci.plugins.credentialsbinding.impl.BindingStep$Execution2$Callback2.finished(BindingStep.java:161)
		at PluginClassLoader for workflow-step-api//org.jenkinsci.plugins.workflow.steps.GeneralNonBlockingStepExecution$TailCall.lambda$onFailure$1(GeneralNonBlockingStepExecution.java:157)
		at PluginClassLoader for workflow-step-api//org.jenkinsci.plugins.workflow.steps.GeneralNonBlockingStepExecution.lambda$run$0(GeneralNonBlockingStepExecution.java:77)
		at PluginClassLoader for opentelemetry-api//io.opentelemetry.context.Context.lambda$wrap$1(Context.java:241)
		at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
		at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
		at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
		at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
		at java.base/java.lang.Thread.run(Thread.java:840)
Caused: hudson.remoting.ProxyException: hudson.remoting.RequestAbortedException: java.io.StreamCorruptedException: invalid stream header: 636F7272
	at hudson.remoting.Request.abort(Request.java:358)
	at hudson.remoting.Channel.terminate(Channel.java:1189)
	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:95)
Caused: hudson.remoting.ProxyException: java.util.concurrent.ExecutionException: hudson.remoting.RequestAbortedException: java.io.StreamCorruptedException: invalid stream header: 636F7272
	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsBodyExecution.get(CpsBodyExecution.java:303)
	at PluginClassLoader for folder-properties//com.mig82.folders.step.FolderPropertiesStep$Execution.run(FolderPropertiesStep.java:55)
	at PluginClassLoader for folder-properties//com.mig82.folders.step.FolderPropertiesStep$Execution.run(FolderPropertiesStep.java:36)
	at PluginClassLoader for workflow-step-api//org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:49)
	at PluginClassLoader for opentelemetry-api//io.opentelemetry.context.Context.lambda$wrap$1(Context.java:241)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)
Finished: FAILURE

[jvanoosterom]

we have this problem too.

also if a parallel build (on gcloud node) succeeds, this it not picked up by the master (where Jenkins is started).

The agent is started by the preinstalled java, the build runs on a java we install on the fly for this build.
So the build continues. But the agent is gone (java update by system)

we indeed have a vm with an update pending for the java openjdk (installed via apt) when we start a fresh node .
i will disable 'unattended-upgrades' and see if this is the solution

[jvanoosterom]

our build just succeeded for the first time this week so this must have been the unattended-upgrades of the java jdk on the node

[smithdm]

Just to follow up on my original observations, unattended upgrades kicking in while Jenkins agent connected, my solution is here (my code - I disable in the VM instance template):

https://github.com/GoogleCloudPlatform/horizon-sdv/blob/647d41cd1392b60d38647613733775e020deea03/workloads/android/pipelines/environment/cf_instance_template/cf_host_initialise.sh#L73

It may be a line or so overkill but it works and I've not suffered the issues ever since.