Remote agents connection problems. SEVERE: Connection error has occurred

[menvol3]

Jenkins and plugins versions report

Environment

Jenkins: 2.504.3
OS: Linux - 6.12.55-74.119.amzn2023.x86_64
Java: 21.0.7 - Eclipse Adoptium (OpenJDK 64-Bit Server VM)
---
amazon-ecr:1.161.v1a_1e8df852d6
ansicolor:536.v13fa_b_860c267
ant:520.vd082ecfb_16a_9
antisamy-markup-formatter:173.v680e3a_b_69ff3
apache-httpcomponents-client-4-api:4.5.14-269.vfa_2321039a_83
apache-httpcomponents-client-5-api:5.6-183.ve5a_8a_b_e71e59
artifactory:4.0.8
asm-api:9.9.1-189.vb_5ef2964da_91
authentication-tokens:1.144.v5ff4a_5ec5c33
authorize-project:2.0.0
aws-credentials:254.v978a_5e206a_d7
aws-java-sdk:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-api-gateway:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-autoscaling:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-cloudformation:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-cloudfront:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-cloudwatch:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-codebuild:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-codedeploy:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-ec2:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-ecr:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-ecs:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-efs:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-elasticbeanstalk:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-elasticloadbalancingv2:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-iam:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-kinesis:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-lambda:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-logs:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-minimal:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-organizations:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-secretsmanager:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-sns:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-sqs:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-ssm:1.12.780-480.v4a_0819121a_9e
aws-java-sdk2-core:2.33.4-62.vc1a_8df64b_4c9
aws-java-sdk2-ec2:2.33.4-62.vc1a_8df64b_4c9
aws-java-sdk2-ecr:2.33.4-62.vc1a_8df64b_4c9
aws-java-sdk2-netty-nio-client:2.33.4-62.vc1a_8df64b_4c9
aws-java-sdk2-s3:2.33.4-62.vc1a_8df64b_4c9
aws-java-sdk2-secretsmanager:2.33.4-62.vc1a_8df64b_4c9
aws-secrets-manager-credentials-provider:2.222.v376939a_9ffb_b
aws-secrets-manager-secret-source:2.87.v49cc86c50c3b_
blueocean:1.27.25
blueocean-autofavorite:1.2.5
blueocean-bitbucket-pipeline:1.27.25
blueocean-commons:1.27.25
blueocean-config:1.27.25
blueocean-core-js:1.27.25
blueocean-dashboard:1.27.25
blueocean-display-url:2.4.4
blueocean-events:1.27.25
blueocean-git-pipeline:1.27.25
blueocean-github-pipeline:1.27.25
blueocean-i18n:1.27.25
blueocean-jira:1.27.25
blueocean-jwt:1.27.25
blueocean-personalization:1.27.25
blueocean-pipeline-api-impl:1.27.25
blueocean-pipeline-editor:1.27.25
blueocean-pipeline-scm-api:1.27.25
blueocean-rest:1.27.25
blueocean-rest-impl:1.27.25
blueocean-web:1.27.25
bootstrap5-api:5.3.8-890.v1c5cf4fa_178e
bouncycastle-api:2.30.1.82-277.v70ca_0b_877184
branch-api:2.1268.v044a_87612da_8
build-discarder:158.vce570d01ce4c
build-environment:1.7
build-name-setter:2.5.1
build-timeout:1.38
build-user-vars-plugin:212.vd6b_e9f6d0cdb_
caffeine-api:3.2.3-194.v31a_b_f7a_b_5a_81
checks-api:402.vca_263b_f200e3
cloudbees-bitbucket-branch-source:937.2.3
cloudbees-disk-usage-simple:256.v20ec4eb_884f1
cloudbees-folder:6.1053.vd62fb_b_f7367b_
command-launcher:123.v37cfdc92ef67
commons-collections4-api:4.5.0-8.va_d5448ef9011
commons-compress-api:1.28.0-2
commons-lang3-api:3.20.0-109.ve43756e2d2b_4
commons-text-api:1.15.0-210.v7480a_da_70b_9e
config-file-provider:1006.vc7366c201f57
configuration-as-code:2031.veb_a_fdda_b_3ffd
copyartifact:795.ve8e151429b_27
credentials:1480.v2246fd131e83
credentials-binding:702.vfe613e537e88
dashboard-view:2.558.v96b_901978e47
data-tables-api:2.3.4-1400.vb_1e3e3c4dfc8
display-url-api:2.217.va_6b_de84cc74b_
docker-commons:457.v0f62a_94f11a_3
docker-workflow:634.vedc7242b_eda_7
durable-task:651.v1f5e074fc83f
ec2:2045.v06da_da_a_46422
echarts-api:6.0.0-1146.v5c8f3b_8f0573
eddsa-api:0.3.0.1-29.v67e9a_1c969b_b_
editable-choice:71.v02a291ebbe45
email-ext:1933.v45cec755423f
extended-choice-parameter:388.ve7b_d0b_920e10
extended-read-permission:68.vd270568a_7520
favorite:2.253.v9b_413168133b_
font-awesome-api:7.0.1-859.v128d3a_efb_6e5
generic-webhook-trigger:2.4.1
git:5.8.1
git-client:6.5.0
git-server:137.ve0060b_432302
github:1.45.0
github-api:1.330-492.v3941a_032db_2a_
github-branch-source:1917.v9ee8a_39b_3d0d
gitlab-api:6.2.0-111.vf174d6b_8c12e
gitlab-branch-source:740.v04f287f9194d
gitlab-plugin:1.9.12
gradle:2.18.1203.v2c96b_1243c72
gson-api:2.13.2-173.va_a_092315913c
handy-uri-templates-2-api:2.1.8-38.vcea_5d521d5f3
htmlpublisher:427
http_request:1.24
instance-identity:203.v15e81a_1b_7a_38
ionicons-api:94.vcc3065403257
ivy:604.v6dd214d04ef3
jackson2-api:2.20.1-423.v13951f6b_6532
jakarta-activation-api:2.1.4-1
jakarta-mail-api:2.1.5-1
jakarta-xml-bind-api:4.0.6-12.vb_1833c1231d3
javadoc:354.vee1a_660b_4990
javax-activation-api:1.2.0-8
javax-mail-api:1.6.2-11
jaxb:2.3.9-143.v5979df3304e6
jdk-tool:83.v417146707a_3d
jenkins-design-language:1.27.25
jersey2-api:2.47-165.ve7809a_3e87e0
jersey3-api:3.1.11-4.v77818819c2e1
jira:3.21
jjwt-api:0.11.5-120.v0268cf544b_89
job-dsl:1.93
jobConfigHistory:1356.ve360da_6c523a_
joda-time-api:2.14.0-177.vd7e9347b_e7d5
jquery:1.12.4-3
jquery3-api:3.7.1-594.vb_3864f326cf0
jsch:0.2.16-95.v3eecb_55fa_b_78
json-api:20251224-185.v0cc18490c62c
json-path-api:2.10.0-202.va_9cc16c1e476
jsoup:1.22.1-76.v9cdb_2456c0e3
junit:1369.v15da_00283f06
kubernetes:4419.v36079e3b_eb_5b_
kubernetes-client-api:7.3.1-256.v788a_0b_787114
kubernetes-credentials:207.v492f58828b_ed
kubernetes-credentials-provider:1.301.v2eed6d439a_79
labeled-test-groups-publisher:1.2.9
lockable-resources:1438.v3c0f8c9e2060
mailer:525.v2458b_d8a_1a_71
matrix-auth:3.2.9
matrix-project:870.v9db_fcfc2f45b_
maven-plugin:3.27
mercurial:1323.ve69d2a_db_8a_b_d
metrics:4.2.37-494.v06f9a_939d33a_
mina-sshd-api-common:2.16.0-167.va_269f38cc024
mina-sshd-api-core:2.16.0-167.va_269f38cc024
mina-sshd-api-scp:2.16.0-167.va_269f38cc024
netty-api:4.1.130.Final-21.v5a_db_3eb_ea_034
node-iterator-api:72.vc90e81737df1
nodelabelparameter:851.vd94e5048d321
okhttp-api:4.12.0-195.vc02552c04ffd
oss-symbols-api:442.v99039087229b_
parameter-separator:296.v9b_a_90c81160d
parameterized-scheduler:374.v531b_4f4d99b_3
pipeline-agent-build-history:138.vc85ee72b_cb_4c
pipeline-aws:1.45
pipeline-build-step:584.vdb_a_2cc3a_d07a_
pipeline-githubnotify-step:49.vf37bf92d2bc8
pipeline-graph-analysis:245.v88f03631a_b_21
pipeline-groovy-lib:787.ve2fef0efdca_6
pipeline-input-step:540.v14b_100d754dd
pipeline-milestone-step:138.v78ca_76831a_43
pipeline-model-api:2.2277.v00573e73ddf1
pipeline-model-definition:2.2277.v00573e73ddf1
pipeline-model-extensions:2.2277.v00573e73ddf1
pipeline-rest-api:2.39
pipeline-stage-step:322.vecffa_99f371c
pipeline-stage-tags-metadata:2.2277.v00573e73ddf1
pipeline-stage-view:2.39
pipeline-utility-steps:2.20.0
plain-credentials:199.v9f8e1f741799
plugin-util-api:6.1167.v022176c7e0ca_
prism-api:1.30.0-609.vf0a_df102d9a_f
prometheus:847.v8440e5c21e7c
pubsub-light:1.19
pyenv-pipeline:2.1.2
rebuild:338.va_0a_b_50e29397
resource-disposer:0.25
rich-text-publisher-plugin:1.5
role-strategy:848.va_a_ea_673cf0b_c
run-condition:276.v97298f3a_cd51
s3:517.va_43007f79db_a_
saml:4.595.vec7523b_5d543
scm-api:724.v7d839074eb_5c
script-security:1385.v7d2d9ec4d909
simple-theme-plugin:230.v8b_fd91b_b_800c
slack:795.v4b_9705b_e6d47
snakeyaml-api:2.5-143.v93b_c004f89de
snsnotify:2.1
sonar:2.18.2
splunk-devops:1.11.1
sse-gateway:1.28
ssh-agent:386.v36cc0c7582f0
ssh-credentials:361.vb_f6760818e8c
ssh-slaves:3.1096.v0b_cc466e4323
sshd:3.374.v19b_d59ce6610
structs:362.va_b_695ef4fdf9
throttle-concurrents:625.vc8b_e469e9a_b_c
timestamper:1.30
token-macro:477.vd4f0dc3cb_cf1
trilead-api:2.284.v1974ea_324382
uno-choice:2.8.8
variant:70.va_d9f17f859e0
versioncolumn:400.v3c5c3004f31d
vsphere-cloud:2.27
workflow-aggregator:608.v67378e9d3db_1
workflow-api:1384.vdc05a_48f535f
workflow-basic-steps:1098.v808b_fd7f8cf4
workflow-cps:4253.vc894cd5750c6
workflow-durable-task-step:1464.v2d3f5c68f84c
workflow-job:1559.va_a_533730b_ea_d
workflow-multibranch:821.vc3b_4ea_780798
workflow-scm-step:466.va_d69e602552b_
workflow-step-api:710.v3e456cc85233
workflow-support:1010.vb_b_39488a_9841
ws-cleanup:0.49

What Operating System are you using (both controller, and any agents involved in the problem)?

Server is running on AWS EKS 1.34
Agents running on bot macOS (15.6,15,7 26.1) and Ubuntu Server 24.04
We use Amazon Corretto Java version 21

Reproduction steps

Observed Failure Sequence

1. All agents have their last successful communication at the same moment - responding to the server's periodic node monitoring burst (ArchitectureMonitor, ClockDifference, ResponseTimeMonitor, SwapSpaceMonitor, JVMVersionMonitor, VersionMonitor)
2. After the monitoring burst, there is a natural silence period (no scheduled communication from either side)
3. During this silence, something terminates the TCP connections (RST) — the agents don't know because they have nothing to send
4. Each agent's next scheduled PingThread$Ping (on its own 5-minute cadence) is the first data sent after the silence
5. The ping gets a TCP RST back (~185ms latency), and the agent detects Connection reset
6. Agent immediately attempts to reconnect - server rejects with "already connected"
7. Agent retries with exponential backoff (1s, 3s, 7s, 10s, 10s, 10s...) - all rejected
8. After ~4 minutes, server finally clears the stale connection and agent reconnects

Agent-1 log

Agent Log — Last Activity Before Disconnect (agent-1)

The last successful communication is the server-initiated node monitoring burst. After the agent responds to VersionMonitor$SlaveVersion, there is complete silence until the next scheduled ping:

Feb 19, 2026 1:58:23 PM hudson.remoting.Channel$1 handle
FINE: Received UserRequest:hudson.node_monitors.ArchitectureMonitor$GetArchTask@3e5d95f3
Feb 19, 2026 1:58:23 PM hudson.remoting.Channel$1 handle
FINE: Completed command UserRequest:hudson.node_monitors.ArchitectureMonitor$GetArchTask@3e5d95f3. It took 1ms
Feb 19, 2026 1:58:23 PM hudson.remoting.Channel send
FINE: Send Response:UserRequest:hudson.node_monitors.ArchitectureMonitor$GetArchTask@3e5d95f3(hudson.remoting.UserRequest$NormalResponse)
Feb 19, 2026 1:58:23 PM hudson.remoting.Channel$1 handle
FINE: Received UserRequest:hudson.model.Slave$GetClockDifference1@35e138ef
Feb 19, 2026 1:58:23 PM hudson.remoting.Channel$1 handle
FINE: Completed command UserRequest:hudson.model.Slave$GetClockDifference1@35e138ef. It took 0ms
Feb 19, 2026 1:58:23 PM hudson.remoting.Channel send
FINE: Send Response:UserRequest:hudson.model.Slave$GetClockDifference1@35e138ef(hudson.remoting.UserRequest$NormalResponse)
Feb 19, 2026 1:58:23 PM hudson.remoting.Channel$1 handle
FINE: Received UserRequest:hudson.FilePath$CallableWith@55c50292
Feb 19, 2026 1:58:23 PM hudson.remoting.Channel$1 handle
FINE: Completed command UserRequest:hudson.FilePath$CallableWith@55c50292. It took 1ms
Feb 19, 2026 1:58:23 PM hudson.remoting.Channel send
FINE: Send Response:UserRequest:hudson.FilePath$CallableWith@55c50292(hudson.remoting.UserRequest$NormalResponse)
Feb 19, 2026 1:58:23 PM hudson.remoting.Channel$1 handle
FINE: Received UserRequest:hudson.node_monitors.ResponseTimeMonitor$Step1@77f88f2e
Feb 19, 2026 1:58:23 PM hudson.remoting.Channel$1 handle
FINE: Completed command UserRequest:hudson.node_monitors.ResponseTimeMonitor$Step1@77f88f2e. It took 1ms
Feb 19, 2026 1:58:23 PM hudson.remoting.Channel send
FINE: Send Response:UserRequest:hudson.node_monitors.ResponseTimeMonitor$Step1@77f88f2e(hudson.remoting.UserRequest$NormalResponse)
Feb 19, 2026 1:58:23 PM hudson.remoting.Channel$1 handle
FINE: Received UserRequest:hudson.node_monitors.SwapSpaceMonitor$MonitorTask@6c8b7161
Feb 19, 2026 1:58:23 PM hudson.remoting.Channel$1 handle
FINE: Completed command UserRequest:hudson.node_monitors.SwapSpaceMonitor$MonitorTask@6c8b7161. It took 1ms
Feb 19, 2026 1:58:23 PM hudson.remoting.Channel$1 handle
FINE: Received UserRequest:hudson.FilePath$CallableWith@5f784a6f
Feb 19, 2026 1:58:23 PM hudson.remoting.Channel$1 handle
FINE: Completed command UserRequest:hudson.FilePath$CallableWith@5f784a6f. It took 0ms
Feb 19, 2026 1:58:23 PM hudson.remoting.Channel send
FINE: Send Response:UserRequest:hudson.node_monitors.SwapSpaceMonitor$MonitorTask@6c8b7161(hudson.remoting.UserRequest$NormalResponse)
Feb 19, 2026 1:58:23 PM hudson.remoting.Channel send
FINE: Send Response:UserRequest:hudson.FilePath$CallableWith@5f784a6f(hudson.remoting.UserRequest$NormalResponse)
Feb 19, 2026 1:58:24 PM hudson.remoting.Channel$1 handle
FINE: Received UserRequest:hudson.plugin.versioncolumn.JVMVersionMonitor$JavaVersion@1040306a
Feb 19, 2026 1:58:24 PM hudson.remoting.Channel$1 handle
FINE: Completed command UserRequest:hudson.plugin.versioncolumn.JVMVersionMonitor$JavaVersion@1040306a. It took 0ms
Feb 19, 2026 1:58:24 PM hudson.remoting.Channel send
FINE: Send Response:UserRequest:hudson.plugin.versioncolumn.JVMVersionMonitor$JavaVersion@1040306a(hudson.remoting.UserRequest$NormalResponse)
Feb 19, 2026 1:58:26 PM hudson.remoting.Channel$1 handle
FINE: Received UserRequest:hudson.plugin.versioncolumn.VersionMonitor$SlaveVersion@6e1002dc
Feb 19, 2026 1:58:26 PM hudson.remoting.Channel$1 handle
FINE: Completed command UserRequest:hudson.plugin.versioncolumn.VersionMonitor$SlaveVersion@6e1002dc. It took 1ms
Feb 19, 2026 1:58:26 PM hudson.remoting.Channel send
FINE: Send Response:UserRequest:hudson.plugin.versioncolumn.VersionMonitor$SlaveVersion@6e1002dc(hudson.remoting.UserRequest$NormalResponse)

--- 43 seconds of complete silence (no log entries at all) ---

Agent Log — Connection Reset and Failed Reconnection (agent-1)

The next scheduled PingThread$Ping triggers the error:

Feb 19, 2026 1:59:09 PM hudson.remoting.Channel send
FINE: Send UserRequest:hudson.remoting.PingThread$Ping@6f7a7efa
Feb 19, 2026 1:59:09 PM io.jenkins.remoting.shaded.org.glassfish.tyrus.container.jdk.client.ClientFilter processError
SEVERE: Connection error has occurred
java.io.IOException: Connection reset
	at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.finishRead(UnixAsynchronousSocketChannelImpl.java:425)
	at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.finish(UnixAsynchronousSocketChannelImpl.java:195)
	at java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.onEvent(UnixAsynchronousSocketChannelImpl.java:217)
	at java.base/sun.nio.ch.EPollPort$EventHandlerTask.run(EPollPort.java:305)
	at java.base/java.lang.Thread.run(Thread.java:1583)

Feb 19, 2026 1:59:09 PM hudson.remoting.Launcher$CuiListener status
INFO: Read side closed
Feb 19, 2026 1:59:09 PM hudson.remoting.Launcher$CuiListener status
INFO: Terminated
Feb 19, 2026 1:59:09 PM hudson.remoting.Launcher$CuiListener status
INFO: Performing onReconnect operation.
Feb 19, 2026 1:59:09 PM hudson.slaves.ChannelPinger$2 onClosed
FINE: Terminating ping thread for <agent-1>
Feb 19, 2026 1:59:09 PM jenkins.slaves.restarter.JnlpSlaveRestarterInstaller$EngineListenerAdapterImpl onReconnect
INFO: Restarting agent via jenkins.slaves.restarter.UnixSlaveRestarter@6b01c72d

Agent Log — Repeated Reconnection Failures (agent-1)

All reconnection attempts fail with "Did not receive X-Remoting-Capability header" for ~4 minutes:

Feb 19, 2026 1:59:09 PM hudson.remoting.Launcher createEngine
INFO: Setting up agent: <agent-1>
Feb 19, 2026 1:59:09 PM hudson.remoting.Engine startEngine
INFO: Using Remoting version: 3301.v4363ddcca_4e7
Feb 19, 2026 1:59:11 PM hudson.remoting.Engine$1HeaderHandler afterResponse
WARNING: Did not receive X-Remoting-Capability header
Feb 19, 2026 1:59:11 PM hudson.remoting.Launcher$CuiListener status
INFO: Failed to connect: Handshake error.
Feb 19, 2026 1:59:11 PM hudson.remoting.Launcher$CuiListener status
INFO: Waiting 1 second before retry
Feb 19, 2026 1:59:13 PM hudson.remoting.Engine$1HeaderHandler afterResponse
WARNING: Did not receive X-Remoting-Capability header
Feb 19, 2026 1:59:13 PM hudson.remoting.Launcher$CuiListener status
INFO: Failed to connect: Handshake error.
Feb 19, 2026 1:59:13 PM hudson.remoting.Launcher$CuiListener status
INFO: Waiting 3 seconds before retry
Feb 19, 2026 1:59:17 PM hudson.remoting.Engine$1HeaderHandler afterResponse
WARNING: Did not receive X-Remoting-Capability header
Feb 19, 2026 1:59:17 PM hudson.remoting.Launcher$CuiListener status
INFO: Failed to connect: Handshake error.
Feb 19, 2026 1:59:17 PM hudson.remoting.Launcher$CuiListener status
INFO: Waiting 7 seconds before retry
Feb 19, 2026 1:59:24 PM hudson.remoting.Engine$1HeaderHandler afterResponse
WARNING: Did not receive X-Remoting-Capability header
Feb 19, 2026 1:59:24 PM hudson.remoting.Launcher$CuiListener status
INFO: Failed to connect: Handshake error.
Feb 19, 2026 1:59:24 PM hudson.remoting.Launcher$CuiListener status
INFO: Waiting 10 seconds before retry
[... continues every 10 seconds with same error until 14:03:11 ...]

Agent successfully reconnects after ~4 minutes:

Feb 19, 2026 2:03:11 PM hudson.remoting.Launcher$CuiListener status
INFO: WebSocket connection open
Feb 19, 2026 2:03:11 PM hudson.remoting.Launcher$CuiListener status
INFO: Connected

Server-Side Log — "Already Connected" Rejections

During the agent's reconnection attempts, the Jenkins server logs ConnectionRefusalException for every attempt. For agent-2 alone, 48 rejections were logged:

ConnectionRefusalException: <agent-2> is already connected

Across all 7 agents during the incident window (13:59 - 14:07 UTC), the server logged 1000+ ConnectionRefusalException errors.

Network Capture Evidence (pcap analysis)

Packet captures from two agents confirm the TCP-level behavior:

agent-1:

• 13:58:26.163 UTC: Last normal data exchange — agent sends 1799 bytes
• 13:58:26.348 UTC: Server ACKs the data
• 13:58:27.378 UTC: TCP Keep-Alive ACK from agent (last packet before silence)
• 42 seconds of complete silence — zero packets in either direction (not even TCP keep-alives)
• 13:59:09.288 UTC: Agent attempts to send 1913 bytes (PingThread$Ping)
• 13:59:09.474 UTC: Server responds with two RST+ACK packets (~186ms later)

agent-2:

• 13:58:26.343 UTC: Last normal data exchange — agent sends 1799 bytes
• 13:58:26.528 UTC: Server ACKs the data (last packet before silence)
• 2 minutes 37 seconds of complete silence — zero packets in either direction
• 14:01:03.721 UTC: Agent attempts to send 1913 bytes (PingThread$Ping)
• 14:01:03.906 UTC: Server responds with two RST+ACK packets (~185ms later)

The RST packets came from the server side (port 443) in response to the agent's post-silence data. The ~185ms delay is consistent with normal network round-trip time, confirming the server's TCP stack had already timed out the connection during the silence period.

Expected Results

A successful PingThread$Ping cycle produces 6 log messages (bidirectional exchange):

FINE: Send UserRequest:hudson.remoting.PingThread$Ping@3c7be347
FINE: Received UserRequest:hudson.remoting.PingThread$Ping@75605f7c
FINE: Completed command UserRequest:hudson.remoting.PingThread$Ping@75605f7c. It took 1ms
FINE: Send Response:UserRequest:hudson.remoting.PingThread$Ping@75605f7c(hudson.remoting.UserRequest$NormalResponse)
FINE: Received Response(hudson.remoting.UserRequest$NormalResponse)
FINE: Completed command Response:UserRequest:hudson.remoting.PingThread$Ping@3c7be347(hudson.remoting.UserRequest$NormalResponse). It took 0ms

Actual Results

The fatal ping only shows 1 message (the send) followed immediately by the Connection reset error:

FINE: Send UserRequest:hudson.remoting.PingThread$Ping@6f7a7efa
SEVERE: Connection error has occurred
java.io.IOException: Connection reset

Anything else?

Infrastructure Path

Agent (remote office) 
  → Internet 
    → AWS NLB (TLS termination, ALPN HTTP2Preferred) 
      → nginx-ingress controller (EKS pod) 
        → Jenkins controller (EKS pod)

• NLB: TLS listener with TLSv1.3, no explicit idle timeout configured (AWS default: 350s for TCP)
• nginx-ingress: No explicit proxy-read-timeout or proxy-send-timeout annotations on the Jenkins ingress (nginx defaults: 60s)
• Jenkins ingress annotations: only `nginx.ingress.kubernetes.io/use-regex: "true"

Recurring Pattern

This is not a one-off event. The same pattern has been observed across multiple incidents over 10 days, affecting different office locations:

Date	Time (UTC)	Location	Agents Affected	Duration
Feb 10	11:40-11:54	Office A	15	~14 min
Feb 10	16:14-16:15	Office B	~2	~1 min
Feb 11	08:46-08:49	Office B	~3	~3 min
Feb 16	13:19-13:46	Office C	~2	~27 min
Feb 19	13:59-14:07	Office A	7	~8 min

Are you interested in contributing a fix?

No response

[hriztam]

Hello, I would like to work on this issue

[Flamki]

Thanks for the detailed logs and timeline.

This pattern looks like an infrastructure idle-timeout + stale-channel sequence, not an immediate Jenkins core defect:

• remoting ping default interval is 300s (5 min), so there is a long idle window
• a proxy/LB can drop the idle TCP connection during that window
• next ping gets Connection reset
• immediate reconnect is rejected as already connected until the stale channel is cleared

Could you test with these settings and report back?

1. Ingress (nginx):

• nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
• nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
• nginx.ingress.kubernetes.io/proxy-http-version: "1.1"

2. Ensure NLB/TG idle timeout is greater than remoting ping interval (for safety, use >= 600s).

3. Controller JVM options:

• -Dhudson.slaves.ChannelPinger.pingIntervalSeconds=30
• -Dhudson.slaves.ChannelPinger.pingTimeoutSeconds=90
• if using -webSocket agents also add:
  • -Djenkins.websocket.pingInterval=25
  • -Djenkins.websocket.idleTimeout=120

If this removes the resets, we should treat this as infra configuration guidance. If it still reproduces, we can continue with a core-side fix investigation.