Skip to content

[JENKINS-70538] Cannot execute builds using SVN SCM with SVN+SSH using private key with passphrase #324

@jenkins-infra-bot

Description

@jenkins-infra-bot

I upgraded one of my build enviromments from Jenkins 2.346.2 to Jenkins 2.375.2 today, and upgraded all plugins installed on it to the latest versions (including, but not limited to bouncycastle-api and subversion).  After doing so, all jobs running on agent nodes begun failing with the following error:

Started by user 
Running as 
[EnvInject] - Loading node environment variables.
Building remotely on  in workspace 
Cleaning up 
Updating svn+ssh:// at revision ''
Using sole credentials  in realm ‘’
FATAL: java.lang.ExceptionInInitializerError
java.lang.NullPointerException
    at java.base/javax.crypto.ProviderVerifier.verify(ProviderVerifier.java:122)
    at java.base/javax.crypto.JceSecurity.verifyProvider(JceSecurity.java:191)
    at java.base/javax.crypto.JceSecurity.getVerificationResult(JceSecurity.java:217)
    at java.base/javax.crypto.Cipher.getInstance(Cipher.java:688)
Caused: java.lang.SecurityException: JCE cannot authenticate the provider BC
    at java.base/javax.crypto.Cipher.getInstance(Cipher.java:692)
    at java.base/javax.crypto.Cipher.getInstance(Cipher.java:623)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.apache.sshd.common.util.security.SecurityEntityFactory$2.getInstance(SecurityEntityFactory.java:130)
    at org.apache.sshd.common.util.security.SecurityUtils.getCipher(SecurityUtils.java:748)
    at org.apache.sshd.common.config.keys.loader.AESPrivateKeyObfuscator$LazyKeyLengthsHolder.detectSupportedKeySizes(AESPrivateKeyObfuscator.java:134)
    at org.apache.sshd.common.config.keys.loader.AESPrivateKeyObfuscator$LazyKeyLengthsHolder.(AESPrivateKeyObfuscator.java:121)
Caused: java.lang.ExceptionInInitializerError
    at org.apache.sshd.common.config.keys.loader.AESPrivateKeyObfuscator.getAvailableKeyLengths(AESPrivateKeyObfuscator.java:110)
    at org.apache.sshd.common.config.keys.loader.AESPrivateKeyObfuscator.getSupportedKeySizes(AESPrivateKeyObfuscator.java:51)
    at org.apache.sshd.common.config.keys.loader.AESPrivateKeyObfuscator.resolveKeyLength(AESPrivateKeyObfuscator.java:87)
    at org.apache.sshd.common.config.keys.loader.AESPrivateKeyObfuscator.applyPrivateKeyCipher(AESPrivateKeyObfuscator.java:58)
    at org.apache.sshd.common.config.keys.loader.pem.AbstractPEMResourceKeyPairParser.applyPrivateKeyCipher(AbstractPEMResourceKeyPairParser.java:227)
    at org.apache.sshd.common.config.keys.loader.pem.AbstractPEMResourceKeyPairParser.extractKeyPairs(AbstractPEMResourceKeyPairParser.java:170)
    at org.apache.sshd.common.config.keys.loader.AbstractKeyPairResourceParser.loadKeyPairs(AbstractKeyPairResourceParser.java:117)
    at org.apache.sshd.common.config.keys.loader.KeyPairResourceParser$2.loadKeyPairs(KeyPairResourceParser.java:166)
    at org.apache.sshd.common.config.keys.loader.pem.PEMResourceParserUtils$1.loadKeyPairs(PEMResourceParserUtils.java:53)
    at org.apache.sshd.common.config.keys.loader.KeyPairResourceParser$2.loadKeyPairs(KeyPairResourceParser.java:166)
    at org.apache.sshd.common.config.keys.loader.KeyPairResourceLoader.loadKeyPairs(KeyPairResourceLoader.java:157)
    at org.apache.sshd.common.config.keys.loader.KeyPairResourceLoader.loadKeyPairs(KeyPairResourceLoader.java:148)
    at org.apache.sshd.common.config.keys.loader.KeyPairResourceLoader.loadKeyPairs(KeyPairResourceLoader.java:139)
    at org.apache.sshd.common.config.keys.loader.KeyPairResourceLoader.loadKeyPairs(KeyPairResourceLoader.java:130)
    at org.apache.sshd.common.util.security.SecurityUtils.loadKeyPairIdentities(SecurityUtils.java:522)
    at org.tmatesoft.svn.core.internal.io.svn.SVNSSHPrivateKeyUtil.isValidPrivateKey(SVNSSHPrivateKeyUtil.java:99)
    at org.tmatesoft.svn.core.internal.io.svn.SVNSSHConnector.open(SVNSSHConnector.java:102)
    at org.tmatesoft.svn.core.internal.io.svn.SVNConnection.open(SVNConnection.java:80)
    at org.tmatesoft.svn.core.internal.io.svn.SVNRepositoryImpl.openConnection(SVNRepositoryImpl.java:1282)
    at org.tmatesoft.svn.core.internal.io.svn.SVNRepositoryImpl.testConnection(SVNRepositoryImpl.java:100)
    at org.tmatesoft.svn.core.io.SVNRepository.getRepositoryUUID(SVNRepository.java:268)
    at org.tmatesoft.svn.core.internal.wc2.SvnRepositoryAccess.createRepository(SvnRepositoryAccess.java:103)
    at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgRepositoryAccess.createRepository(SvnNgRepositoryAccess.java:211)
    at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.updateInternal(SvnNgAbstractUpdate.java:210)
    at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.update(SvnNgAbstractUpdate.java:115)
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to 
        at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1784)
        at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
        at hudson.remoting.Channel.call(Channel.java:1000)
        at hudson.FilePath.act(FilePath.java:1186)
        at hudson.FilePath.act(FilePath.java:1175)
        at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:970)
        at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:892)
        at hudson.scm.SCM.checkout(SCM.java:540)
        at hudson.model.AbstractProject.checkout(AbstractProject.java:1241)
        at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:649)
        at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:85)
        at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:521)
        at hudson.model.Run.execute(Run.java:1900)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
        at hudson.model.ResourceController.execute(ResourceController.java:107)
        at hudson.model.Executor.run(Executor.java:449)
Caused: java.lang.RuntimeException
    at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.throwThrowable(SvnNgAbstractUpdate.java:918)
    at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.update(SvnNgAbstractUpdate.java:125)
    at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgUpdate.run(SvnNgUpdate.java:40)
    at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgUpdate.run(SvnNgUpdate.java:18)
    at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgOperationRunner.run(SvnNgOperationRunner.java:20)
    at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:21)
    at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1239)
    at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294)
    at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:311)
    at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:291)
    at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:387)
    at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:165)
    at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:168)
    at hudson.scm.SubversionSCM$CheckOutUpdateTask.perform(SubversionSCM.java:1086)
    at hudson.scm.SubversionSCM$CheckOutUpdateTask.run(SubversionSCM.java:1067)
    at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:1037)
    at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:1020)
    at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3492)
    at hudson.remoting.UserRequest.perform(UserRequest.java:211)
    at hudson.remoting.UserRequest.perform(UserRequest.java:54)
    at hudson.remoting.Request$2.run(Request.java:377)
    at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:834)
Archiving artifacts
Started calculate disk usage of build
Finished Calculation of disk usage of build in 0 seconds
Started calculate disk usage of workspace
Finished Calculation of disk usage of workspace in 0 seconds
Extended Email Publisher is currently disabled in project settings
Finished: FAILURE 

This impacts agents on Windows, Linux, and macOS whether launched via JNLP or SSH.

 

The remoting logs don't indicate any issues:

Feb 01, 2023 3:04:31 PM org.apache.sshd.common.util.security.AbstractSecurityProviderRegistrar getOrCreateProvider
INFO: getOrCreateProvider(BC) created instance of org.bouncycastle.jce.provider.BouncyCastleProvider
Feb 01, 2023 3:04:31 PM org.apache.sshd.common.util.security.AbstractSecurityProviderRegistrar getOrCreateProvider
INFO: getOrCreateProvider(EdDSA) created instance of net.i2p.crypto.eddsa.EdDSASecurityProvider 

Enabling debugging of jar sigignature validation (-Djavax.net.debug=jar) indicates no issues.  Jobs run successfully on the built-in executors on the master instance.  They also run successfully if the SSH key is decrypted and loaded into Jenkins with no password.

 

I theorize that this is somehow an artifact of the migration from the prior SSH plugin to the newer Mina-based solution.  Not having any familiarity with the plumbing responsible for shipping BC to the nodes and loading it, nor the wiring of BC and Mina, this is only a theory.


Originally reported by mattkosem, imported from: Cannot execute builds using SVN SCM with SVN+SSH using private key with passphrase
  • assignee: alobato
  • status: Open
  • priority: Major
  • component(s): ssh-credentials-plugin, ssh-plugin
  • resolution: Unresolved
  • votes: 0
  • watchers: 1
  • imported: 20251215-193512
Raw content of original issue

I upgraded one of my build enviromments from Jenkins 2.346.2 to Jenkins 2.375.2 today, and upgraded all plugins installed on it to the latest versions (including, but not limited to bouncycastle-api and subversion).  After doing so, all jobs running on agent nodes begun failing with the following error:

Started by user <REDACTED>
Running as <REDACTED>
[EnvInject] - Loading node environment variables.
Building remotely on <REDACTED> in workspace <REDACTED>
Cleaning up <REDACTED>
Updating svn+ssh://<REDACTED> at revision '<REDACTED>'
Using sole credentials <REDACTED> in realm ‘<REDACTED>’
FATAL: java.lang.ExceptionInInitializerError
java.lang.NullPointerException
    at java.base/javax.crypto.ProviderVerifier.verify(ProviderVerifier.java:122)
    at java.base/javax.crypto.JceSecurity.verifyProvider(JceSecurity.java:191)
    at java.base/javax.crypto.JceSecurity.getVerificationResult(JceSecurity.java:217)
    at java.base/javax.crypto.Cipher.getInstance(Cipher.java:688)
Caused: java.lang.SecurityException: JCE cannot authenticate the provider BC
    at java.base/javax.crypto.Cipher.getInstance(Cipher.java:692)
    at java.base/javax.crypto.Cipher.getInstance(Cipher.java:623)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.apache.sshd.common.util.security.SecurityEntityFactory$2.getInstance(SecurityEntityFactory.java:130)
    at org.apache.sshd.common.util.security.SecurityUtils.getCipher(SecurityUtils.java:748)
    at org.apache.sshd.common.config.keys.loader.AESPrivateKeyObfuscator$LazyKeyLengthsHolder.detectSupportedKeySizes(AESPrivateKeyObfuscator.java:134)
    at org.apache.sshd.common.config.keys.loader.AESPrivateKeyObfuscator$LazyKeyLengthsHolder.<clinit>(AESPrivateKeyObfuscator.java:121)
Caused: java.lang.ExceptionInInitializerError
    at org.apache.sshd.common.config.keys.loader.AESPrivateKeyObfuscator.getAvailableKeyLengths(AESPrivateKeyObfuscator.java:110)
    at org.apache.sshd.common.config.keys.loader.AESPrivateKeyObfuscator.getSupportedKeySizes(AESPrivateKeyObfuscator.java:51)
    at org.apache.sshd.common.config.keys.loader.AESPrivateKeyObfuscator.resolveKeyLength(AESPrivateKeyObfuscator.java:87)
    at org.apache.sshd.common.config.keys.loader.AESPrivateKeyObfuscator.applyPrivateKeyCipher(AESPrivateKeyObfuscator.java:58)
    at org.apache.sshd.common.config.keys.loader.pem.AbstractPEMResourceKeyPairParser.applyPrivateKeyCipher(AbstractPEMResourceKeyPairParser.java:227)
    at org.apache.sshd.common.config.keys.loader.pem.AbstractPEMResourceKeyPairParser.extractKeyPairs(AbstractPEMResourceKeyPairParser.java:170)
    at org.apache.sshd.common.config.keys.loader.AbstractKeyPairResourceParser.loadKeyPairs(AbstractKeyPairResourceParser.java:117)
    at org.apache.sshd.common.config.keys.loader.KeyPairResourceParser$2.loadKeyPairs(KeyPairResourceParser.java:166)
    at org.apache.sshd.common.config.keys.loader.pem.PEMResourceParserUtils$1.loadKeyPairs(PEMResourceParserUtils.java:53)
    at org.apache.sshd.common.config.keys.loader.KeyPairResourceParser$2.loadKeyPairs(KeyPairResourceParser.java:166)
    at org.apache.sshd.common.config.keys.loader.KeyPairResourceLoader.loadKeyPairs(KeyPairResourceLoader.java:157)
    at org.apache.sshd.common.config.keys.loader.KeyPairResourceLoader.loadKeyPairs(KeyPairResourceLoader.java:148)
    at org.apache.sshd.common.config.keys.loader.KeyPairResourceLoader.loadKeyPairs(KeyPairResourceLoader.java:139)
    at org.apache.sshd.common.config.keys.loader.KeyPairResourceLoader.loadKeyPairs(KeyPairResourceLoader.java:130)
    at org.apache.sshd.common.util.security.SecurityUtils.loadKeyPairIdentities(SecurityUtils.java:522)
    at org.tmatesoft.svn.core.internal.io.svn.SVNSSHPrivateKeyUtil.isValidPrivateKey(SVNSSHPrivateKeyUtil.java:99)
    at org.tmatesoft.svn.core.internal.io.svn.SVNSSHConnector.open(SVNSSHConnector.java:102)
    at org.tmatesoft.svn.core.internal.io.svn.SVNConnection.open(SVNConnection.java:80)
    at org.tmatesoft.svn.core.internal.io.svn.SVNRepositoryImpl.openConnection(SVNRepositoryImpl.java:1282)
    at org.tmatesoft.svn.core.internal.io.svn.SVNRepositoryImpl.testConnection(SVNRepositoryImpl.java:100)
    at org.tmatesoft.svn.core.io.SVNRepository.getRepositoryUUID(SVNRepository.java:268)
    at org.tmatesoft.svn.core.internal.wc2.SvnRepositoryAccess.createRepository(SvnRepositoryAccess.java:103)
    at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgRepositoryAccess.createRepository(SvnNgRepositoryAccess.java:211)
    at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.updateInternal(SvnNgAbstractUpdate.java:210)
    at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.update(SvnNgAbstractUpdate.java:115)
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to <REDACTED>
        at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1784)
        at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
        at hudson.remoting.Channel.call(Channel.java:1000)
        at hudson.FilePath.act(FilePath.java:1186)
        at hudson.FilePath.act(FilePath.java:1175)
        at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:970)
        at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:892)
        at hudson.scm.SCM.checkout(SCM.java:540)
        at hudson.model.AbstractProject.checkout(AbstractProject.java:1241)
        at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:649)
        at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:85)
        at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:521)
        at hudson.model.Run.execute(Run.java:1900)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
        at hudson.model.ResourceController.execute(ResourceController.java:107)
        at hudson.model.Executor.run(Executor.java:449)
Caused: java.lang.RuntimeException
    at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.throwThrowable(SvnNgAbstractUpdate.java:918)
    at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.update(SvnNgAbstractUpdate.java:125)
    at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgUpdate.run(SvnNgUpdate.java:40)
    at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgUpdate.run(SvnNgUpdate.java:18)
    at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgOperationRunner.run(SvnNgOperationRunner.java:20)
    at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:21)
    at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1239)
    at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294)
    at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:311)
    at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:291)
    at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:387)
    at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:165)
    at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:168)
    at hudson.scm.SubversionSCM$CheckOutUpdateTask.perform(SubversionSCM.java:1086)
    at hudson.scm.SubversionSCM$CheckOutUpdateTask.run(SubversionSCM.java:1067)
    at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:1037)
    at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:1020)
    at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3492)
    at hudson.remoting.UserRequest.perform(UserRequest.java:211)
    at hudson.remoting.UserRequest.perform(UserRequest.java:54)
    at hudson.remoting.Request$2.run(Request.java:377)
    at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:834)
Archiving artifacts
Started calculate disk usage of build
Finished Calculation of disk usage of build in 0 seconds
Started calculate disk usage of workspace
Finished Calculation of disk usage of workspace in 0 seconds
Extended Email Publisher is currently disabled in project settings
Finished: FAILURE 

This impacts agents on Windows, Linux, and macOS whether launched via JNLP or SSH.

 

The remoting logs don't indicate any issues:

Feb 01, 2023 3:04:31 PM org.apache.sshd.common.util.security.AbstractSecurityProviderRegistrar getOrCreateProvider
INFO: getOrCreateProvider(BC) created instance of org.bouncycastle.jce.provider.BouncyCastleProvider
Feb 01, 2023 3:04:31 PM org.apache.sshd.common.util.security.AbstractSecurityProviderRegistrar getOrCreateProvider
INFO: getOrCreateProvider(EdDSA) created instance of net.i2p.crypto.eddsa.EdDSASecurityProvider 

Enabling debugging of jar sigignature validation (-Djavax.net.debug=jar) indicates no issues.  Jobs run successfully on the built-in executors on the master instance.  They also run successfully if the SSH key is decrypted and loaded into Jenkins with no password.

 

I theorize that this is somehow an artifact of the migration from the prior SSH plugin to the newer Mina-based solution.  Not having any familiarity with the plumbing responsible for shipping BC to the nodes and loading it, nor the wiring of BC and Mina, this is only a theory.

environment
Oracle Linux 7.9<br/>
JDK 11.0.18<br/>
Jenkins 2.375.2<br/>
Credentials plugin 1214.v1de940103927<br/>
SSH Credentials plugin 305.v8f4381501156<br/>
SSH plugin 2.6.1<br/>
bouncycastle-api plugin 2.27

Metadata

Metadata

Assignees

No one assigned

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions