Runtime CA certifications not added to JVM truststore

[ldauvilaire]

The CA certificates provides at runtime via the /platform/bindings folder are not added to the JVM truststore at the startup of the image. Only the CA certificates provided at build time are embedded and added.

Expected Behavior

The CA certificates provided at runtime via the /platform/bindings folder (containing the .pem file and the type file)
should be added to the JVM truststore as well as to the system truststore.

Current Behavior

The JVM truststore contains only the default JRE CA certificates plus the CA certificates embedded at build time.

Possible Solution

At runtime, launch a copy (or a link) of the .pem files from the bindings folder to the ca-certificate buildpack folder (/layers/paketo-buildpacks_ca-certificates/ca-certificates/ca-certificates).
Can also do the same into the folder /etc/pki/ca-trust/source/anchors for the system truststore
and then run the linux command : update-ca-trust

Steps to Reproduce

1. Build an image with Java with options :

• BP_EMBED_CERTS set to true
• SERVICE_BINDING_ROOT set to /bindings
• mount a volume on /bindings containing
  • folder ca-certificates containing :
    • file : ca-certificate-1.pem
    • file : type (containing text ca-certificates)

2. Run the generated image with :

• SERVICE_BINDING_ROOT set to /mnt/bindings
• mount a volume on /mnt/bindings containing
  • folder ca-certificates containing :
    • file : ca-certificate-2.pem
    • file : type (containing text ca-certificates)

The JVM truststore (cacerts file) should contain both ca-certificate-1 and ca-certificate-2
but only contains ca-certificate-1.

Motivations

The addition of a truststore at build time is needed (to add the corportate ca certificates) for external communications.

The addition of a truststore at runtime is needed because the kubernetes platform (openshift) provides dynamically the .pem file for re-encryption of the https communications (internal communications inside the cluster).
The openshift ca certificate is re-generated regularly
and just a restart of a pod is needed (not a rebuild of the image).

[dmikusa]

What you're describing should work. Do you have steps to reproduce the problem? What commands are you running & what's the full build & run output? Also, do you have a test app you're using? Thanks

[ldauvilaire]

I have made a sample project to demonstrate that.
here is the URL : paketo-test
extract of the runtime output :

12:33:37.000 paketo> Adding 0 container CA certificates to JVM truststore
12:33:37.000 paketo> Added 1 additional CA certificate(s) to system truststore

it seems that the ca-certificate caadmin-netskope-com.pem is well detected and added to system truststore but not to JVM truststore

[dmikusa]

You need to flip the order of your buildpacks around:

                                <buildpacks>
                                    <buildpack>paketo-buildpacks/bellsoft-liberica</buildpack>
                                    <buildpack>paketo-buildpacks/ca-certificates</buildpack>
                                    <buildpack>paketo-buildpacks/java</buildpack>
                                    <buildpack>paketo-buildpacks/image-labels</buildpack>
                                    <buildpack>paketo-buildpacks/spring-boot</buildpack>
                                </buildpacks>

You need to have ca-certificates first.

                                <buildpacks>
                                    <buildpack>paketo-buildpacks/ca-certificates</buildpack>
                                    <buildpack>paketo-buildpacks/bellsoft-liberica</buildpack>
                                    <buildpack>paketo-buildpacks/java</buildpack>
                                    <buildpack>paketo-buildpacks/image-labels</buildpack>
                                    <buildpack>paketo-buildpacks/spring-boot</buildpack>
                                </buildpacks>

https://paketo.io/docs/howto/java/#use-an-alternative-jvm

There is one drawback to this approach. When using the method above to specify an alternative JVM vendor buildpack, this alternate buildpack ends up running before the CA certs buildpack and therefore traffic from the alternate JVM vendor buildpack won’t trust any additional CA certs.

The helpers that run at runtime before your app to load certs will run in the same order as buildpacks. Normally, you should see it update the system truststore first, then JVM truststore.