Cannot upgrade java version

[brailsmt]

🗣️ Foreword

While working with the latest version of the java cookbook and chef 16, we are unable to upgrade to a later java version. We often need to uplift to a newer java version due to security and bug fixes. We are not yet running in a containerized environment so we need the ability to upgrade java versions even if previous versions were installed. We are also unable to safely remove all old installations of java prior to upgrading.

👻 Brief Description

If java has already been installed on the node in question, then subsequent runs of the sous-chef/java recipe will silently fail to upgrade the java version. We have created a thin wrapper around the java resource to install java. The recipe is essentially this (actual values changed, obviously):

adoptopenjdk_install '8' do
  variant node['some']['config']['java_variant']
  url node['some]['config']['java_url']
  checksum node['some']['config']['java_checksum']
end

If we run chef-client -o "recipe[company_java]", the chef-client run succeeds. If we then override the config and install a new version of java, it fails to upgrade the java version, but the cookbook still succeeds.

🥞 Cookbook version

We are experiencing this with 8.4.0.

👩‍🍳 Chef-Infra Version

16.6.14

🎩 Platform details

Oracle Linux 7.9

Steps To Reproduce

Steps to reproduce the behavior:

1. Use the java resource to install a version of java, for example 'adoptopenjdk', '8', 'hotspot' with this URL: https://github.com/AdoptOpenJDK/openjdk8-upstream-binaries/releases/download/jdk8u272-b10/OpenJDK8U-jdk_x64_linux_8u272b10.tar.gz
2. Attempt to upgrade the java version using the same variant, version, and flavor but using this URL: https://github.com/AdoptOpenJDK/openjdk8-upstream-binaries/releases/download/jdk8u275-b01/OpenJDK8U-jdk_x64_linux_8u275b01.tar.gz

/usr/lib/jvm/java-8-adoptopenjdk-hotspot will contain the jdk8u272-b10 directory, and there will not be a jdk8u275-b01 directory.

🚓 Expected behavior

The expected behavior is that version of java that was installed in step 2 would be the java version installed and linked through /etc/alternatives.

➕ Additional context

Here is a snippet from a chef-client run that was intended to install jdk8u232-b09:

  * adoptopenjdk_install[8] action install[2021-01-15T15:39:01-06:00] INFO: Processing adoptopenjdk_install[8] action install (cloudiam_java::default line 2)

    * adoptopenjdk_linux_install[8] action install[2021-01-15T15:39:01-06:00] INFO: Processing adoptopenjdk_linux_install[8] action install (/var/chef/cache/cookbooks/java/resources/adoptopenjdk_install.rb line 49)

      * directory[/usr/lib/jvm] action create[2021-01-15T15:39:01-06:00] INFO: Processing directory[/usr/lib/jvm] action create (/var/chef/cache/cookbooks/java/resources/adoptopenjdk_linux_install.rb line 57)
 (up to date)
      * remote_file[/var/chef/cache/OpenJDK8U-jdk_x64_linux_hotspot_8u232b09.tar.gz] action create[2021-01-15T15:39:01-06:00] INFO: Processing remote_file[/var/chef/cache/OpenJDK8U-jdk_x64_linux_hotspot_8u232b09.tar.gz] action create (/var/chef/cache/cookbooks/java/resources/adoptopenjdk_linux_install.rb line 64)
 (up to date)
      * archive_file[/var/chef/cache/OpenJDK8U-jdk_x64_linux_hotspot_8u232b09.tar.gz] action extract[2021-01-15T15:39:02-06:00] INFO: Processing archive_file[/var/chef/cache/OpenJDK8U-jdk_x64_linux_hotspot_8u232b09.tar.gz] action extract (/var/chef/cache/cookbooks/java/resources/adoptopenjdk_linux_install.rb line 72)
[2021-01-15T15:39:02-06:00] DEBUG: Not extracting archive as /usr/lib/jvm/java-8-adoptopenjdk-hotspot exists and resource not set to overwrite.

Instead of the application being run with jdk8u232-b09, it was running with the version of java that was previously installed which happened to be jdk8u265-b01. While the version that was previously installed was newer is beside the point and is due to a separate issue in our own cookbooks.

[ramereth]

@brailsmt I'm having a hard time getting this to work on my end. Can you provide some exact code that can replicate this problem?

[jakauppila]

I would venture to guess this is the same problem as #599

[brailsmt]

#599 seems focused on the permissions of the directory. This issue is unrelated to permissions issues. It stems from the fact that the cookbook checks for the existence of /usr/lib/jvm/java-8-adoptopenjdk-hotspot, if that directory exists the cookbook will refuse to extract the java tarball into the directory. The code that introduces the issue is https://github.com/sous-chefs/java/blob/master/resources/adoptopenjdk_linux_install.rb#L53, which is used in archive_file on line 73. The java_home property defaults to /usr/lib/jvm/java-8-adoptopenjdk-hotspot/jdk8u232-b09 (for a java 8, adoptopenjdk, hotspot jvm). The extract_dir is then /usr/lib/jvm/java-8-adoptopenjdk-hotspot/. The archive_file resource will then refuse to extract the tarball to the extract directory if it already exists, regardless of whether /usr/lib/jvm/java-8-adoptopenjdk-hotspot/jdk8u232-b09 exists.

In the above scenario, I could recreate the issue following these steps:

$ mkdir /usr/lib/jvm/java-8-adoptopenjdk-hotspot/
$ chef-client -o 'recipe[java]'

Where the java recipe just contains:

adoptopenjdk_install '8' do
  variant adoptopenjdk
  url 'https://github.com/AdoptOpenJDK/openjdk8-upstream-binaries/releases/download/jdk8u272-b10/OpenJDK8U-jdk_x64_linux_8u272b10.tar.gz'
  checksum '654a0b082be0c6830821f22a9e1de5f9e2feb9705db79e3bb0d8c203d1b12c6a'
end

EDIT: I apologize if there are errors in how to recreate the issue, I will work up a complete example this afternoon or this evening when I have time to more properly test it, instead of trying to recreate it from memory as I did here.

[bkonick]

I think I am hitting this too. Recently upgraded our wrapper from using ~> 8.3.1 to ~> 8.5.0 and I was not seeing the correct (new) tarball be extracted until I removed the /usr/lib/jvm/java-8-adoptopenjdk-hotspot directory. I believe the issue is here. Maybe the overwrite property of archive_file needs to be set to true ?

UPDATE:
Confirmed that is the issue. See debug output below:

* adoptopenjdk_linux_install[8] action install[2021-03-25T17:15:23-04:00] INFO: Processing adoptopenjdk_linux_install[8] action install (cg_java::adoptopenjdk line 17)

    * directory[/usr/lib/jvm] action create[2021-03-25T17:15:23-04:00] INFO: Processing directory[/usr/lib/jvm] action create (/var/chef/cache/cookbooks/java/resources/adoptopenjdk_linux_install.rb line 57)
 (up to date)
    * remote_file[/var/chef/cache/OpenJDK8U-jdk_x64_linux_hotspot_8u282b08.tar.gz] action create[2021-03-25T17:15:23-04:00] INFO: Processing remote_file[/var/chef/cache/OpenJDK8U-jdk_x64_linux_hotspot_8u282b08.tar.gz] action create (/var/chef/cache/cookbooks/java/resources/adoptopenjdk_linux_install.rb line 64)
 (up to date)
    * archive_file[/var/chef/cache/OpenJDK8U-jdk_x64_linux_hotspot_8u282b08.tar.gz] action extract[2021-03-25T17:15:23-04:00] INFO: Processing archive_file[/var/chef/cache/OpenJDK8U-jdk_x64_linux_hotspot_8u282b08.tar.gz] action extract (/var/chef/cache/cookbooks/java/resources/adoptopenjdk_linux_install.rb line 72)
[2021-03-25T17:15:23-04:00] DEBUG: Not extracting archive as /usr/lib/jvm/java-8-adoptopenjdk-hotspot exists and resource not set to overwrite.
 (up to date)

Directory contents:

$ ls -l /usr/lib/jvm/java-8-adoptopenjdk-hotspot/
total 0
drwxr-xr-x 8 root root 176 Jul 28  2020 jdk8u265-b01

[brailsmt]

@bkonick We worked around this issue by modifying java_home in our recipe prior to calling the java resource. We changed it include the java version like this:

jdk_install_base_dir = "/usr/lib/jvm/java-#{java_version}-#{java_flavor}-#{java_variant}"
jdk_install_dir = File.join(jdk_install_base_dir, jdk_version, jdk_version)
node.default['java']['java_home'] = jdk_install_dir

This awkwardly doubles up the version in the directory like this /usr/lib/jvm/java-8-adoptopenjdk-hotspot/jdk8u232-b0/jdk8u232-b0, but it works. I'm not sure that just setting overwrite is correct because the java resource also sets JAVA_HOME for everyone on the system. That will break other applications unless they are all stopped and restarted with the correct, new JAVA_HOME for the java version. On long running servers with multiple applications, that isn't desirable.

[bkonick]

@bkonick We worked around this issue by modifying java_home in our recipe prior to calling the java resource. We changed it include the java version like this:

jdk_install_base_dir = "/usr/lib/jvm/java-#{java_version}-#{java_flavor}-#{java_variant}"
jdk_install_dir = File.join(jdk_install_base_dir, jdk_version, jdk_version)
node.default['java']['java_home'] = jdk_install_dir

This awkwardly doubles up the version in the directory like this /usr/lib/jvm/java-8-adoptopenjdk-hotspot/jdk8u232-b0/jdk8u232-b0, but it works. I'm not sure that just setting overwrite is correct because the java resource also sets JAVA_HOME for everyone on the system. That will break other applications unless they are all stopped and restarted with the correct, new JAVA_HOME for the java version. On long running servers with multiple applications, that isn't desirable.

I am going to do some local testing today to see how the overwrite property behaves. I think in this case the JDK is always unpacked to a subdirectory, so I am hoping that it doesn't actually clear out the directory, but just permits it to have contents extracted to it.

[bkonick]

Interestingly, the overwrite property causes the extract to happen during every run which breaks enforce_idempotency in test-kitchen. I may have to go with that workaround after all @brailsmt

[brailsmt]

This is a limitation of the archive_file resource, and is a regression from previous versions of the java cookbook.

[bkonick]

@bkonick We worked around this issue by modifying java_home in our recipe prior to calling the java resource. We changed it include the java version like this:

jdk_install_base_dir = "/usr/lib/jvm/java-#{java_version}-#{java_flavor}-#{java_variant}"
jdk_install_dir = File.join(jdk_install_base_dir, jdk_version, jdk_version)
node.default['java']['java_home'] = jdk_install_dir

This awkwardly doubles up the version in the directory like this /usr/lib/jvm/java-8-adoptopenjdk-hotspot/jdk8u232-b0/jdk8u232-b0, but it works. I'm not sure that just setting overwrite is correct because the java resource also sets JAVA_HOME for everyone on the system. That will break other applications unless they are all stopped and restarted with the correct, new JAVA_HOME for the java version. On long running servers with multiple applications, that isn't desirable.

@brailsmt do you have a more specific example of how you are implementing this workaround? I was trying to re-create it in our wrapper but was having some issues. What is jdk_version defined as?

[wbuck3]

For me I added the following to the archive_file for tarball with adoptopenjdk. This would make it extract contents.

    overwrite true
    not_if { ::Dir.exist?(new_resource.java_home) }

This stops it from extract again if the java_home exists. It was skipping if the partial path was there.

I added the not_if for archive_file as well to keep it from downloading since I don't backup the file.

[brailsmt]

@bkonick The jdk_version can be seen in the path that I mentioned. It is jdk8u232-b09 and is derived from the URL where the jdk is downloaded. In our case, it is a mirror of https://github.com/AdoptOpenJDK/openjdk8-binaries/releases that we host internally. In this specific example that URL is, and the jdk_version is the path directly preceding the tar.gz:
https://internalrepo.net/github-releases/AdoptOpenJDK/openjdk8-binaries/releases/download/jdk8u232-b09/OpenJDK8U-jdk_x64_linux_hotspot_8u232b09.tar.gz

[damacus]

Instead of overwriting and possibly halting running programs we probably need to place AdoptOpenJDK in sub-folders and have alternatives take care of it for us. That way if files are removed during a new patch they're not left behind.