Merge pull request #207 from mendix/address-user-feedback

Address user feedback and prepare new major version release

Author: zlogic

README.md

@@ -4,7 +4,7 @@
 
 The Mendix Buildpack for Docker (aka docker-mendix-buildpack) is an **example project** you can use to build and run your Mendix Application in a [Docker](https://www.docker.com/) container.
 
-**⚠️ Warning** If your pipeline is based on Docker Buildpack V4 or an earlier version, see the [upgrading from Docker Buildpack v4](upgrading-from-v4.md) document. To use Docker Buildpack v5, some changes will be required in your build process.
+**⚠️ Warning** If your pipeline is based on Docker Buildpack v5 or an earlier version, see the [upgrading from Docker Buildpack v5](upgrading-from-v6.md) document. To use Docker Buildpack v6, some changes will be required in your build process.
 
 For a Kubernetes native solution to run Mendix apps, see [Mendix for Private Cloud](https://www.mendix.com/evaluation-guide/app-lifecycle/mendix-for-private-cloud/).
 
@@ -42,8 +42,14 @@ This project is a goto reference for the following scenarios :
 * Docker 20.10 (Installation [here](https://docs.docker.com/engine/installation/))
   * Earlier Docker versions are no longer compatible because they don't support multistage builds.
     To use Docker versions below 20.10, download an earlier Mendix Docker Buildpack release, such as [v2.3.2](https://github.com/mendix/docker-mendix-buildpack/releases/tag/v2.3.2)
-* Python 3.8
-* For preparing, a local installation of `curl`
+  * Alternatively, Podman version 5 or later
+* Python 3, version 3.8 or later
+  * No additional dependencies are needed
+  * In Windows, only Python running in Windows Subsystem for Linux is supported
+* A UNIX-like operating system, such as Linux or macOS, or Windows Subsystem for Linux
+* An x86-64 (AMD64) based CPU
+  * ARM64 CPUs are not fully supported
+* For running the example tests, a local installation of `curl`
 * For local testing, make sure you can run the [docker-compose command](https://docs.docker.com/compose/install/)
 * A Mendix app based on Mendix 8 or a later version
 
@@ -52,7 +58,7 @@ This project is a goto reference for the following scenarios :
 ### Preparation: rootfs
 
 To save build time, the build pack needs a prebuilt rootfs containing the base OS and additional packages.
-This rootfs is based on [Red Hat Universal Base Image 8 minimal](https://developers.redhat.com/articles/ubi-faq) image.
+This rootfs is based on [Red Hat Universal Base Image 9 minimal](https://developers.redhat.com/articles/ubi-faq) image.
 
 To build the rootfs, run the following commands
 
@@ -83,15 +89,15 @@ docker push <app-root-fs-image-tag>
 
 When building the the `rootfs-builder.dockerfile` file, you can provide the following additional arguments:
 
-- **CF_BUILDPACK** is a version of CloudFoundry buildpack. Defaults to `v5.0.16`. For stable pipelines, it's recommended to use a fixed **v5.0.16** version. Other Cloud Foundry buildpacks might not work with this version of Docker Buildpack.
+- **CF_BUILDPACK** is a version of CloudFoundry buildpack. Defaults to `v5.0.23`. For stable pipelines, it's recommended to use a fixed **v5.0.23** version. Other Cloud Foundry buildpacks might not work with this version of Docker Buildpack.
 - **CF_BUILDPACK_URL** specifies the URL where the CF buildpack should be downloaded from (for example, a local mirror). Defaults to `https://github.com/mendix/cf-mendix-buildpack/releases/download/${CF_BUILDPACK}/cf-mendix-buildpack.zip`. Specifying **CF_BUILDPACK_URL** will override the version from **CF_BUILDPACK**.
 - **BUILDPACK_XTRACE** can be used to enable CF Buildpack [debug logging](https://github.com/mendix/cf-mendix-buildpack#logging-and-debugging). Set this variable to `true` to enable debug logging.
 
 ### Compile an MDA
 
 If your app is a source MPK file, an MPR project directory or a compressed MDA file, it needs to be converted or compiled into a format supported by CF Buildpack - an extracted MDA file.
 
-This feature is available in Docker Buildpack version v5.1.0 and later, and is intended to allow building Mendix 10 apps in custom CI/CD pipelines.
+This feature is available in Docker Buildpack version v6.0.0 and later, and is intended to allow building Mendix 8, 9 and 10 apps in custom CI/CD pipelines.
 
 To do this, run:
 
@@ -121,13 +127,12 @@ Before running the container, it is necessary to build the image with your appli
 
 ```
 docker build \
-  --build-arg BUILD_PATH=<mendix-project-location> \
   --tag mendix/mendix-buildpack:v1.2 .
 ```
 
 For build you can provide next arguments:
 
-- **BUILD_PATH** indicates where the application model is located. It is a root directory of an unzipped .MDA or .MPK file. In the latter case, this is the directory where your .MPR file is located. Must be within [build context](https://docs.docker.com/engine/reference/commandline/build/#extended-description). Defaults to `./project`.
+- **BUILD_PATH** indicates where the application model is located. It is a root directory of an unzipped .MDA or .MPK file. In the latter case, this is the directory where your .MPR file is located. Must be within [build context](https://docs.docker.com/engine/reference/commandline/build/#extended-description). Should not be used when using the `build.py` script. Defaults to `./project`.
 - **ROOTFS_IMAGE** is a type of rootfs image. Defaults to `mendix-rootfs:app` (a locally prebuilt image).
 - **BUILDER_ROOTFS_IMAGE** is a type of rootfs image used for downloading the Mendix app dependencies and compiling the Mendix app from source. Defaults to `mendix-rootfs:builder` (a locally prebuilt image).
 - **EXCLUDE_LOGFILTER** will exclude the `mendix-logfilter` binary from the resulting Docker image if set to `true`. Defaults to `true`. Excluding `mendix-logfilter` will reduce the image size and remove a component that's not commonly used; the `LOG_RATELIMIT` environment variable option will be disabled.
@@ -398,9 +403,7 @@ Contributions are welcomed:
 
 ### Build Details
 
-This was built with the following:
-
-* Docker version 20.10
+Docker Buildpack is tested by running a Github Actions pipeline.
 
 ### Versioning
 

build.py

@@ -189,12 +189,15 @@ def parse_version(version):
     return tuple([ int(n) for n in version.split('.') ])
 
 def prepare_destination(destination_path):
-    with os.scandir(destination_path) as entries:
-        for entry in entries:
-            if entry.is_dir() and not entry.is_symlink():
-                shutil.rmtree(entry.path)
-            else:
-                os.remove(entry.path)
+    if os.path.exists(destination_path):
+        with os.scandir(destination_path) as entries:
+            for entry in entries:
+                if entry.is_dir() and not entry.is_symlink():
+                    shutil.rmtree(entry.path)
+                else:
+                    os.remove(entry.path)
+    else:
+        os.makedirs(destination_path, 0o755)
     project_path = os.path.join(destination_path, 'project')
     os.mkdir(project_path, 0o755)
     shutil.copytree('scripts', os.path.join(destination_path, 'scripts'))
@@ -224,14 +227,6 @@ def prepare_mda(source_path, destination_path, artifacts_repository=None):
     else:
         raise Exception('No supported files found in source path')
 
-def build_image(mda_dir):
-    # TODO: build the full image, or just copy MDA into destination?
-    mda_path = mda_dir.name if isinstance(mda_dir, tempfile.TemporaryDirectory) else mda_dir
-    mda_metadata = get_metadata_value(mda_path)
-    mx_version = mda_metadata['RuntimeVersion']
-    java_version = mda_metadata.get('JavaVersion', 11)
-    logging.debug("Detected Mendix {} Java {}".format(mx_version, java_version))
-
 if __name__ == '__main__':
     parser = argparse.ArgumentParser(description='Build a Mendix app')
     parser.add_argument('--source', metavar='source', required=True, nargs='?', type=pathlib.Path, help='Path to source Mendix app (MDA file, MPK file, MPR directory or extracted MDA directory)')
@@ -247,4 +242,3 @@ def build_image(mda_dir):
     except KeyboardInterrupt:
         stop_processes()
         raise
-    # build_image(args.destination)

mxbuild/build

@@ -15,18 +15,28 @@ fi
 
 cd /workdir
 
-if [ -f /workdir/project ]; then
-    JAVA_VERSION=$(cat java-version)
+if [ -f /workdir/project/java-version ]; then
+    echo "Using java-version to select Java version"
+    JAVA_VERSION=$(cat /workdir/project/java-version)
 elif [ -f /opt/mendix/modeler/mx ]; then
-    JAVA_VERSION=$(/opt/mendix/modeler/mx dump-mpr --unit-type 'Settings$ProjectSettings' /workdir/project/"${MPR_FILENAME}" | \
-         jq -r '.units[] | select(.["$Type"]=="Settings$ProjectSettings") | .["settingsParts"][] | select(.["$Type"]=="Settings$RuntimeSettings").javaVersion | if (. == null or . == "null") then "Java11" else . end')
+    if JAVA_VERSION=$(/opt/mendix/modeler/mx show-java-version /workdir/project/"${MPR_FILENAME}"); then
+        echo "Using mx show-java-version to select Java version"
+    elif JAVA_VERSION=$(PROJECT_SETTINGS=`/opt/mendix/modeler/mx dump-mpr --unit-type 'Settings$ProjectSettings' /workdir/project/"${MPR_FILENAME}"` && \
+        echo -n $PROJECT_SETTINGS | \
+        jq -r '.units[] | select(.["$Type"]=="Settings$ProjectSettings") | .["settingsParts"][] | select(.["$Type"]=="Settings$RuntimeSettings").javaVersion | if (. == null or . == "null") then "Java11" else . end'); then
+        echo "Using mx dump-mpr to select Java version"
+    else
+        echo "Unable to detect Java version using mx tool"
+        JAVA_VERSION=11
+    fi
 else
+    echo "mx tool is not available"
     JAVA_VERSION=11
 fi
 
 JAVA_VERSION=$(echo -n $JAVA_VERSION| sed s/\^Java// | head)
 
-echo "Detected Java $JAVA_VERSION"
+echo "Using Java $JAVA_VERSION"
 export JDK_HOME=/etc/alternatives/java_sdk_${JAVA_VERSION}
 
 $MXBUILD_COMMAND \

rootfs-builder.dockerfile

@@ -10,7 +10,7 @@ ENV LANG C.UTF-8
 ENV LC_ALL C.UTF-8
 
 # CF buildpack version
-ARG CF_BUILDPACK=v5.0.20
+ARG CF_BUILDPACK=v5.0.23
 # CF buildpack download URL
 ARG CF_BUILDPACK_URL=https://github.com/mendix/cf-mendix-buildpack/releases/download/${CF_BUILDPACK}/cf-mendix-buildpack.zip
 

upgrading-from-v5.md

@@ -0,0 +1,61 @@
+# Upgrading from Docker Buildpack v5
+
+Docker Buildpack v6 contains a breaking change and might require some changes in your CI/CD pipeline:
+
+Building Mendix projects from source (\*.mpr or \*.mpk files) is now done using a build.py script.
+If your CI/CD pipeline uses Docker Buildpack to build \*.mda files (compiled Mendix apps), no further changes are needed.
+
+If you're upgrading from Docker Buildpack v4 (or an older version), you'll also need to follow the [upgrading from Docker Buildpack v4](upgrading-from-v4.md) instructions.
+
+⚠️ If your current pipeline is failing with an _Only Ubuntu is supported_ error, your pipeline depends on CF Buildpack to build Mendix MPR files, and needs to be updated as described in this document.
+
+## Using the build.py script
+
+Docker Buildpack v6 no longer uses CF Buildpack to compile MPR (or MPK) files - to continue supporting newer versions of Mendix, Java and the base OS.
+Instead, a custom `build.py` script will:
+
+1. Prepare a clean [Docker context](https://docs.docker.com/build/concepts/context/) in the path specified by `--destination`. All files required to build the app image will be copied to this destination. If the directory doesn't exist, the `build.py` script will create it; if the directory is not empty, `build.py` will delete its contents.
+2. Detect the file type of the source path specified by the `--source` arg (an MPK file, an MPR file, an MDA file or an unpacked MDA directory).
+3. If necessary (`--source` specifies project that needs to be compiled)
+   1. Create an image containing [mxbuild](https://docs.mendix.com/refguide/mxbuild/) and its dependencies.
+   2. Run an `mxbuild` in a container, and copy the resulting MDA contents to the destination path specified by `--destination`.
+4. Otherwise (`--source` specifies a path to an MDA file or unpacked MDA directory), `build.sh` will just copy the MDA contents to the destination path specified by `--destination`.
+
+Once the `build.py` script runs successfully, the path specified by `--destination` will contain a Docker context and everything needed to run a `docker build` command.
+
+### Updating an existing pipeline to use build.py
+
+There instructions are provided as a reference, based on a typical pipeline. Your CI/CD pipeline might be different - for support with updating a custom pipeline, please check the [Mendix Support Policy](https://www.mendix.com/evaluation-guide/evaluation-learning/support/).
+
+1. Verify your pipeline image or runner has Python 3.8 available, and uses a UNIX-like operating system (Linux, macOS or Windows Subsystem for Linux).
+2. Locate the `docker build` step in your CI/CD pipeline that builds the app image. This should be the step that builds the Mendix app, and not the rootfs or its dependencies. Any `docker build` commands that build the rootfs should not be changed.
+3. Before the `docker build` step, add the following lines (replacing `<path-to-source>` with the path to the project source, and `<destination-dir>` with an empty/temporary writable path):
+   ```shell
+   ./build.py --source <path-to-source> --destination <destination-dir> build-mda-dir
+   ```
+4. In the `docker build` step:
+    * Remove `--build-arg BUILD_PATH` args.
+    * Remove `-f` and `--file` args specifying a Dockerfile, if they exist.
+    * Update the [Docker context](https://docs.docker.com/build/concepts/context/) path to the `<destination-dir>`.
+
+After the update, your pipeline might look like this:
+
+```shell
+# Preparation steps
+# Downloag Docker Buildpack
+DOCKER_BUILDPACK_VERSION=v6.0.0
+curl -LJ -o - https://github.com/mendix/docker-mendix-buildpack/archive/refs/tags/${DOCKER_BUILDPACK_VERSION}.tar.gz | tar --strip-components=1 -xvz
+# Checkout the Mendix app source
+git clone <mendix-app-git> mendix-app-src
+# Build the Mendix app from mendix-app-src to a temporary location
+./build.py --source mendix-app-src --destination /tmp/docker-buildpack-context build-mda-dir
+# Prepare and push the Docker image, using /tmp/docker-buildpack-context as the Docker context
+docker build --tag example.com/mendix-app:latest /tmp/docker-buildpack-context
+docker push example.com/mendix-app:latest
+# Follow-up steps
+```
+
+# Other changes
+
+Docker Buildpack v6 switched from `ubi8` to `ubi9` images ([Red Hat Universal Base Images](https://developers.redhat.com/articles/ubi-faq) whenever possible.
+Building Mendix 8 and 9 apps still uses `ubi8`, as those versions depend on an older version of Mono that doesn't work in newer operating systems.