Backport of packaging in stable-2.541 for LTS 2.541.1

.gitignore

@@ -9,6 +9,10 @@ generated
 .iml
 .idea/
 *.orig
+*.war
+*.war.asc
+*.sha256
 
 venv/
 pkg.jenkins.io/
+jenkins.io.key

Makefile

@@ -56,11 +56,3 @@ ${CLI}:
 	@mkdir ${TARGET} || true
 	wget -O $@.tmp ${JENKINS_URL}jnlpJars/jenkins-cli.jar
 	mv $@.tmp $@
-
-
-
-test.local.setup:
-	# start a test Apache server that acts as package server
-	# we'll refer to this as 'test.pkg.jenkins.io'
-	@mkdir -p ${TESTDIR} || true
-	docker run --rm -t -i -p 9200:80 -v ${TESTDIR}:/var/www/html fedora/apache

README.md

@@ -7,7 +7,24 @@ The following platforms are currently supported:
   * RedHat/CentOS/openSUSE RPM: `rpm/`
   * Debian/Ubuntu DEB: `deb/`
 
-# Pre-requisites
+## Pre-requisites
+
+### Easy Method
+
+If you are able to run Docker containers with `docker compose` (usually Docker Desktop is enough),
+then you can use the same environment as the Jenkins infrastructure production:
+
+```bash
+# Start the setup (2 containers: one for packaging and another for a webserver)
+docker compose up -d
+
+# Spawn an interactive terminal in the packaging environment (with all dependencies)
+docker compose exec packaging bash
+# You can run all the `make` command and even `prep.sh` script - see sections below
+```
+
+### Container-less Method (aka "The Hard Way")
+
 Running the main package script requires a Linux environment (currently Ubuntu, see [JENKINS-27744](https://issues.jenkins-ci.org/browse/JENKINS-27744).)
 Run `make setup` to install (most of the) necessary tools.  Alternatively you can manually install the following onto a base install of Ubuntu:
 * make
@@ -24,21 +41,11 @@ Run `make setup` to install (most of the) necessary tools.  Alternatively you ca
 
 You also need a Jenkins instance with [dist-fork plugin](https://wiki.jenkins-ci.org/display/JENKINS/DistFork+Plugin)
 installed. URL of this Jenkins can be fed into `make` via the `JENKINS_URL` variable.
-This Jenkins needs to have a Windows build agent that has [WiX Toolset](http://wixtoolset.org/) (currently 3.5), msbuild, [cygwin](https://www.cygwin.com/) and .net 2.0. This build agent is used to build MSI packages, which
-can be only built on Windows.
-
-You'll also need a `jenkins.war` file that you are packaging, which comes from the release process.
-The location of this file is set via the `WAR` variable.
-
-Remark:
+This Jenkins needs to have a Windows build agent that has [WiX Toolset](http://wixtoolset.org/) (currently 3.5), msbuild, [cygwin](https://www.cygwin.com/) and .net 2.0.
+This build agent is used to build MSI packages, which can be only built on Windows.
 
-A docker image is available to run following script
+## Generating packages
 
-[![logo](https://img.shields.io/docker/pulls/jenkinsciinfra/packaging?label=jenkinsciinfra%2Fpackaging&logo=docker&logoColor=white)](https://hub.docker.com/r/jenkinsciinfra/packaging)
-
-Run `docker-compose run --rm packaging bash` to get a shell in the official Docker image for this repository.
-
-# Generating packages
 Run `./prep.sh` to perform the preparatory actions of downloading the WAR and importing the GPG key.
 Run `make package` to build all the native packages.
 At minimum, you have to specify the `WAR` variable that points to the war file to be packaged and the `BRAND` variable that points to the branding file for licensing, artifact names, and package descriptions.
@@ -53,7 +60,7 @@ make package BRAND=./branding/jenkins.mk BUILDENV=./env/test.mk CREDENTIAL=./cre
 Packages will be placed into `target/` directory.
 See the definition of the `package` goal for how to build individual packages selectively.
 
-# Running functional tests
+## Running functional tests
 
 The functional tests require Python 3 and Docker.
 Having built the packages as described above, run the functional tests with:
@@ -66,26 +73,27 @@ molecule test
 deactivate
 ```
 
-# Publishing packages
+## Publishing packages
+
 This repository contains scripts for copying packages over to a remote web server to publish them.
 Run `make publish` to publish all native packages.
 
-See the definition of the `publish` goal for individual package publishment.
+See the definition of the `publish` goal for individual package publication.
 
 ## Running local tests
+
 These tests install packages from a web server where they are published. So if you want to
 run tests prior to publishing them, you need to create a temporary web server that you can mess up.
 
 The default branding & environment (`branding/test.mk` and `env/test.mk`) are designed to support
 this scenario. To make local testing work, you also need to have `/etc/hosts` entry that maps
-`test.pkg.jenkins.io` hostname to `127.0.0.1`, and your computer has to be running ssh that
-lets you login as you.
+`test.pkg.jenkins.io` hostname to `127.0.0.1`.
 
-Once you verified the above prerequisites, open another terminal and run `make test.local.setup`
-This will run a docker container that acts as your throw-away package web server. When done, Ctrl+C
-to kill it.
+Once you verified the above prerequisites, open another terminal and run `docker compose up -d pkgserver`
+This will run a docker container that acts as your throw-away package web server.
+
+## Branding
 
-# Branding
 `branding/` directory contains `*.mk` files that control the branding of the generated packages.
 It also include text files which are used for large, branded text blocks (license and descriptions).
 Specify the branding file via the `BRAND` variable.
@@ -95,19 +103,18 @@ See [branding readme](branding/README.md) for more details. In the rest of the p
 these branding parameters are referenced via `@@NAME@@` and get substituted by `bin/branding.py`.
 To escape a string normally like @@VALUE@@, add an additional two @@ symbols as a prefix: @@@@VALUE@@.
 
-# Environment
+## Environment
+
 `env/` directory contains `*.mk` files that control the environment into which
 you publish packages.  Specify the environment file via the `BUILDENV` variable.
 
 You can create your own environment definition to customize the package generation process.
 See [environment readme](env/README.md) for more details.
 
-# Credentials
+## Credentials
+
 `credentials/` directory contains `test.mk` file that controls the locations of code-signing keys,
 their passwords, and certificates. Specify the credentials file via the `CREDENTIAL` variable.
 
 For production use, you need to create your own credentials file. See [credentials readme](credentials/README.md)
 for more details.
-
-# TODO (mostly note to myself)
-* Split resource templates to enable customization

bin/indexGenerator.py

@@ -60,6 +60,7 @@ def __init__(self, argv):
         self.product_name = os.getenv("PRODUCTNAME", "Jenkins")
         self.distribution = os.getenv("OS_FAMILY", "debian")
         self.gpg_pub_key_info_file = os.getenv("GPGPUBKEYINFO", ".")
+        self.gpg_public_key_filename = os.getenv("GPG_PUBLIC_KEY_FILENAME", "jenkins.io.key")
         self.target_directory = "./target/" + self.distribution
 
         try:
@@ -107,6 +108,7 @@ def show_information(self):
         print("Root header generated: " + self.root_header)
         print("Root footer generated: " + self.root_footer)
         print("GPG Key Info File: " + self.gpg_pub_key_info_file)
+        print("GPG Public Key Filename: " + self.gpg_public_key_filename)
 
     def generate_root_header(self):
 
@@ -169,6 +171,7 @@ def generate_repository_header(self):
             "releaseline": self.releaseline,
             "web_url": self.web_url,
             "pub_key_info": self.fetch_pubkeyinfo(),
+            "gpg_public_key_filename": self.gpg_public_key_filename,
         }
 
         env = jinja2.Environment(
@@ -192,6 +195,7 @@ def generate_repository_index(self):
             "releaseline": self.releaseline,
             "web_url": self.web_url,
             "pub_key_info": self.fetch_pubkeyinfo(),
+            "gpg_public_key_filename": self.gpg_public_key_filename,
         }
 
         env = jinja2.Environment(

deb/publish/publish.sh

@@ -8,6 +8,7 @@ set -euxo pipefail
 : "${DEBDIR:? Require where to put binary files}"
 : "${DEB_WEBDIR:? Require where to put repository index and other web contents}"
 : "${DEB_URL:? Require Debian repository Url}"
+: "${GPG_PUBLIC_KEY_FILENAME:="${ORGANIZATION}.key"}"
 
 # $$ Contains current pid
 D="$AGENT_WORKDIR/$$"
@@ -20,14 +21,14 @@ function clean() {
 
 # Generate and publish site content
 function generateSite() {
-	cp -R "$bin/contents/." "$D/contents"
-
-	gpg --export -a --output "$D/contents/${ORGANIZATION}.key" "${GPG_KEYNAME}"
-	gpg --import-options show-only --import "$D/contents/${ORGANIZATION}.key" >"$D/contents/${ORGANIZATION}.key.info"
+	local gpg_publickey_file="$D/contents/${GPG_PUBLIC_KEY_FILENAME}"
+	local gpg_publickey_info_file="$D/contents/${GPG_PUBLIC_KEY_FILENAME}.info"
+	gpg --export -a --output "${gpg_publickey_file}" "${GPG_KEYNAME}"
+	gpg --import-options show-only --import "${gpg_publickey_file}" > "${gpg_publickey_info_file}"
 
 	"$BASE/bin/indexGenerator.py" \
 		--distribution debian \
-		--gpg-key-info-file "${D}/contents/${ORGANIZATION}.key.info" \
+		--gpg-key-info-file "${gpg_publickey_info_file}" \
 		--targetDir "$D/html"
 
 	"$BASE/bin/branding.py" "$D"
@@ -59,7 +60,7 @@ function generateSite() {
 }
 
 function init() {
-	mkdir -p "$D/binary" "$D/contents" "$D/html" \
+	mkdir -p "$D/binary" "$D/contents" "$D/html" "$D/contents/binary" \
 		"$DEBDIR" `# where to put binary files` \
 		"$DEB_WEBDIR" `# where to put repository index and other web contents`
 }
@@ -87,14 +88,10 @@ function uploadPackageSite() {
 }
 
 function uploadHtmlSite() {
-	# Html file need to be located in the binary directory
 	rsync --archive \
 		--verbose \
 		--progress \
-		--include "HEADER.html" \
-		--include "FOOTER.html" \
-		--exclude "*" \
-		"$D/html/" "$DEBDIR/"
+		"$D/html/" "$DEB_WEBDIR/"
 }
 
 function show() {
@@ -103,6 +100,7 @@ function show() {
 	echo "DEBDIR: $DEBDIR"
 	echo "DEB_WEBDIR: $DEB_WEBDIR"
 	echo "GPG_KEYNAME: $GPG_KEYNAME"
+	echo "GPG_PUBLIC_KEY_FILENAME: $GPG_PUBLIC_KEY_FILENAME"
 	echo "---"
 }
 

docker-compose.yaml

@@ -1,12 +1,11 @@
-# docker exec -i -t packaging_packaging_1 gpg --import --batch credentials/sandbox.gpg
-version: '3'
-volumes:
-  sshd:
-  pkgserver:
+# docker compose exec packaging gpg --import --batch credentials/sandbox.gpg
 services:
   packaging:
     image: jenkinsciinfra/packaging:latest
-    command: "sleep 99d"
+    platform: linux/amd64
+    entrypoint:
+      - "sleep"
+      - "99d"
     environment:
     - "BUILDENV=/srv/releases/jenkins/env/test.mk"
     - "BRANDING_DIR=/srv/releases/jenkins/branding"
@@ -19,25 +18,13 @@ services:
     - "MSI=/srv/releases/jenkins/jenkins.msi"
     - "RELEASELINE=-experimental"
     volumes:
-      - ".:/srv/releases/jenkins:z"
-      - "./credentials/ssh:/home/jenkins/.ssh:z"
-    working_dir: "/srv/releases/jenkins"
-
-  remote:
-    image: jenkinsciinfra/packaging:latest
-    user: root # In order to start, sshd needs to be root
-    command: "/usr/sbin/sshd -D"
-    ports:
-      - "2222:22"
-    volumes:
-      - "./credentials/ssh/id_rsa.pub:/home/jenkins/.ssh/authorized_keys:ro"
-      - sshd:/run/sshd
-      - "pkgserver:/srv/releases/jenkins"
+      - .:/srv/releases/jenkins:z
+    working_dir: /srv/releases/jenkins
 
   pkgserver:
-    image: httpd
+    image: nginx
     ports:
       - "80:80"
     volumes:
-      - "./pkgConfig/httpd.conf:/usr/local/apache2/conf/httpd.conf"
-      - "./target:/usr/local/apache2/htdocs"
+      - ./target:/usr/share/nginx/html:ro
+      - ./pkgserver/nginx-default-vhost.conf:/etc/nginx/conf.d/default.conf:ro

env/test.mk

@@ -3,7 +3,7 @@
 #
 
 # where to put binary files
-export TESTDIR=$(realpath .)/pkg.jenkins.io
+export TESTDIR=$(realpath .)/target
 export WARDIR=${TESTDIR}/war${RELEASELINE}
 export MSIDIR=${TESTDIR}/windows${RELEASELINE}
 export DEBDIR=${TESTDIR}/debian${RELEASELINE}/binary

msi/publish/publish.sh

@@ -30,23 +30,16 @@ function uploadPackage() {
 	sha256sum "${MSI}" >"${MSI_SHASUM}"
 	cat "${MSI_SHASUM}"
 
-	# Update the symlink to point to most recent Windows build
-	#
-	# Remove anything in current directory named 'latest'
-	# This is a safety measure just in case something was left there previously
-	rm -rf latest
-
-	# Create a local symlink pointing to the MSI file in the VERSION directory.
-	# Don't need VERSION directory or MSI locally, just the unresolved symlink.
-	# The jenkins.io page downloads http://mirrors.jenkins-ci.org/windows/latest
-	# and assumes it points to the most recent MSI file.
-	ln -s "${VERSION}/$(basename "$MSI")" latest
-
-	# Local
 	rsync --archive \
 		--verbose \
 		--progress \
-		"${MSI}" "${MSI_SHASUM}" latest "${MSIDIR}/${VERSION}/"
+		"${MSI}" "${MSI_SHASUM}" "${MSIDIR}/${VERSION}/"
+
+	# Update the symlink to point to most recent MSI directory
+	pushd "${MSIDIR}"
+	rm -rf latest # This is a safety measure just in case something was left there previously
+	ln -s "${VERSION}" latest
+	popd
 }
 
 # The site need to be located in the binary directory