The generate-reprepro-codename script is too restrictive

[nuxwin]

Good morning,

Right now, when the generate-reprepro-codename get called by the underlying build-and-provide-package script, nothing happen if the configuration file for the given REPOSITORY/codename already exists. The script is just existing without further care... That is not really a good behavior in my eyes. Imagine the following scenario:

We are setting a Jenkins job with matrix for specific architectures such as amd64 and i386. By default, no ARCHITECTURES environment variable is exported and so, the generate-reprepro-codename script generate repository configuration file using hardcoded architectures: "amd64 i386 source". At this stage, that fine. Not exporting the ARCHITECTURES environment variable is OK as long as our matrix is targeting default architectures as hardcoded in the generate-reprepro-codename script.

However, problem arise later on when we want add further architectures to our matrix such as arm64. Currently when doing so, the script is existing without further care because the configuration file for the REPOSITORY/Codename already exists. Furthermore, exporting the ARCHITECTURES environment variable with value such as "amd64 i386 arm64 source" doesn't change anything. The build therefore result in a failure as the incoming fails to be processed due to unlisted architecture (here arm64).

04:04:56 + reprepro -b /srv/repository/php -v --waitforlock 1000 --ignore=wrongdistribution processincoming stretch libgd2_2.2.5-5+0~20181105025823.9+stretch~1.gbp911a4a_arm64.changes
04:04:56 'libgd2_2.2.5-5+0~20181105025823.9+stretch~1.gbp911a4a_arm64.changes' contains 'libgd-dev_2.2.5-5+0~20181105025823.9+stretch~1.gbp911a4a_arm64.deb' not matching an valid architecture in any distribution known!
04:04:56 There have been errors!
04:04:56 + local RC=255
04:04:56 + cd /var/lib/jenkins/jobs/php_libgd2-binaries/workspace/architecture/arm64/distribution/stretch
04:04:56 + '[' 255 -ne 0 ']'
04:04:56 + bailout 1 'Error: Failed to execute processincoming for release stretch.'
04:04:56 + '[' -n 1 ']'
04:04:56 + EXIT=1

Expected because the arm64 architecture isn't listed in the repository configuration file:

root@jenkins:/srv/repository# cat php/conf/distributions 

Codename: stretch
Architectures: amd64 i386 source
Components: main
UDebComponents: main
Tracking: minimal
SignWith: 6AC2D04029F007823416F4B7CC6BF829B212C177

Of course we can always edit the configuration file manually but...

My thinking is that the generate-reprepro-codename script should be able to update an already existing repository configuration file according the exported environment variables and also, according the current build parameters.

Regarding the current build parameters

For any build, and if the repository configuration file already exists, the architecture for which the package is being build should be added into the repository configuration file if that isn't already the case. I don't see any other build parameter to handle in that context.

Regarding the environment variables

The script should check whether or not values of exported environment variables remain same and if that not the case, update the repository configuration file with new value.

Thank you.

[mika]

tl;dr: it's a tricky problem and I don't see an easy way how to solve this in a generic way within j-d-g right now

I agree that this would be nice to have, though adjusting reprepro repositories isn't really straight forward and quite error-prone. :-/ For example it complains (IIRC) if there are architectures in reprepro's internal database which aren't listed in the Architectures file of conf/distributions and you'd have to manually invoke reprepro clearvanished or so. For example, when you have an existing repository for let's say amd64 i386 arm64 but then no longer build arm64 and we would modify conf/distributions on the fly then reprepro might fail hard on you. Rewriting the repository in a generic way by our j-d-g scripts isn't trivial, so I'm really reluctant to doing that.

From my PoV there are two approaches from here, until someone comes up with a bullet-proof way¹ how to handle this from within generate-reprepro-codename:

• set ARCHITECTURES by default to all the architectures you (might) care about, then it will be used when generate-reprepro-codename creates a new repository and everything you need is there already (we could do this by default in generate-reprepro-codename, but this might 1) add additional overhead to the database + repositories, 2) confuse users and 3) we'd have to keep track of available/supported architectures; AFAICT the majority of j-d-g users don't care for architectures other than amd64 + i386 though)

• handle this in your own build scripts, you can easily invoke your own scripts before/after generate-reprepro-codename invocation (or just call your own script instead of generate-reprepro-codename)

¹ It could be enough to always just extend the Architectures list and never reduce it again. This might be fine enough as well for your use case, though has the risk of leaving outdated stuff for Architectures that are no longer built/supported, which is also a dangerous approach for CI/CD environments. ;(

[nuxwin]

@mika

You're right of the fact that removing an architecture can be error prone, specially if we operate on a per build basis where a package for a given repository must be built only for a restricted set of architectures. So yes, my thinking was about extending the repository for new architectures only. For other parameters which could be updated through global environment variables such as ORIGIN, that's another story. But if you say that when changing such attributes, the whole repository database must be updated, that not a good idea.

Right now, I set the ARCHITECTURES environment variable globaly in the sourced /etc/jenkins/debian_glue file. That works pretty well as long as I add it on first setup.

At least from my point of view, the new architectures should be added (and never removed) automatically. This would avoid build failure when a project matrix get updated with new architectures.