[RFE] Add API endpoint for querying the version (e.g. NEVRA) of the running composer deployment.

[thozza]

The EXD team responsible for RHEL composes would like to implement an optimisation which would allow them to not build images for every new compose/respin, in case no relevant part of the tooling or used content changed.

Motivation

When building compose for the same release next time, check the metadata to find out if something meaningful changed (e.g. version of package in the image, compose configuration, version of deployed image-builder)

• if yes, submit new build
• if no, copy image from previous compose (since a new one would be practically identical)

Requirement

1. Upon finishing an osbuild-koji task, store enough metadata to identify all content in the image.
2. Have an API endpoint for querying the deployed versions of relevant parts of Image Builder deployment (osbuild-composer, osbuild-composer-worker, osbuild). Version stored in Koji build metadata won't allow to check for possible changes before submitting another build.
3. Have the same information as mentioned in (2.), but attached as metadata to the Koji Build via the Koji API.

[teg]

I totally support this. I think our main identifier should be the git sha for both the distro definitions (for the time being that means osbuild-composer), and the git sha of osbuild in the worker. NEVRA might be more tricky as we don't restrict ourselves to deploying releases.

Suggestion for implementation: extend the spec file to write out the sha to a file that we ship in both osbuild and osbuild-composer.

[bcl]

Seems like something that should be added to the status output:

osbuild-composer/internal/weldr/api.go

Line 547 in 0865720

func (api *API) statusHandler(writer http.ResponseWriter, request *http.Request, _ httprouter.Params) {

You can embed constants into the compiled binary by using -ldflags="-X path/to/variable" like I do for the composer-cli version in weldr-client here https://github.com/osbuild/weldr-client/blob/main/Makefile#L9

[thozza]

@bcl thanks for your suggestion. I like embedding the version into the binary more than writing it to a file on disk. Mainly because I think that the value should be immutable for a specific build. I think that we need this also for other APIs, not just Weldr or have some kind of API just for telemetry. We don't have SSH access to the system where the internal composer (integrated with Brew) is running...

[teg]

@thozza my intention was to write it to a file, but keep that internal and have composer read it andn expose it via APIs, so all for @bcl's suggestion :)

[msehnout]

I totally support this. I think our main identifier should be the git sha for both the distro definitions (for the time being that means osbuild-composer), and the git sha of osbuild in the worker. NEVRA might be more tricky as we don't restrict ourselves to deploying releases.

The git SHA will be tricky as well because we don't always build from Git. I'm not sure about production builds in Koji, but when running mock during the execution of fedpkg scratch-build --srpm, it doesn't have any git repository so it fails to git rev-parse HEAD. Reporting Git revision when available and NEVRA otherwise might work here because there should always be exactly one build with the same NEVRA.

Combined with #1788, osbuild-composer could return something like:

build: devel
revision: abc123...

or

build: prod
revision: osbuild-composer-33-1.f36.x86_64

[ondrejbudai]

Yeah, when I originally picked this up, I also stumbled upon the same issue. The proposed solution works for me. @teg wdyt?

[thozza]

Hmm, I see the problem... However I don't think that having the git SHA is a complete must. The main idea is IMO to be able to determine if the version of any relevant component changed. RPM NEVRA is IMO as good as git SHA, since the SPEC can be used to determine the exact upstream release of sources which was used, plus any additional changes which were backported.

I would say that reporting only git SHA with RPM builds could be actually wrong. With downstream patches, the SHA would most probably not change, because the base upstream release tarball used for the build would not change. However in such case, we would like composer to report different version.

If we decide that we really want to include the git SHA in all cases, one option could be to always embed it into the sources tarball when it is created from git and then use the value when building binaries. This would probably not work with the release archives created by Github. But as the last resort we could be creating them with a script and uploading to Github? However in case of a production RPM build, I do believe that the NEVRA should be part of the reported version. Nevertheless the upstream git SHA can be there at least as a reference.

So all in all, I think that the proposed solution will cover the needs...

[msehnout]

@thozza I had exactly the same thoughts about the issues with tarballs. One additional idea that came to my mind is that we could report git sha and the git repository (upstream vs downstream), but that feels overcomplicated and I'm not sure it is even feasible.

Regarding the custom tarballs, I'd prefer to keep away from them as it makes everything more complicated and fragile (building, interaction with Packit, manual uploads).

The problem with using NEVRA only is that our upstream CI will happily build multiple RPMs with the same NEVRA, so we need the additional Git SHA to actually find the right revision.

[teg]

The git SHA v NEVRA issue is tricky. We struggled with this when we first set up Schutzbot. The ideal back then was that there should be a direct mapping between the upstream git commit and the downstream RPM (so no downstream patches), and that this should be reflected in the NEVRA. This would allow someone to figure out exactly what code is running for a given NEVRA. This is done for RPMs we build in Schutzbot, but not downstream (yet?).

For this particular use-case, we should keep in mind that the service will eventually only be run from containers, and as such won't have a NEVRA at all (no RPMs are built), so we should at least support exposing the git SHA and only fall back to the NEVRA when no git SHA is set.

Ideally we would figure out a way to always set the git sha in the specfile that is used for our downstream releases, so that we could use the git SHA in all cases. I wonder if that is something Packit has support for and that we could easily hook into? If that's not feasible, falling back to NEVRA might be the next best thing.

[msehnout]

If we wanted to keep the tarball generated by Github unchanged and if we wanted to have the same spec file in upstream and downstream, I believe we could create a file in downstream git with a reference to the upstream commit. Something like:

Source1:   upstream-revision

%build
# load the %{source1} file

On the other hand, we already point to a specific release, which in turn points to a specific upstream revision, so it is already possible with one extra step.

[thozza]

@teg did you consider the internal Brew deployment when you mentioned the service? While I'm aware of the goal to have the osbuild-composer server running in RHOCP, I'm not sure if this includes the Brew workers. My expectation is that the osbuild-composer-worker will be running on the bare-metal or virtualised workers in the future. Or is there a plan to run workers for each architecture also in a container on our workers? If not, I do believe that we will have to continue deploying our software as internally-built RPMs on them.

Once we move the distro definitions to the distribution itself, we will basically have no other option than using RPMs for them and report the definition's NEVRA.

I also expect that we may need to make a specific change (fix) in the future to our internal Brew service, without pulling in the latest upstream main. From my point of view we will have to either use a backported downstream patch in such a case, or have a branch in upstream for it and make a release with some reasonable release version. Upstream release versioning may become a very interesting task in this scenario, if you consider that we may have to maintain multiple variants based on a specific upstream release (for RHEL minor releases, various deployments, ...). It may actually be easier to maintain a change as a downstream patch until we can rebase to some newer upstream release. I'm not advocating that we should default to downstream patches or use them extensively, but I would not disregard them as never happening.

I think that we won't be able to solve this by using only upstream git SHA or RPM NEVRA. It will most probably be a combination of both, when one of them (or both) is used depending on the deployment and build environment. It would be great if we could use the same approach for the whole stack of project which makes up our deployment.