Add Flatpak build workflows

[djantti]

We have Flatpak manifests in the tree now, so let's put them to good use! 🏗️

Right now this PR has the basics for creating single-file Mixxx bundles and uploading them as build artifacts. It's using the official Flatpak actions and containers from Flathub. The container images have all the necessary tools and SDKs included, saving on download time.

https://github.com/flatpak/flatpak-github-actions
https://github.com/flathub-infra/actions-images

These can be used for many things - verifying manifests and metadata, creating a package on other events, building Mixxx releases, uploading to repositories and even to Flathub.

There's a lot to do and my GitHub workflow experience is limited, so help is very much appreciated. 🙏

[daschuer]

This looks straight forward. Thank you.

We have discussed to pride different streams for our branches and for stable builds like we do on our PPA.
Did you look into this? Is this still a reasonable idea? How can users switch form one branch to another?
Which infrastructure do we need for this?

How does it relate to:
https://nightly.gnome.org
Can we follow their ideas?

[acolombier]

Indeed, this looking very good already. One thing I would like to see is the flatpak uploaded to the downloads.mixxx.org server, like for any Mixxx build artifacts.

For that reason, it might be easier to merge the new job to the existing build matrix, so you can leverage the prepare-deploy and generate-manifest Python task.

Finally, we should look into automating the upload of a .flatpakref as part of the deploy.py script (or else), so we can easily distribute Mixxx snaphots or release with something like

flatpak install https://downloads.mixxx.org/snapshots/2.5/mixxx-2.5.4-10-ga161b85ea9-x86_64.flatpakref

Of course, we will also claim the Flathub repo for stable release, but allowing these flatpakref to be easily tested would be really good for user IMO.

I'll try to spin a PR to your repo when time allows it, but feel free to get a head start too :)

[djantti]

Yeah, the Flatpak build tools and containers are easy to use. I like them a lot!

For hosting a Mixxx Flatpak repo, we could run flat-manager on the download server or just rsync new OSTree repo structures after a daily build or a release. The latter approach is simple, but AFIK only the latest build would then be available on the repo. In this case we could keep an archive of .flatpak single-file bundles for older releases.

https://github.com/flatpak/flat-manager

Gnome likely uses flat-manager since they have multiple releases (flatpak remote-info --log <repo> <app-id>) available. KDE only has latest builds for nightlies. They also have .flatpakref files nicely in each app repo.

https://cdn.kde.org/flatpak

Installs from different branches can be switched using flatpak make-current <branch>, but it's also possible to use different app IDs (org.mixxx.Mixxx, org.mixxx.Mixxx.Nightly etc.). This would allow fully simultaneous installs and keep Mixxx's config files separated. The user would then have multiple desktop entries (Mixxx, Mixxx (Nightly) etc.) on their system.

For Flathub we could eventually request direct upload credentials and ask for the flathub/org.mixxx.Mixxx repo to be archived. But that's definitely something for later. 😅

[djantti]

For that reason, it might be easier to merge the new job to the existing build matrix, so you can leverage the prepare-deploy and generate-manifest Python task.

I was tinkering with this earlier, but quite a few steps in the matrix needed to be skipped for Flatpak build. So I was having many nested if: statements and my spider-senses started to tingle.

Maybe there's a neat way to split things into even more separate jobs? It's always difficult to figure out a clean solution with limited knowledge of what is even possible.

Anyway, I'll keep on working with this and add debug extension builds next. And PRs are most welcome! 🤗

[daschuer]

but it's also possible to use different app IDs (org.mixxx.Mixxx, org.mixxx.Mixxx.Nightly etc.). This would allow fully simultaneous installs and keep Mixxx's config files separated.

We don't have that yet for the other targets. Since it comes with some challenges I think we should go for flatpak make-current <branch>

Is the KDE solution a matter of sharing a directory? Would it be possible to maintain this directly on out download server without any active server component?

Lets go in small steps. This is already great because it has the flatpaks as artefacts for PRs. There is only one thing we need to double check. How do they play with RPM/Debian install and future possible release streams.
What happens if a use is on a release stream and than installs a PR artefact.

How is signing handled?

[djantti]

Sure thing, let's go with branches. They're quite flexible. 👍

The way Flatpak branches seem to work is that you can have multiple installations with the same app ID, provided they use different branches. And to make things even more flexible (and sometimes confusing), that limitation is separate for system-wide / user installations.

Branch name of master is used unless something else is specified during build. This is the current behavior for the PR and manual Flatpak builds. We can totally keep it like this, so other branch installs won't get overwritten by a PR artifact install. Also the latest package install always becomes the active one. The active branch can be switched with something like flatpak make-current org.mixxx.Mixxx beta and it's also possible to run from a specific branch with say flatpak run org.mixxx.Mixxx//stable.

The KDE nightly Flatpak repos are indeed just directories on a standard web server. I took a peek at the build jobs and they use rsync -Ha --delete to upload new app repos. That's why only the latest build is available for each app. But I'd say it's fine for a simple nightly unstable repo.

But yeah, we prolly should take small steps and limit the scope of this PR. Should we just finish up the artifact builds or add anything else?

I'll test next how the branch option behaves in the workflow. There's also a gpg-sign option for package / repo commit signing. I'll give that a try too.

[daschuer]

That sounds great. I think the only part we need to consider here to have the naming convention correct that it fits to your future plans.

[djantti]

Alrighty, I managed to get signing working after figuring out it's not possible to use GitHub secrets in a forked repo. It does make sense. 😄

Signing works well for Flatpak repositories. We build a repo and use flatpak build-sign --gpg-sign=<keyid> repo org.mixxx.Mixxx to sign the commit and use flatpak build-update-repo repo --gpg-sign=<keyid> to sign the summary. Signing is more of a repo thing, but bundles created from commits are signed too. Validating them is up to the user though.

It's also possible to build bundles for x86_64 and aarch64, create a new repo with ostree --repo=repo init --mode=archive-z2, import them with flatpak build-import-bundle and sign the repo summary. That creates a multi-arch repo directory we can rsync to the download server. This needs to be done in two workflow jobs - first build and upload bundle artifacts, then download the artifacts and create the unified repo.

For bundle names I was thinking of using the same convention that other Mixxx packages use. Looks like tools/deploy.py does it with git describe --always --first-parent --dirty=-modified, so let's go with that. I'm still hoping we can rely on repositories and only use bundle files for the PR builds.

I'll add the new bundle naming and repo signing steps next. Shall we use the existing Ubuntu PPA GPG key also for Flatpak signing?

[daschuer]

Shall we use the existing Ubuntu PPA GPG key also for Flatpak signing?

Yes.

[daschuer]

Validating them is up to the user though.

So only a limited number of people will do this. Is there a different way in the flatpak eco system to verify integrity?

[djantti]

Installs from Flatpak repositories are validated automatically unless the repo is configured with --no-gpg-verify. And if I remember correctly, system-wide repositories can never be configured like that. For bundle installs it's necessary to use flatpak install --gpg-file=<key>, so prolly not many people use it.

[daschuer]

Ah ok, so we probably need to do the reposity thing than, right?
How is validating the key done?
I'm case of our PPA, the user needs to download the public key first, usually done via 'apt-add-repository' from independent keyserver.ubuntu.com.
Does it work similarly with flatpak?

[djantti]

Yeah, definitely. Flatpak is very much based on repositories, so we should use them wherever possible.

For Flatpaks the public key is included in .flatpakrepo and .flatpakref files, so it's automatically configured for repositories. Installs are then refused if repo commits and summary is not signed with the private key.

[djantti]

I did some testing and it's possible to do a PR repo for each build - kinda like what Flathub does on their build service.

First we build for x86_64 and aarch64 and import the bundle artifacts into a unified repo. Then we sign the repo, create a .flatpakref file and rsync everything to a uniquely named directory on the download server. When the build is done, we print the .flatpakref URL into the logs for easy installation.

Each repo uses about 500 MB of disc space, so the download server should periodically purge old builds. But storing even a weeks worth of PR builds is a lot, especially if we keep builds for every run. Maybe having one repo for each PR and just keeping the latest build is more feasible. 🤔

[acolombier]

Great to hear the progress and sorry I didn't help much till now. I should have some time over the weekend to help getting this over the line.

Two elements:

• Build cleanup: this should be fairly easy. We could add a schedule job which garbage collect all builds on a weekly basis. Maybe something like any build alder than 14 days. We could always retrigger a build for a commit in case we need to test something that would have expired, but since this would be exclusive a development repo, it sounds fine to have a short lifecycle for builds there. Obviously, we could keep release build for a longer time, but the point it, we can adjust this number to accommodate the disk space!
• Accessing the download server will require a "privileged" CI. pull_request actions don't have access to the download server as this could be an easy way to hack it. I guess we have two options:
  1. Move flatpak to build as pull_request_target - suboptimal as it would make it hard to debug and and test, when making changes to it
  2. Build the flatpak artifact in the normal pull_request build step and publish artifacts in Github action, and add a /publish-flatpack PR command. This sounds like the most promising options since:
  • We don't have to publish every single builds.
  • We reduce the attack vector of somebody publishing a rogue build and tricking somebody (e.g on Discourse or Zulip) to install it, tho the build is signed by our GPG key
  • We still ensure every flatpack build does it builds, and update the the CI/build scripts is still tested. Only the pushing of the artifact to the repo is in a CI command

What do you think?

[djantti]

No worries! 🙏 Flathub does a build delete after two weeks, so that's prolly enough time for dev builds. Plus we have the bundle artifacts as a backup and those apparently stay around for three whole months.

The second option does sound better and might be easier to extend for other publishing tasks (releases, scheduled daily builds, Flathub etc.). It would be awesome to have everything modular and pass options to the build / publish jobs, depending on how the build was called. OBS does something like this with their Flatpak workflows, but it's difficult to follow how everything is actually connected.

Here's a simple job I used for testing a PR repo. It uses the upload / download artifact actions and creates a unified repo directory. It worked great with just a standard web server, so we can totally do this. 🤩

publish-flatpak:
  name: "Publish"
  runs-on: ubuntu-latest
  needs: build-flatpak
  steps:
    - name: "Install packages"
      run: |
        sudo apt-get update
        sudo apt-get install flatpak ostree

    - name: "Download bundles"
      uses: actions/download-artifact@v7
      with:
        pattern: Flatpak*
        path: bundles
        merge-multiple: true

    - name: "Create repo directory"
      run: |
        ostree --repo=${{ github.run_id }} init --mode=archive-z2
        for file in bundles/*.flatpak; do
          if [ -n "${{ secrets.PRIVATE_KEY }}" ]; then
            flatpak build-import-bundle \
              --gpg-sign=${{ secrets.PRIVATE_KEY }} ${{ github.run_id }} $file
          else
            flatpak build-import-bundle ${{ github.run_id }} $file
          fi
        done

    - name: "Publish repo"
      uses: actions/upload-artifact@v6
      with:
        name: Repo ${{ github.run_id }}
        path: ${{ github.run_id }}

[acolombier]

I have prepared the automation in https://github.com/mixxxdj/mixxx/tree/refs/heads/chore/prepare-flatpak-ci

You should be able to test it with

flatpak remote-add mixxx https://downloads.mixxx.org/flatpak/repo.flatpakrepo

It will have 4 channels:

• stable - for stable release, so currently 2.5.4, eventually will be migrated to Flathub once we have claimed it
• beta - for 2.6 release
• alpha - for 2.7 release
• testing for PR build, published with the PR command.

Only need to add some doc about the newly introduced /publish-flatpak command and the expiry notice. How do you want to proceed? Do you want me to open a PR to your fork?

Note that I am new to ostree so I could have missed some elements, but since this is development repo, I guess we could start testing it?

[daschuer]

On our PPA we have also "patches" for the not yet releases stable branch.

[daschuer]

This PR focusses to upload unsigned flatpacks as GitHub artefact. Is this a sufficient goal for PRs? Let's go in small steps.
Do we later really need a /publish-flatpak trigger? It is probably cumbersome if a user follows this repository and receives random PRs. Maybe we should keep it a manual artefact download like we do with our other targets. I cant remember that the missing signing was a problem for Windows and macOS for PR builds.

[djantti]

It works, let's gooo! So it's cloning the repo locally, importing the new bundles and rsyncing the changes back. That means we can have a single repo with multiple branches, even without flat-manager. 🥳

I'm not entirely sure, but it's possible that the build step needs a branch: option set based on what's being built. I've previously run into errors when importing a bundle into a different repo branch it was built for. We might also want to change ${CHANNEL} to ${BRANCH} in the workflow to keep it in line with upstream OSTree / Flatpak documentation.

I think we can also have the repo signed by adding a --gpg-sign option in the "Push the updated repo" step commands (or maybe just one of them?) and including the public key (single GPGKey line) in the .flatpakrepo file.

But let's test things! The new prepare-flatpak-ci branch already has everything, so we can discuss things here and work on it directly too. Or we can keep this PR as just the artifact build feature and start a follow-up for the repo stuff. It's all good.

[djantti]

Do we later really need a /publish-flatpak trigger? It is probably cumbersome if a user follows this repository and receives random PRs.

The main repo prolly should not have PR builds, unless they're all in their own branches (say, org.mixxx.Mixxx//PR12345). Using separate repos for each PR build is another possibility, or we can just rely on the bundle artifacts and use the main repo for releases / daily builds.

[daschuer]

What is the state of this branch? Is it still a draft?

We now have these PR artifacts in this PR:

Looks good.

I see the step:
"Create Debug Extention"
Is this an extension in terms of symbol files only. Or is it a second full flatpak with symbol files embedded?

[daschuer]

It seems to be the symbol files only. How is the naming conventions? Maybe rename:
"Flatpak aarch64 Debug Extention"
.. that people do not download that package only.

[acolombier]

This PR focusses to upload unsigned flatpacks as GitHub artefact. Is this a sufficient goal for PRs? Let's go in small steps.

Yep, agreed!

I'm not entirely sure, but it's possible that the build step needs a branch: option set based on what's being built.

Technically, and if I understood how every works together, we shouldn't need to do this in the push workflow (untrusted). Instead, we have in the publish workflow the following:

flatpak build-import-bundle ... repo.tmp ...
checksum=$(ostree --repo=repo.tmp rev-parse ".../master")
ostree --repo=repo pull-local repo.tmp "${checksum}"
ostree --repo=repo commit -b "${type}/${id}/${arch}/${CHANNEL}" -s "$COMMIT_SUMMARY" "--add-metadata-string=version=$VERSION" "--tree=ref=${checksum}"

This was written according to ostree doc, so under reserve of my correct understanding. It does mean that we need a predictable branch name. master (default) is maybe not ideal but it would be good to have something always identical, so whatever promotion logic, it knows where to get the source ref from

Do we later really need a /publish-flatpak trigger? It is probably cumbersome if a user follows this repository and receives random PRs.

This is not the expected behaviour. PRs would be published under a different channel, so user are not expected to receive updates.
This feature is however required as it will allow us to test Flatpak related changes (e.g a user raises an issue on Flatpak, we can invite them to install the test build)

[djantti]

Ah excellent, hopefully the branches / bundle imports are flexible like that. I may have just messed up something in my testing. There's very little information on running a custom Flatpak repo and using OSTree directly, so a lot of trial and error is involved.

I'll change the Debug extension naming to be more descriptive. Other than that this PR is pretty much ready for the artifact upload feature. Right now the bundle artifacts should get signed, but it's not that useful since their verification is not automatic. Shall we keep things simple and leave them unsigned for now?

[daschuer]

Shall we keep things simple and leave them unsigned for now?

Yes. Artefact self have a high level of trust via GitHub. An attacker can file an evil PR. We should not sign it.
Testers have to check themself if it is evil or not. GitHub allows to trace it.

Do you want to remove the Draft stet of this PR now. I think I can merge than.

[djantti]

Okay, I think this should be good to go now.

[daschuer]

Looks Good now. I nice step for our upcoming flatpak repository. Thank you.