Supporting additional architectures

[LecrisUT]

Description

From a recent discussion, we have some users who want some automation of the builds for additional architectures. These seem to involve a separate instance of the infrastructure (example riscv). So on the implementation side, this would need:

• Supporting additional koji urls to push and report from
• Opt-in mechanism for .packit.yaml and Fedora-CI

Benefit

No response

Importance

No response

What is the impacted category (job)?

Fedora release automation

Workaround

• There is an existing workaround that can be used until this feature is implemented.

Participation

• I am willing to submit a pull request for this issue. (Packit team is happy to help!)

[davidlt]

The latest version of Koji instance has FAS integration thus anyone with a FAS account should be able to submit a scratch build. A few folks have been using it already.

F43 (and maybe F42 too) already has riscv Koji profile included by default. If you use koji -p riscv or riscv-koji shortcut it will submit to the RISC-V Koji instance.

This is a minimal Koji setup right now thus not all plugins are enabled (or tested).

I think, we might want to have ability to limit this service when we need compute (e.g. mass rebuilds that take quite some time). I have no idea how much resources Packit/CI consumes daily in general.

Another thing is that we have different boards with different ST/MT performance, total RAM, RAM per core, etc. configurations. We are manually routing specific packages to particular boards to avoid timeouts, for ISA reasons too, or in general for build time.

For example, there are some packages that support vectors already. If you want to test suite to detect relevant ISA and then pick particular optimized implementations you would need an ability to target particular boards.

[LecrisUT]

I think, we might want to have ability to limit this service when we need compute (e.g. mass rebuilds that take quite some time). I have no idea how much resources Packit/CI consumes daily in general.

Wouldn't that be managed by submitting to different queues? We have 2 integrations:

• submit koji builds after a PR is merged. This depends on the project's .packit.yaml
• build scratch builds in the PR for each commit

But regardless, we would probably keep it to opt-in, unless you would have different ideas on it.

Another thing is that we have different boards with different ST/MT performance, total RAM, RAM per core, etc. configurations. We are manually routing specific packages to particular boards to avoid timeouts, for ISA reasons too, or in general for build time.

On our side, it is mostly about the interfacing with koji cli. If you are needing to pass some specific koji parameters for the builds, then it would indeed be problematic

For example, there are some packages that support vectors already. If you want to test suite to detect relevant ISA and then pick particular optimized implementations you would need an ability to target particular boards.

Would that eventually be a spec design issue?

[davidlt]

I am not familiar with the current Fedora CI/Packit setup thus I don't know if there is a separate queue or/and other things. I looked at ~3 minutes quick video on YouTube, poked infra Ansible configuration and looked at Koji for a few minutes 😄 (so far).

As far as I can see there is a packit user(s) in Koji, and scratch and normal builds go to the default channel.

This should be opt-in. Our mass rebuilds usually take some weeks, during that period we would probably want to disable Packit builds in general or rate-limit/delay for lower usage periods (or something like that). Opt-in basically means that we already reducing the load as it requires an action to enable it.

Packages have different hardware requirements. Most of the Fedora builds don't require heavy hardware, and amount of our best hardware is limited. Basically you don't want to send something like GCC to our weakest hardware as that means it will take forever (<2 weeks), tests will timeout, etc. So we tell Koji scheduler once there is buildArch task that it needs to go to a particular host. There are multiple ways to slice the builders: vendor, total memory (important for large projects using LTO for linking), best ST performance (important for some tests [timeouts and more]), memory per core, ISA capabilities (some packages built optimized libraries for multiple ISA variants and you only get to test one; this is mostly relevant for project that are working on vector support).

[LecrisUT]

I am not familiar with the current Fedora CI/Packit setup thus I don't know if there is a separate queue or/and other things.

Iirc the queue on koji for Fedora CI builds is defined from the submitting user (example see channel ci there). I thought it is done the same for packit, but apparently not 🤔, I'll check up on infra why that is not the case. But on your side, you should be able to do something similar.

So we tell Koji scheduler once there is buildArch task that it needs to go to a particular host.

Ooph, is this done manually? If there is anyway that we could help with some automation, then we could try something, maybe have a hosted json file to check how to submit koji jobs, but ideally we would like minimal special sauce to deal with. The integration here is mostly to support the users who opted in to packit workflow for submitting koji builds, and Fedora CI that runs scratch builds on all PRs, but maybe we could also make it available as a default integration for your automation as well

[davidlt]

Iirc the queue on koji for Fedora CI builds is defined from the submitting user (example see channel ci there). I thought it is done the same for packit, but apparently not 🤔, I'll check up on infra why that is not the case. But on your side, you should be able to do something similar.

Correct. There is CI channel, but it seems to be used by Zuul and Jenkins CI.
Otherwise adding a channel is easy.

From Ansible:

channel =
[..]
    user bpeck/jenkins-continuous-infra.apps.ci.centos.org :: use ci
    user zuul :: use ci

I don't see any configuration for Packit for the main Koji instance.

Ooph, is this done manually? If there is anyway that we could help with some automation, then we could try something, maybe have a hosted json file to check how to submit koji jobs, but ideally we would like minimal special sauce to deal with. The integration here is mostly to support the users who opted in to packit workflow for submitting koji builds, and Fedora CI that runs scratch builds on all PRs, but maybe we could also make it available as a default integration for your automation as well

The long-running ideas was/is to use channels for that. One thing is that assignTask is done per host (if done manually API or CLI). We have a feature request (future stuff) to allow listing a channel instead of host. See: https://pagure.io/koji/issue/4466

We might be able to do that on Koji Hub Policy side, but so far we failed. We basically tried once, failed and never looked bad. It causes problems as we basically experimenting on production instance.

So it's a too part problem:

1. Maintain tags or TOML/YAML/JSON/Python/whatever that lists hosts under particular tags.
2. Have a list of packages that must go on a particular board by default.
3. A tiny Python script that scans all buildArch tasks, and if needed does something like: koji -p riscv call assignTask <TASK_ID> <HOST> override=True force=True

There also could be a feature in packit.yaml that would allow listing a preferred channel or/and a host. This could acts as a recommendation for scheduling system. Even on x86_64 and aarch64 not all nodes are identical. On RISC-V side ST and MT performance, ISA, and total memory support is changing rapidly. Today we have to choose the best board for the package to avoid a number of things mentioned before.

There is that extra bit where developer itself might want to pick a different board (mainly for ISA). I think, packages like highway, zlib-ng, ffmpeg, etc. are already supporting vectors and other ISA extensions. We have a few builders that provide vectors. That's probably not the main feature we want, but something developers might find useful.

[nforro]

I thought it is done the same for packit, but apparently not 🤔

See #2776.

[majamassarini]

@LecrisUT do you know how many users are interested in this?

[LecrisUT]

@LecrisUT do you know how many users are interested in this?

Currently I only see sporadic interest. E.g. this PR sparked the conversation since it was not clear how to test the change (they did submit the scratch build manually in the end). Best judge for how necessary it is would be @davidlt. Afaiu they have some other automation for doing the builds and keep them in sync with src.fp.o