Parallelize builds across OS and architecture

[bschwedler]

Problem

One build-test job builds every OS and architecture for a version. As we add more operating systems, this job needs more disk space and more time.

Workbench hit this limit when we added Ubuntu 26.04. We increased the runner size as a temporary fix.

• Use a larger runner for the amd64 builds images-workbench#190

This problem will return. RHEL 10 support is planned for Workbench and other products. Each new OS adds more work to the same job.

• Add RHEL 10 (UBI) support for workbench (Standard/Minimal) images-workbench#175
• Add RHEL 10 (UBI) support for workbench-session / workbench-session-init images-workbench#176
• 2026.10: 🏗️ RHEL 10 (UBI) image support #575
• Design: model UBI as a first-class concept in the bakery OS schema #698

Scope

This affects two shared workflows: bakery-build-native.yml and bakery-build-pr.yml. All three product repos use these workflows — images-connect, images-package-manager, and images-workbench — including every pull request build.

Goal

Run each OS and architecture combination as its own parallel job, instead of building all of them in one job per version.

See the comments below for the full design. Each comment covers one part.

[bschwedler]

Build-test matrix: split by OS

Today, bakery ci matrix outputs one entry per {image, version, platform}. The build-test job builds every OS for that version in one call.

New design: bakery ci matrix resolves the OS list for each version. Then it crosses each OS with the platforms of that OS. The result is one flat list of {image, version, os, platform} entries.

build-test becomes one flat matrix job over this list. Each job builds one OS on one architecture.

Cross each OS with its own platform list

Do not use the platform list of the version. ImageVersion.supported_platforms returns the union of all OS platforms (config/image/version.py:334-347). An OS with no platforms key defaults to amd64 (config/image/build_os.py:13).

The union invents combinations that do not exist. Those jobs match zero targets, and exit_if_no_targets exits 1 (cli/common.py:30-50). The job fails. It does not skip.

Workbench 2026.08.1 has five valid jobs, not six:

OS	Platforms in bakery.yaml	Jobs
Ubuntu 22.04	default (linux/amd64)	1
Ubuntu 24.04	linux/amd64, linux/arm64	2
Ubuntu 26.04	linux/amd64, linux/arm64	2

Add the OS to the metadata names

Both names use <image>-<version>-<platform>-metadata today, and the upload sets overwrite: 'true' (bakery-build-native.yml:394,464-469). Three OS jobs for one image, version, and platform write one artifact name. The last upload wins.

bakery ci publish acts only on the targets in the metadata (imagetools.py:605-620). The other OS never get their final tags, and no step reports an error. ImageVersionOS.extension holds a safe slug, such as ubuntu2204. The PR workflow needs no change here, because it writes no metadata.

Pass --image-os to three steps

Build, Test, and Scan all need the flag. bakery build, bakery dgoss run, and bakery wizcli scan all support it (plugins/builtin/wizcli/__init__.py:80).

The filter is a regex search against the OS name (config/config.py:1054). Anchor and escape the value, for example ^Ubuntu 22\.04$. Without an anchor, RHEL 1 also matches RHEL 10.

Change all four call sites together

BakeryCIMatrixFieldEnum needs an OS member (cli/ci.py:33-36), so a caller can pass --exclude os. Four call sites read the matrix: bakery-build-native.yml:222, bakery-build.yml:190, bakery-build-pr.yml:127, and bakery-build-pr.yml:148. No product repo pins the bakery version: input, so the tool and the workflows both float on @main.

Note: bakery-build-native.yml already splits architecture. The matrix carries platform, and line 251 selects the arm64 runner. OS and variant are the axes that still share a job. Variants stay together, because Standard and Minimal for one OS share base layers.

[bschwedler]

What needs: can and cannot do

needs: names one job. It does not select part of a matrix. A job that needs a matrix job waits for every instance of that matrix.

Inside one workflow file, a job cannot wait for the instances of one version only. A flat matrix over {image, version, os, platform} therefore gives every later job the same barrier. All builds must finish first.

A matrixed reusable workflow call removes that limit. Each call instance runs its own copy of the inner workflow, and needs: inside the inner workflow covers only that instance. See "A nested matrix is possible" below.

Two conclusions still hold:

• A static chain of merge jobs, one per version, does not work. The version list changes over time, and a fixed chain of jobs cannot grow with it.
• Merge stays one ordered job per image, because push_sort_key spans versions. See "Merge must stay one job per image" below.

Docs:

• https://docs.github.com/en/actions/using-jobs/using-jobs-in-a-workflow#defining-prerequisite-jobs
• https://docs.github.com/en/actions/using-jobs/using-a-matrix-for-your-jobs

[bschwedler]

Merge must stay one job per image

Docker Hub lists tags in about push order. So bakery pushes in a fixed order: oldest version first, primary OS and variant last. push_sort_key sets this order. bakery ci publish pushes the images one at a time, inside one job.

push_sort_key compares the parsed version (image/image_target.py:358-378), so the order spans versions. A merge job per version cannot hold that order. Docker Hub can then show the wrong tag as "latest".

Merge stays one job per image. It waits for every build-test job for that image, across all versions, OS, and architectures.

Docker does not document this behavior

The closest evidence available:

• The Hub API spec documents only page and page_size for the tags endpoint. The repositories endpoint right below it does document ordering.
• The Hub tags page says only that you can view the available tags and the image size.
• A Docker forum moderator: "The order of tags is not defined. I'd interpret it as it can always change."
• The OCI distribution spec requires lexical order for the registry tags/list endpoint, not push order.
• A live query of posit/connect returns tags in about push order, but not strictly by last_updated. latest (19:02:59.372) comes before ubuntu-24.04-std (19:03:00.810).

The order costs about 30 minutes per weekly run

See the Stage 1 comment for the measurements. One option can recover most of that time. Push every target in parallel. Then re-copy only the display tags in order. The display tags are latest and the primary OS and variant aliases. A re-copy writes a manifest against blobs that already exist, so it takes seconds.

This option depends on the same undocumented behavior. It needs a test against Docker Hub first.

References:

• Push image versions in reverse order #484
• Push image versions in deterministic order to fix Docker Hub display #505
• https://docs.docker.com/reference/api/hub/latest/
• https://docs.docker.com/docker-hub/repos/manage/hub-images/tags/
• https://forums.docker.com/t/order-of-tags-in-rest-api/35412
• https://github.com/opencontainers/distribution-spec/blob/main/spec.md

[bschwedler]

Split merge Stage 1 from Stage 2

Merge does two things in one job. Stage 1 creates the multi-platform index and converts the image for SOCI. Stage 2 pushes the final images, in order.

Merge is the larger bottleneck today

On the weekly workbench build, the longest build-test job takes 42.9 minutes. The merge job takes 84.1 minutes. Its log reports Found 30 targets.

Marker	Time
First SOCI conversion starts	04:14:32
Last SOCI conversion starts	04:57:56
First Stage 3 verify	05:36:28

Four workers ran the conversions. DEFAULT_MAX_CONCURRENCY is 4 (const.py:46) and the workflow passes no --jobs. The 38 minutes before the first verify cover the last conversions plus the 30 sequential index copies of Stage 2. Stage 2 logs nothing on success, so that split is an estimate.

Proposal

Move Stage 1 to its own matrix job, one per {image, version, variant, os}. Stage 2 stays one ordered job per image.

Stage 1 cannot split per platform. ImageTarget holds no platform (image/image_target.py:220-239), and the index combines the platforms of one target.

Stage 1 passes only a registry tag string per target to Stage 2. The split needs a small JSON file between jobs, not another image push and pull.

Stage 1 does write image content to local disk

SociConvertWorkflow copies the source into a local OCI layout with oras, converts layout to layout, then pushes the result (plugins/builtin/imagetools/soci.py:139-230). A SOCI target holds about twice the image size while it converts, times four workers. Workbench enables SOCI for the whole image. The merge runner already has the disk problem this comment predicts.

Two cheap steps come first

• Pass --jobs to bakery ci publish. Stage 1 waits on the network and the disk, not the CPU.
• Log a duration per stage, so the numbers above stop being an estimate.

This work does not block the build-test change. It returns more wall-clock time.

Run: https://github.com/posit-dev/images-workbench/actions/runs/31924407538

[bschwedler]

A nested matrix is possible

This builds on "What needs: can and cannot do" above.

A job that calls a reusable workflow supports strategy. Inside the called workflow, needs: covers only that call instance. So a version-level job can start its own OS-and-architecture matrix.

The nesting limit is also no longer four. Since November 2025 it is ten levels and 50 workflows. The comment at the top of bakery-build-native.yml is out of date.

Shape

The contract for product repos stays the same:

• bakery-build-native.yml keeps its name, inputs, and secrets.
• matrix emits one entry per image and version. Each entry carries its own target list: [{image, version, targets: [{os, os_slug, platform}]}].
• build matrixes over those entries and calls a new inner workflow. It passes targets: ${{ toJSON(matrix.entry.targets) }}.
• The inner workflow runs one build-test job over fromJson(inputs.targets).
• merge stays in the outer workflow, one ordered job per image. It needs the matrix call job, so it waits for every instance.

The depth is three of ten: product repo, native, targets.

What nesting buys

Nesting keeps each matrix small. GitHub allows 256 jobs per matrix. A flat matrix puts connect-content at 125 jobs, then 175 with RHEL 10, then about 210 with one more Python version.

Nesting also groups jobs by version in the UI, and it makes a per-version fan-in possible. The Stage 1 split needs that fan-in.

Nesting adds no parallelism to build-test on its own. That job graph is the same as a flat matrix.

Four things to get right

• The dependency version cache. setup-bakery hashes cache-scope into the cache artifact name, and callers pass toJSON(inputs) (setup-bakery/action.yml:81-91). The inputs of the inner workflow differ per version, so every build job misses the cache that matrix wrote. Each job then resolves versions from the CDN again. For dev channel builds, two jobs can resolve different versions in one run. Pass the scope of the parent down as an input.
• Unsupported keywords. A calling job supports only name, uses, with, secrets, strategy, needs, if, concurrency, and permissions. Move the timeout-minutes: 120 at bakery-build-pr.yml:197 into the inner workflow.
• Secrets. Product repos pass all eight explicitly. Forward them, or use secrets: inherit on the inner call.
• No expressions in uses. The inner reference must be a static string.

Two questions to answer first

Both are cheap to test.

1. Does the limit of 50 workflows count matrix instances or unique files? The docs say "50 unique reusable workflows from a single workflow file". The changelog says "call up to 50 workflows in total from a given workflow run". connect-content creates 25 calls. Test a matrix of 60 calls to a trivial workflow.
2. Does uses: ./.github/workflows/inner.yml resolve inside images-shared when a product repo calls the outer workflow? The docs say the called workflow comes from the same commit as the caller workflow. A fully qualified @main reference works, but it breaks same-commit testing of nested changes in images-shared PRs.

Docs:

• https://docs.github.com/en/actions/how-tos/reuse-automations/reuse-workflows
• https://docs.github.com/en/actions/reference/workflows-and-actions/reusing-workflow-configurations
• https://github.blog/changelog/2025-11-06-new-releases-for-github-actions-november-2025/