Simplify caa multi arch build #2777

stevenhorsman · 2026-01-12T14:56:08Z

The CAA multi-arch build is confusing (at least to me) with uploaded files pointing to sha that are later downloaded to create a dynamic manifest based on them. I think by hardcoded the arches we include in the manifest (which doesn't change often), we can re-use the "standard" caa build and then create a manifest in a more standard way.

Along the way fix some inconsistency with tags and zizmor issues.

stevenhorsman · 2026-01-12T14:59:20Z

I'm not sure how to test this - I tried running against my fork, but for reasons (maybe permissions related?) github won't run the CAA job: https://github.com/stevenhorsman/cloud-api-adaptor/actions/runs/20923681415 and just says "Waiting for pending jobs". As such I guess we should wait until post the 0.18 release to merge this, so we have time to debug & fix/revert any issues?

src/cloud-api-adaptor/hack/publish.sh

ldoktor

@stevenhorsman do you intend to convert this from draft? To me it looks better than the current approach. I haven't reviewed thoroughly the new image tags (which is why I'm not giving the ack) but the publishing seems very good and the new tags schema is cleaner.

stevenhorsman · 2026-01-26T10:26:45Z

@stevenhorsman do you intend to convert this from draft? To me it looks better than the current approach. I haven't reviewed thoroughly the new image tags (which is why I'm not giving the ack) but the publishing seems very good and the new tags schema is cleaner.

Thanks for the comments. As I think I mentioned above - as this is potentially disruptive and we should have a release very soon, I'm planning on leaving this as draft until the release is done. I need to do more work to test what I can in my fork as well before that.

stevenhorsman · 2026-02-05T17:05:13Z

Ok, this seems to be tested and working okay in my fork, so maybe it's not risky like I though: https://github.com/stevenhorsman/cloud-api-adaptor/actions/runs/21719552373

ldoktor · 2026-02-09T14:02:36Z

src/cloud-api-adaptor/hack/image-manifest.sh

-    manifest="${registry}/${name}:$tag"
-    for arch in "${all_arches_array[@]}"; do
-        docker pull --platform="linux/$arch" "${registry}/${name}:$tag"
-        docker inspect "${registry}/${name}:$tag" |grep Architecture


This verification is not included in the hack/publish.sh. The question is whether we care... (I'd rely on docker to generate them correctly)

Yeah, my view is that if we don't trust docker here then we need to pull down and check images in a lot of builds

ldoktor · 2026-02-09T14:07:33Z

src/cloud-api-adaptor/hack/build.sh

-		# strip the `linux/` from the docker buildx platform format
-		arch_suffix="-${arch#"linux/"}"
+		# strip the `linux/` from the docker buildx platform format in the tag
+		arch_suffix="-${arch#${arch_prefix}}"


This confused me a bit, I was looking for the change but it's only an unrelated cleanup :D (very minor, mentioning it just to save other reviewers time)

Btw two lines above is also a typoe "if we ar ebuilding for a single arch" (again, not critical but if you happen to be updating it you might consider including that fix as well)

Yeah, I just spotted that we set a "constant" value and then ignored it and redefined it. I'm happy to just remove arch_prefix="linux/" and revert this line if you'd prefer?

Ideally I'd like to have it in a separate commit along with the other typo fix, but it's fine keeping it here as well as it's improving the situation...

ldoktor · 2026-02-09T14:18:51Z

src/cloud-api-adaptor/hack/publish.sh

+    for tag in "${TAGS[@]}"; do
+        amend=""
+        for arch in "${MULTI_ARCHES[@]}"; do
+            amend+="--amend ${IMAGE_REGISTRY}:${tag}-${arch} "


The default for registry in caa_build_and_push_all_arches.yaml is quay.io/confidential-containers so this becomes quay.io/confidential-containers:${tag}-${arch}. Shouldn't the cloud-api-adaptor be included as well?

Definitely - I had intended it to be included in the registry, but this isn't the case. I don't want to hard-code it as we probably want to swap the other image builds to use this, so I'll maybe add an IMAGE_NAME parameter too. Thanks for the spot

ldoktor · 2026-02-09T14:20:17Z

src/cloud-api-adaptor/hack/publish.sh

+
+function _publish_multiarch_manifest()
+{
+    ${IMAGE_REGISTRY:?}


I guess you wanted to print these? (this will execute the IMAGE_REGISTRY as a command!)

No - it is supposed to check that the value is set rather than executing it? I can add a better error string though here

Maybe It only works for assignment?

I mean echo this will check the value, but leaving it on a separate line evaluates that expression :-/. Ai suggests also : ${IMAGE_REGISTRY:?} but it sounds odd to me :-) (Usually I explicitly check by [[ -z "$VAR" ]] in my code...

Ah ok : was the bit I'd seen before that I missed. I reworked it anyway

ldoktor · 2026-02-09T14:21:26Z

src/cloud-api-adaptor/hack/publish.sh

+function _publish_multiarch_manifest()
+{
+    ${IMAGE_REGISTRY:?}
+	${IMAGE_TAGS:?}


There is an inconsistency between tabs/spaces, could you please pick one and keep the style?

ldoktor · 2026-02-09T14:28:04Z

.github/workflows/caa_build_and_push_all_arches.yaml

-      - name: Generate image manifest
+      - name: Publish multi-arch dev manifest
+        run:  |
+          ./src/cloud-api-adaptor/publish.sh publish-multiarch-manifest


We should already be in src/cloud-api-adaptor dir due to defaults.run.working-directory (previously hack/image-manifest.sh was also in that location)

ldoktor · 2026-02-09T14:33:10Z

Thanks @stevenhorsman, this unifies the image names (note the azure-e2e-test.yml still uses the dev-${GITHUB_SHA} format, is this intentional?). I noticed few little breakages (incorrect path, ...) and left even few little less-important remarks. Please find them in-line.

There is an issue in setup-go that it lacks endian awareness, so port the fix from caa_build_and_push_per_arch.yaml to the standard CAA build workflow, to enable us to run the workflow on the ppc runner, rather than needing to use emulation, which can be slow Drive-by-fix of zizmor warning Signed-off-by: stevenhorsman <[email protected]>

The difference between the caa_build_and_push and caa_build_and_push_per_arch is confusing. I hope to address this in this PR, but let's start by renaming for better clarity. Signed-off-by: stevenhorsman <[email protected]>

.github/workflows/caa_build_and_push_all_arches.yaml

+name: (Callable) Build and push cloud-api-adaptor multi-arch manifest image for all type/arch combinations
+
+on:
+  workflow_call:
+    inputs:
+      registry:
+        default: 'quay.io/confidential-containers'
+        description: 'Image registry (e.g. "quay.io/confidential-containers") where the built image will be pushed to'
+        required: false
+        type: string
+      dev_tags:
+        description: 'Comma-separated list of tags for the dev built image (e.g. latest,<sha>-dev)'
+        required: true
+        type: string
+      release_tags:
+        description: 'Comma-separated list of tags for the release built image (e.g. <sha>)'
+        required: true
+        type: string
+      git_ref:
+        default: 'main'
+        description: Git ref to checkout the cloud-api-adaptor repository. Defaults to main.
+        required: false
+        type: string
+    secrets:
+      QUAY_PASSWORD:
+        required: true
+
+defaults:
+  run:
+    working-directory: src/cloud-api-adaptor
+
+permissions: {}
+
+jobs:
+  build_push_job:
+    name: build and push single arch dev and release images
+    strategy:
+      fail-fast: false
+      matrix:
+        types: [
+          { arch: linux/amd64,   runner: ubuntu-24.04 },
+          { arch: linux/arm64,   runner: ubuntu-24.04-arm },
+          { arch: linux/ppc64le, runner: ubuntu-24.04-ppc64le },
+          { arch: linux/s390x,   runner: ubuntu-24.04-s390x },
+        ]
+    uses: ./.github/workflows/caa_build_and_push.yaml
+    with:
+      dev_arches: ${{ matrix.types.arch }}
+      dev_tags: ${{ inputs.dev_tags }}
+      git_ref: ${{ inputs.git_ref}}
+      registry: ${{ inputs.registry }}
+      release_arches: ${{ matrix.types.arch }}
+      release_tags: ${{ inputs.release_tags }}
+      runner: ${{ matrix.types.runner}}
+    secrets:
+      QUAY_PASSWORD: ${{ secrets.QUAY_PASSWORD }}
+    permissions:
+      packages: write  # Needed to push the images to GHCR
+
+  manifest_job:
+    name: generate images manifest
+    runs-on: ubuntu-24.04
+    needs: [build_push_job]
+    steps:
+      - name: Checkout the code
+        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          fetch-depth: 0
+          ref: "${{ inputs.git_ref }}"
+          persist-credentials: false
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3.12.0
+
+      - name: Login to quay Container Registry
+        if: ${{ startsWith(inputs.registry, 'quay.io') }}
+        uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3.7.0
+        with:
+          registry: ${{ inputs.registry }}
+          username: ${{ vars.QUAY_USERNAME }}
+          password: ${{ secrets.QUAY_PASSWORD }}
+
+      - name: Login to Github Container Registry
+        if: ${{ startsWith(inputs.registry, 'ghcr.io') }}
+        uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3.7.0
+        with:
+          registry: ${{ inputs.registry }}
+          username: ${{ github.repository_owner }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Publish multi-arch dev manifest
+        run: |
+          ./publish.sh publish-multiarch-manifest
+        env:
+          IMAGE_REGISTRY: ${{ inputs.registry }}
+          IMAGE_NAME: "cloud-api-adaptor"
+          IMAGE_TAGS: ${{ inputs.dev_tags}}
+
+      - name: Publish multi-arch release manifest
+        run: |
+          ./publish.sh publish-multiarch-manifest
+        env:
+          IMAGE_REGISTRY: ${{ inputs.registry }}
+          IMAGE_NAME: "cloud-api-adaptor"


stevenhorsman · 2026-02-09T15:05:39Z

Thanks for the great review @ldoktor - I think I've addressed all the points, and now I've satisfied myself that I was right the first time about the riskiness of this I'm re-running on my fork and have switched it to draft.

Add a helper script that can create one or more multi-arch manifest images for our three supported architectures giving a registry and a list of one, or more tags. Based on the release.sh script from kata-containers Signed-off-by: stevenhorsman <[email protected]>

Currently we have multiple formats of image tags for dev images: - <release>-dev-<arch> for released images - dev-<sha>-<arch> for interim published images - latest-<arch>-dev for daily e2e test images - ci-pr<pr number>-dev (no arch) for the x86 only packer PR e2e test images - ci-pr<pr number>-<arch>-dev for mkosi specific-arch PR e2e test images This shows that we have multiple different code paths, or logic being run to do the same task and we'd like to reduce duplication and increase consistency, so let's move all to the release version: <tag>-dev-<arch> Signed-off-by: stevenhorsman <[email protected]>

Rather than having separate logic and builds for the multi-arch image including the confusing upload/download of a tags file to drive things, we can just swap and use the existing CAA build workflow, to build the images for each arch and the new multi-arch publish to create the multi-arch manifest. Signed-off-by: stevenhorsman <[email protected]>

Nothing should be calling the `image-with-arch` make target anymore now that the process is unified, so remove it and the code that only it called to simplify and remove duplication. Signed-off-by: stevenhorsman <[email protected]>

stevenhorsman force-pushed the simplify-caa-multi-arch-build branch from e52f407 to 8523c10 Compare January 12, 2026 14:57

stevenhorsman force-pushed the simplify-caa-multi-arch-build branch 3 times, most recently from 36db4d5 to c2416c4 Compare January 21, 2026 16:57

ldoktor reviewed Jan 26, 2026

View reviewed changes

src/cloud-api-adaptor/hack/publish.sh Outdated Show resolved Hide resolved

ldoktor reviewed Jan 26, 2026

View reviewed changes

stevenhorsman force-pushed the simplify-caa-multi-arch-build branch from c2416c4 to d0d542a Compare January 26, 2026 09:15

stevenhorsman force-pushed the simplify-caa-multi-arch-build branch 2 times, most recently from 1d8e7bc to 41389cf Compare February 5, 2026 17:04

stevenhorsman marked this pull request as ready for review February 5, 2026 17:05

stevenhorsman requested a review from a team as a code owner February 5, 2026 17:05

ldoktor reviewed Feb 9, 2026

View reviewed changes

stevenhorsman added 2 commits February 9, 2026 14:56

workflow: Rename multi-arch CAA build

56a78ba

The difference between the caa_build_and_push and caa_build_and_push_per_arch is confusing. I hope to address this in this PR, but let's start by renaming for better clarity. Signed-off-by: stevenhorsman <[email protected]>

stevenhorsman force-pushed the simplify-caa-multi-arch-build branch from 41389cf to c80bae8 Compare February 9, 2026 14:58

github-advanced-security bot found potential problems Feb 9, 2026

View reviewed changes

stevenhorsman marked this pull request as draft February 9, 2026 14:58

stevenhorsman force-pushed the simplify-caa-multi-arch-build branch from c80bae8 to df3b3a4 Compare February 9, 2026 15:03

stevenhorsman force-pushed the simplify-caa-multi-arch-build branch 2 times, most recently from b4760e2 to 39aba27 Compare February 9, 2026 17:26

stevenhorsman added 4 commits February 9, 2026 18:05

caa: Remove old build logic

a06934c

Nothing should be calling the `image-with-arch` make target anymore now that the process is unified, so remove it and the code that only it called to simplify and remove duplication. Signed-off-by: stevenhorsman <[email protected]>

stevenhorsman force-pushed the simplify-caa-multi-arch-build branch from 39aba27 to a06934c Compare February 9, 2026 18:05

Simplify caa multi arch build #2777

Are you sure you want to change the base?

Simplify caa multi arch build #2777

Uh oh!

Conversation

stevenhorsman commented Jan 12, 2026

Uh oh!

stevenhorsman commented Jan 12, 2026

Uh oh!

Uh oh!

ldoktor left a comment

Choose a reason for hiding this comment

Uh oh!

stevenhorsman commented Jan 26, 2026

Uh oh!

stevenhorsman commented Feb 5, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stevenhorsman Feb 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ldoktor commented Feb 9, 2026

Uh oh!

Check warning

stevenhorsman commented Feb 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

stevenhorsman Feb 9, 2026 •

edited

Loading