[FEATURE] Matrix Strategy Multi-Arch example

[kaovilai]

Is your feature request related to a problem? Please describe.

Clone of docker/docs#16593

Describe the solution you'd like

similar to docker/docs#17180

Describe alternatives you've considered

Additional context

This should allow us to use more runner resources for longer multiarch builds.

graph TD;
    Workflow---|amd64|runner1;
    Workflow---|arm64|runner2;
    Workflow---|arm32v5|runner3;
    Workflow---|arm32v6|runner4;
    Workflow---|arm32v7|runner5;
    Workflow---|arm64v8|runner6;
    Workflow---|i386|runner7;
    Workflow---|ppc64le|runner8;
    Workflow---|s390x|runner9;
    runner1-->Z[podman push multi-arch-image]
    runner2-->Z
    runner3-->Z
    runner4-->Z
    runner5-->Z
    runner6-->Z
    runner7-->Z
    runner8-->Z
    runner9-->Z

Loading

[Raboo]

I was also caught with the same problem.

I am able to build multiple architectures. But not publish, it simply overwrites the tag so only one architecture is available in ghcr.io.

name: Build container images
on: [push, workflow_dispatch]

concurrency:
  group: '${{ github.event.pull_request.head.label || github.head_ref || github.ref }}'
  cancel-in-progress: true

jobs:
  build-image:
    name: Build image
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write
    strategy:
      matrix:
        platform: [linux/amd64, linux/arm64]

    steps:
    - name: Checkout code
      uses: actions/checkout@v3
      with:
        fetch-depth: 0
        submodules: true

    - name: cache podman
      uses: actions/cache@v3
      with:
        path: ~/.local/share/containers
        key: podman

    - name: Set up QEMU
      uses: docker/setup-qemu-action@v2

    - name: set lower case owner name
      run: |
        echo "OWNER_LC=${OWNER,,}" >>${GITHUB_ENV}
      env:
        OWNER: '${{ github.repository_owner }}'

    - name: Container meta
      id: meta
      uses: docker/metadata-action@v4
      with:
        images: |
          ghcr.io/${{ env.OWNER_LC }}/rtfm
        tags: |
          type=semver,pattern={{version}},value=${{ inputs.version }}
          type=semver,pattern={{major}}.{{minor}},value=${{ inputs.version }}
          type=semver,pattern={{major}},value=${{ inputs.version }}
          type=ref,event=branch
          type=ref,event=pr
          # type=raw,value=latest,enable=${{ github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/tags/') }}
        flavor: |
          latest=${{ github.ref_name == github.event.repository.default_branch }}

    - name: Build Image
      id: build-image
      uses: redhat-actions/buildah-build@v2
      with:
        image: rtfm
        tags: ${{ steps.meta.outputs.tags }}
        labels: ${{ steps.meta.outputs.labels }}
        oci: true
        containerfiles: |
          ./Dockerfile
        platforms: ${{ matrix.platform }}


    # I guess this is the part that needs some type of resource sharing between the runners.
    - name: Push image to GHCR
      uses: redhat-actions/push-to-registry@v2
      id: push
      with:
        image: ${{ steps.build-image.outputs.image }}
        tags: ${{ steps.build-image.outputs.tags }}
        username: ${{ github.actor }}
        password: ${{ github.token }}
        registry: ghcr.io/${{ env.OWNER_LC }}

    - name: Print image url
      run: echo "Image pushed to ${{ steps.push.outputs.registry-paths }}"

Anyone has a good solution for this?

[kaovilai]

you would have to create manifests list containing different (already pushed single-arch, built by different runners) tags then push the manifest list in the final runner

[Raboo]

Ok, I cracked it!

Here is a working example using multiple runners to build for each architecture, that later upload their images to github temporary storage and a runner that downloads the different architecture images and merges them and publishes to GHCR.

name: Build container images
on: [push, workflow_dispatch]

concurrency:
  group: '${{ github.event.pull_request.head.label || github.head_ref || github.ref }}'

jobs:
  build:
    name: Build image
    runs-on: ubuntu-latest
    strategy:
      matrix:
        platform: [linux/amd64, linux/arm64]

    steps:
    - name: Checkout code
      uses: actions/checkout@v3
      with:
        fetch-depth: 0
        submodules: true

    - name: cache podman storage
      uses: actions/cache@v3
      with:
        path: ~/.local/share/containers/storage
        key: podman-storage-${{ matrix.platform }}

    - name: Set up QEMU
      uses: docker/setup-qemu-action@v2

    - name: set lower case owner name
      run: |
        echo "OWNER_LC=${OWNER,,}" >>${GITHUB_ENV}
      env:
        OWNER: '${{ github.repository_owner }}'

    - name: export architecture name (removing os prefix)
      run: |
        echo "PODMAN_ARCH=${PLATFORM#*/}" >>${GITHUB_ENV}
      env:
        PLATFORM: ${{ matrix.platform }}

    - name: Container meta
      id: meta
      uses: docker/metadata-action@v4
      with:
        images: |
          ghcr.io/${{ env.OWNER_LC }}/rtfm
        tags: |
          type=raw,value=build
        flavor: |
          suffix=-${{ env.PODMAN_ARCH }}

    - name: Build Image
      id: build-image
      uses: redhat-actions/buildah-build@v2
      with:
        image: rtfm
        tags: ${{ steps.meta.outputs.tags }}
        labels: ${{ steps.meta.outputs.labels }}
        oci: true
        containerfiles: |
          ./Dockerfile
        platforms: ${{ matrix.platform }}

    - run: skopeo copy containers-storage:ghcr.io/${{ env.OWNER_LC }}/rtfm:build-${{ env.PODMAN_ARCH }} oci-archive:/tmp/${{ env.PODMAN_ARCH }}-oci.tar

    - name: Upload digest
      uses: actions/upload-artifact@v3
      with:
        name: rtfm-build-${{ env.PODMAN_ARCH }}
        path: /tmp/${{ env.PODMAN_ARCH }}-oci.tar
        if-no-files-found: error
        retention-days: 1

  upload:
    name: Upload images
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write
    needs: build
    steps:

    - name: Download podman
      uses: actions/download-artifact@v3
      with:
        name: rtfm-build-amd64
        path: /tmp
    - name: Download podman
      uses: actions/download-artifact@v3
      with:
        name: rtfm-build-arm64
        path: /tmp

    - name: set lower case owner name
      run: |
        echo "OWNER_LC=${OWNER,,}" >>${GITHUB_ENV}
      env:
        OWNER: '${{ github.repository_owner }}'

    - run: podman manifest create ghcr.io/${{ env.OWNER_LC }}/rtfm:manifest
    - run: podman manifest add ghcr.io/${{ env.OWNER_LC }}/rtfm:manifest oci-archive:/tmp/arm64-oci.tar
    - run: podman manifest add ghcr.io/${{ env.OWNER_LC }}/rtfm:manifest oci-archive:/tmp/amd64-oci.tar
    # - run: podman manifest inspect ghcr.io/${{ env.OWNER_LC }}/rtfm:manifest

    - name: Container meta
      id: meta
      uses: docker/metadata-action@v4
      with:
        images: |
          ghcr.io/${{ env.OWNER_LC }}/rtfm
        tags: |
          type=semver,pattern={{version}},value=${{ inputs.version }}
          type=semver,pattern={{major}}.{{minor}},value=${{ inputs.version }}
          type=semver,pattern={{major}},value=${{ inputs.version }}
          type=ref,event=branch
          type=ref,event=pr
          # type=raw,value=latest,enable=${{ github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/tags/') }}
        flavor: |
          latest=${{ github.ref_name == github.event.repository.default_branch }}

    - name: add tags
      run: |
        # fix multi-line issue from steps.meta.outputs.tags
        podman tag ghcr.io/${{ env.OWNER_LC }}/rtfm:manifest $(echo "${{ steps.meta.outputs.tags }}" | tr '\n' ' ')

    - name: Push image to GHCR
      uses: redhat-actions/push-to-registry@v2
      id: push
      with:
        image: rtfm
        tags: ${{ steps.meta.outputs.tags }}
        username: ${{ github.actor }}
        password: ${{ github.token }}
        registry: ghcr.io/${{ env.OWNER_LC }}

    - name: Print image url
      run: echo "Image pushed to ${{ steps.push.outputs.registry-paths }}"

I guess we now need to document this example somewhere.

[kaovilai]

Awesome!

[mikepurvis]

@Raboo Thanks for sharing your method. Unfortunately in my case my component images are ~2.5GB in size, so that adds about 3-4mins of round trip time uploading them and downloading them from the artifact registry; given that my overall pipeline is only about 20 mins total, I don't know that this is an acceptable performance cost.

I see some folks talking about performance in actions/upload-artifact#199, and one thing that's suggested as a faster option is https://github.com/actions/cache instead. But at a more fundamental level I wondered if something in the podman/skopeo ecosystem might permit me to push the component images directly to the registry with dummy tags, then create a master manifest without ever re-downloading anything. Just pull the two manifests, copy the relevant layer hash info into the new multiarch manifest, push. The registry will already have all the blobs for the arch images, so it shouldn't need to request anything further, and this would be way faster. The dummy tags can be deleted afterward or garbage collected in some parallel process.

Is there anything fundamental about how these manifests are that requires having all the content uploaded from the same point?

[kaovilai]

It should be possible to create manifest list without downloading iirc. Used to do in travisci.. at least using docker cli.

[mikepurvis]

Indeed yes, I just googled a bit more and it sounds like per this article the docker CLI can do it pretty easily:

docker manifest create andrewlockdd/alpine-clang:1.0 \
  --amend andrewlockdd/alpine-clang@sha256:038adbc4d6dc2e28f0818d5ae0fc1cae6cc42b854bd809f236435bed33f6ea63 \
  --amend andrewlockdd/alpine-clang@sha256:9df972530f876295787deea7424db90cbd14d5a8fa602b2a3bce82977aa1025e

I assume podman's version of manifest create works much the same way.

[Raboo]

But this still requires that the images are "local" and then pushed. So unless we can get around "matrix/platform". We still need to transfer the images into one runner.
If someone knows a better way I would love to see an example.

[mikepurvis]

Does it? My read is that those are references to already-pushed images, and the follow-on job is only pushing the unified manifest. Same with the example given in the Docker docs that that article links to, using docker buildx imagetools create:

https://docs.docker.com/build/ci/github-actions/multi-platform/#distribute-build-across-multiple-runners

[kaovilai]

From my original docker request :)
docker/docs#17180

[mikepurvis]

I've separately confirmed that the "legacy" docker manifest create/inspect/push suite of tools also do not require the images to be local. For the moment I'm sticking to those over the buildx imagetools option as I'm adapting this from a workflow that interleaves image building with interactive elements, and the buildx builder seems to fare very badly with that, insisting a bunch of extra shifting around of image blobs.