Loading image into a local registry is extremely slow on Github Actions - Sharing image between buildx jobs (3 min) - Alternatives?

[reteps]

Overview

I am trying to accomplish the following workflow:

• Build a set of base images, using a matrix builder for both x86 and arm
• Build a set of main images, using a matrix builder
• Upload the set of base / main images to dockerhub

Here are a few restrictions:

• This workflow should work on forks, even if the fork doesn't have access to push to dockerhub. Thus, I can't push the base image to dockerhub and pull it back down from a later job.
• The base image is ~2GB in size, so it is impractical to use the github cache storage backend for sharing the base image. We have other items in the cache that we wouldn't want to evict.

So circumvent these, I export the base image to a tar file, as outlined in 'Share image between jobs'. Since I am following the steps in 'Multi-platform image: Distribute build across multiple runners', I need to use the buildx backend to push by digest. Since I am using the buildx backend, I cannot simply load the base tarball into docker, and need to follow the steps in 'Named contexts: Using with a container builder' so that the local image is accessible to buildkit.

Problem

My base image takes about 5 minutes to build:

It takes an additional 3 minutes to load the image into the local registry using docker push ....

This is unacceptably slow, this should be 1 minute or less.

Full workflow

Workflow inputs

on:
  workflow_call:
    inputs:
      directory:
        description: 'Either "workspaces" or "graders"'
        required: true
        type: string
      image-prefix:
        description: 'The prefix for the image name (e.g. prairielearn/workspace-)'
        required: true
        type: string
      base-images:
        description: 'The base images to build'
        required: false
        type: string
        default: '[]'
      images:
        description: 'The images to build'
        required: true
        type: string
      upload-artifacts-for:
        description: 'The images to upload artifacts for'
        required: false
        type: string
        default: '[]'
      exclude:
        description: 'The combinations to exclude'
        required: false
        type: string
        default: '[]'

build-base-images

  build-base-images:
   if: ${{ inputs.base-images != '[]' }}
   strategy:
     max-parallel: 5
     matrix:
       image: ${{ fromJSON(inputs.base-images) }}
       os: [ubuntu-24.04, ubuntu-24.04-arm]
       exclude: ${{ fromJSON(inputs.exclude) }}
   runs-on: ${{ matrix.os }}
   env:
     CAN_PUSH_IMAGE: ${{ secrets.DOCKERHUB_USERNAME != '' }}
   steps:
     # Check out the repository with full depth so we can check if the workspace was modified
     - uses: actions/checkout@v4
       with:
         fetch-depth: 0

     - name: Check whether ${{ inputs.directory }}/${{ matrix.image }} was modified
       id: check-modified
       run: |
         scripts/check_path_modified.sh ${{ inputs.directory }}/${{ matrix.image }} ${{ inputs.directory }}_${{ matrix.image }}_modified

     - name: Set up Docker Buildx
       # Currently, we need this 'if' on each step. https://github.com/orgs/community/discussions/82744
       if: ${{ steps.check-modified.outputs.was-modified == 'true' }}
       uses: docker/setup-buildx-action@v3

     - name: Login to DockerHub
       if: ${{ steps.check-modified.outputs.was-modified == 'true' && env.CAN_PUSH_IMAGE == 'true' }}
       uses: docker/login-action@v3
       with:
         username: ${{ secrets.DOCKERHUB_USERNAME }}
         password: ${{ secrets.DOCKERHUB_TOKEN }}

     - name: Set docker tags and labels
       id: meta
       if: ${{ steps.check-modified.outputs.was-modified == 'true' }}
       uses: docker/metadata-action@v5
       with:
         context: git
         images: |
           ${{ inputs.image-prefix }}${{ matrix.image }}
         # On the default branch, we'll also tag the image as `latest`
         tags: |
           type=raw,value=latest,enable={{is_default_branch}}
           type=sha,prefix=,format=long

     - name: Build ${{ (env.CAN_PUSH_IMAGE == 'true' && 'and push') || '' }} digest
       id: build-digest
       if: ${{ steps.check-modified.outputs.was-modified == 'true' }}
       uses: docker/build-push-action@v6
       with:
         context: ${{ inputs.directory }}/${{ matrix.image }}
         labels: ${{ steps.meta.outputs.labels }}
         tags: ${{ inputs.image-prefix }}${{ matrix.image }}
         outputs: |
           type=image,push-by-digest=true,name-canonical=true,push=${{ env.CAN_PUSH_IMAGE == 'true' }}

     - name: Build tarball
       id: build
       if: ${{ steps.check-modified.outputs.was-modified == 'true' }}
       uses: docker/build-push-action@v6
       with:
         context: ${{ inputs.directory }}/${{ matrix.image }}
         labels: ${{ steps.meta.outputs.labels }}
         tags: ${{ steps.meta.outputs.tags }}
         outputs: |
           type=docker,dest=${{ runner.temp }}/${{ matrix.image }}-${{ matrix.os }}.tar

     # Group digests by image name. Tags are associated with each digest, but will be all merged when we push the image.
     - name: Export docker metadata
       if: ${{ steps.check-modified.outputs.was-modified == 'true' }}
       run: |
         mkdir -p ${{ runner.temp }}/digests/${{ matrix.image}}
         digest="${{ steps.build-digest.outputs.digest }}"
         echo "${{ steps.meta.outputs.tags }}" > "${{ runner.temp }}/digests/${{ matrix.image }}/${digest#sha256:}"

     # Workaround for https://github.com/orgs/community/discussions/17245
     - name: Upload docker metadata
       if: ${{ steps.check-modified.outputs.was-modified == 'true' }}
       uses: actions/upload-artifact@v4
       with:
         name: digests-${{ inputs.directory }}-${{ matrix.image }}-${{ matrix.os }}
         path: ${{ runner.temp }}/digests/
         if-no-files-found: error
         retention-days: 1
     # We will upload the tarball so we can use it in the next job
     - name: Upload docker tarball
       if: ${{ steps.check-modified.outputs.was-modified == 'true' }}
       uses: actions/upload-artifact@v4
       with:
         name: ${{ inputs.directory }}-${{ matrix.image }}-${{ matrix.os }}-tarball
         path: ${{ runner.temp }}/${{ matrix.image }}-${{ matrix.os }}.tar
         if-no-files-found: error
         retention-days: 1

build-images

  build-images:
    needs: [build-base-images]
    if: |
      always() && (needs.build-base-images.result == 'success' || needs.build-base-images.result == 'skipped')
    strategy:
      max-parallel: 5
      matrix:
        image: ${{ fromJSON(inputs.images) }}
        os: [ubuntu-24.04, ubuntu-24.04-arm]
        exclude: ${{ fromJSON(inputs.exclude) }}
    runs-on: ${{ matrix.os }}
    services:
      registry:
        image: registry:2
        ports:
          - 5000:5000
    env:
      CAN_PUSH_IMAGE: ${{ secrets.DOCKERHUB_USERNAME != '' }}

    # Associate each workspace with requirements
    steps:
      # Check out the repository with full depth so we can check if the workspace was modified
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Check whether ${{ inputs.directory }}/${{ matrix.image }} was modified
        id: check-modified
        run: |
          scripts/check_path_modified.sh ${{ inputs.directory }}/${{ matrix.image }} ${{ inputs.directory }}_${{ matrix.image }}_modified

      - name: Set up Docker Buildx
        if: ${{ steps.check-modified.outputs.was-modified == 'true' }}
        uses: docker/setup-buildx-action@v3
        with:
          # network=host driver-opt needed to push to local registry
          driver-opts: network=host

      - name: Login to DockerHub
        if: ${{ steps.check-modified.outputs.was-modified == 'true' && env.CAN_PUSH_IMAGE == 'true' }}
        uses: docker/login-action@v3
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}

      - name: Download artifact
        if: ${{ steps.check-modified.outputs.was-modified == 'true' }}
        uses: actions/download-artifact@v4
        # Download all uploaded artifacts that match our OS
        with:
          pattern: '${{ inputs.directory }}-*-${{ matrix.os }}-tarball'
          path: ${{ runner.temp }}
          merge-multiple: true

      - name: Load and tag base images
        id: tag-base
        if: ${{ steps.check-modified.outputs.was-modified == 'true' }}
        run: |
          build_contexts=""
          for file in ${{ runner.temp }}/*-${{ matrix.os }}.tar; do
            tagged_images=$(docker load -i "$file" | sed -n 's/Loaded image: //p')
            for tagged_image in $tagged_images; do
              image="${tagged_image%:*}"
              docker tag "$tagged_image" "localhost:5000/$tagged_image"
              docker push "localhost:5000/$tagged_image"
              build_contexts="$build_contexts$image=docker-image://localhost:5000/$tagged_image\n"
            done
          done
          {
            echo 'build-contexts<<EOF'
            echo -ne "$build_contexts"
            echo EOF
          } >> "$GITHUB_OUTPUT"
          echo "GITHUB OUTPUT---"
          cat "$GITHUB_OUTPUT"
          echo "---GITHUB OUTPUT"
          docker image ls -a

      - name: Set docker tags and labels
        id: meta
        if: ${{ steps.check-modified.outputs.was-modified == 'true' }}
        uses: docker/metadata-action@v5
        with:
          context: git
          images: |
            ${{ inputs.image-prefix }}${{ matrix.image }}
          # On the default branch, we'll also tag the image as `latest`
          tags: |
            type=raw,value=latest,enable={{is_default_branch}}
            type=sha,prefix=,format=long

      - name: Build ${{ (env.CAN_PUSH_IMAGE == 'true' && 'and push') || '' }} digest
        id: build-digest
        if: ${{ steps.check-modified.outputs.was-modified == 'true' }}
        uses: docker/build-push-action@v6
        with:
          context: ${{ inputs.directory }}/${{ matrix.image }}
          labels: ${{ steps.meta.outputs.labels }}
          tags: ${{ inputs.image-prefix }}${{ matrix.image }}
          build-contexts: ${{ steps.tag-base.outputs.build-contexts }}
          outputs: |
            type=image,push-by-digest=true,name-canonical=true,push=${{ env.CAN_PUSH_IMAGE == 'true' }}

      - name: Build tarball
        id: build
        if: ${{ steps.check-modified.outputs.was-modified == 'true' && contains(fromJson(inputs.upload-artifacts-for), format('{0}-{1}', matrix.image, matrix.os)) }}
        uses: docker/build-push-action@v6
        with:
          context: ${{ inputs.directory }}/${{ matrix.image }}
          labels: ${{ steps.meta.outputs.labels }}
          tags: ${{ steps.meta.outputs.tags }}
          outputs: |
            type=docker,dest=${{ runner.temp }}/${{ matrix.image }}-${{ matrix.os }}.tar

      - name: Export docker metadata
        if: ${{ steps.check-modified.outputs.was-modified == 'true' }}
        run: |
          mkdir -p ${{ runner.temp }}/digests/${{ matrix.image }}
          digest="${{ steps.build-digest.outputs.digest }}"
          echo "${{ steps.meta.outputs.tags }}" > "${{ runner.temp }}/digests/${{ matrix.image }}/${digest#sha256:}"

      - name: Upload docker metadata
        if: ${{ steps.check-modified.outputs.was-modified == 'true' }}
        uses: actions/upload-artifact@v4
        with:
          name: digests-${{ inputs.directory}}-${{ matrix.image }}-${{ matrix.os }}
          path: ${{ runner.temp }}/digests/
          if-no-files-found: error
          retention-days: 1

      - name: Upload docker tarball
        if: ${{ steps.check-modified.outputs.was-modified == 'true'  && contains(fromJson(inputs.upload-artifacts-for), format('{0}-{1}', matrix.image, matrix.os)) }}
        uses: actions/upload-artifact@v4
        with:
          name: ${{ inputs.directory }}-${{ matrix.image }}-${{ matrix.os }}-tarball
          path: ${{ runner.temp }}/${{ matrix.image }}-${{ matrix.os }}.tar
          if-no-files-found: error
          retention-days: 1

My question is:

1. Is there is some way to improve performance of sharing images between jobs using the buildx backend?
2. If no, if there is another way to do this sort of thing that is higher performance? My only alternative seems to be to add a tag-base-images step and share the base image between jobs by pushing up to dockerhub and pulling back down, and only using this workflow when we have push permission to dockerhub.

Similar links:

• How to use matrix for multi-platform builds? build-push-action#846 (comment)

[reteps]

I was able to circumvent this by pushing platform specific tags up instead of doing a push-by-digest.