Build action cannot find local image

[Satont]

I'm trying to build base at one job, then transfer that image to other jobs.

But for some reason when building docker saying image not found after restore, even previous step Load base images showed that image as existed.

Load:

Run docker load --input /tmp/builder/builder.tar &
Loaded image: twir-base:go_prod_base
Loaded image: twir-base:node_prod_base
Loaded image: twir-base:builder
REPOSITORY       TAG               IMAGE ID       CREATED         SIZE
twir-base        builder           1bec39f04577   4 minutes ago   2.36GB
twir-base        go_prod_base      c351b08e3f2f   2 hours ago     18.6MB
twir-base        node_prod_base    b8e24b10e8bf   2 hours ago     211MB

Build:

Dockerfile:1
--------------------
   1 | >>> FROM twir-base:builder as builder
   2 |     RUN pnpm turbo run build --filter=@twir/grpc
   3 |     RUN cd apps/api && \
--------------------
ERROR: failed to solve: twir-base:builder: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
Error: buildx failed with: ERROR: failed to solve: twir-base:builder: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

Workflow file: https://github.com/Satont/twir/actions/runs/6882663892/workflow

[crazy-max]

https://github.com/Satont/twir/actions/runs/6882663892/workflow#L62-L63

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

You're using the setup-buildx-action in the other job and therefore it will create a container builder which does not have access to the docker store where your image is loaded.

What you can do is using a named context: https://docs.docker.com/build/ci/github-actions/named-contexts/#using-with-a-container-builder

Would be smth like that in your case:

  backend:
    runs-on: ubuntu-latest
    needs:
      - base
    services:
      registry:
        image: registry:2
        ports:
          - 5000:5000
    strategy:
      matrix:
        target:
          - api
          - bots
          - discord
          - emotes-cacher
          - eval
          - events
          - eventsub
          - integrations
          - language-detector
          - parser
          - scheduler
          - timers
          - tokens
          - websockets
          - ytsr
    steps:
      - uses: actions/checkout@v3
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
        with:
          driver-opts: network=host

      - name: Restore base images
        uses: actions/download-artifact@v3
        with:
          path: /tmp

      - name: Load base images
        run: |
          docker load --input /tmp/builder/builder.tar &
          docker load --input /tmp/node_prod_base/node_prod_base.tar &
          docker load --input /tmp/go_prod_base/go_prod_base.tar &
          wait
          docker image ls
          docker tag twir-base:builder localhost:5000/twir-base:builder
          docker push localhost:5000/twir-base:builder

      - name: Build ${{ matrix.target }}
        uses: docker/build-push-action@v5
        with:
          context: .
          build-contexts: |
            builder=docker-image://localhost:5000/twir-base:builder
          push: false
          load: true
          file: ./apps/${{ matrix.target }}/Dockerfile
          tags: twir:${{ matrix.target }}
          cache-from: type=gha,scope=twir:${{ matrix.target }}
          cache-to: type=gha,mode=max,scope=twir:${{ matrix.target }}

But I wonder why you do this and don't rely instead on GitHub Cache backend for your base images? You would not need to upload/download artifacts and just set cache-from accordingly:

      - name: Build ${{ matrix.target }}
        uses: docker/build-push-action@v5
        with:
          context: .
          build-contexts: |
            builder=docker-image://localhost:5000/twir-base:builder
          push: false
          load: true
          file: ./apps/${{ matrix.target }}/Dockerfile
          tags: twir:${{ matrix.target }}
          cache-from: |
            type=gha,scope=twir-base:builder
            type=gha,scope=twir-base:node_prod_base
            type=gha,scope=twir-base:go_prod_base
            type=gha,scope=twir:${{ matrix.target }}
          cache-to: type=gha,mode=max,scope=twir:${{ matrix.target }}

[crazy-max]

@dvdksn I think we could add this case in https://docs.docker.com/build/ci/github-actions/share-image-jobs/

[crazy-max]

But I wonder why you do this and don't rely instead on GitHub Cache backend for your base images? You would not need to upload/download artifacts and just set cache-from accordingly:

Oh actually this would not work as you're using another Dockerfile named base.Dockerfile.

[Satont]

@crazy-max

But I wonder why you do this and don't rely instead on GitHub Cache backend for your base images? You would not need to upload/download artifacts and just set cache-from accordingly:

You mean even w/o registry service? Without restoring it to local registry it wont work, right? So i'm anyway need to upload/download

That one doesnt works (https://github.com/Satont/twir/actions/runs/6891266755/workflow):

      - name: Build ${{ matrix.target }}
        uses: docker/build-push-action@v5
        with:
          context: .
          push: false
          file: ./apps/${{ matrix.target }}/Dockerfile
          tags: twir:${{ matrix.target }}
          cache-from: |
            type=gha,scope=twir-base:builder
            type=gha,scope=twir-base:node_prod_base
            type=gha,scope=twir-base:go_prod_base
            type=gha,scope=twir:${{ matrix.target }}
          cache-to: type=gha,mode=max,scope=twir:${{ matrix.target }}

[Satont]

Oh actually this would not work as you're using another Dockerfile named base.Dockerfile.

Currently i'm using multi-stage dockerfile. But it starting to feels unmaintainable for me.

What i'm trying to do is same multistage, but base splitted to own docker file, and each service have own docker file. If you can suggest other ways i can achieve that i'll appreciate that.

[Satont]

https://github.com/Satont/twir/actions/runs/6897143191/workflow

Seems like i'm done that, but upload/download actions takes kind of annoying time.
As i'm understand this cannot be improved by me and feels like i need to keep my actions on multistage file. Correct?