How to troubleshoot, getting stuck after first line of `...Storing image for indexing`

[MPV]

My job gets stuck/fails (repeatedly after reruns/retries) after outputting:
...Storing image for indexing

I'm using it like this:

name: Docker image

on:
  push:
    branches:
      - master
  pull_request:
  merge_group:

env:
  IMAGE: my.regist.ry/my/image

jobs:
  build:
    runs-on: ubuntu-latest
    name: Build & test
    permissions:
      contents: read
      id-token: write
      statuses: write
      checks: write
      pull-requests: write
    steps:

      - name: Setup Docker (Buildx)
        uses: docker/setup-buildx-action@f95db51fddba0c2d1ec667646a06c2ce06100226 # v3

      # [...]

      - name: Setup Docker (Docker Hub login, for Docker Scout)
        uses: docker/login-action@343f7c4344506bcbf9b4de18042ae17996df046d # v3
        with:
          username: ${{ vars.SECRET_USERNAME }}
          password: ${{ secrets.SECRET_TOKEN }}

      - name: Build image
        uses: docker/build-push-action@0565240e2d4ab88bba5387d719585280857ece09 # v5
        with:
          push: false
          tags: ${{ env.IMAGE }}:${{ github.sha }},${{ env.IMAGE }}:latest
          cache-from: type=gha
          cache-to: type=gha,mode=max
          load: true # so we can run tests on the image, here in the same workflow

      # [...]

      - name: Docker Scout
        if: ${{ github.event_name == 'pull_request' && !cancelled() }}
        uses: docker/scout-action@b7413c99043c2a9131c0fa39cedaece80f285788 # v1.2.2
        with:
          command: compare
          image: ${{ env.IMAGE }}:${{ github.sha }}
          to: ${{ env.IMAGE }}:latest
          ignore-unchanged: true
          only-severities: critical,high
          write-comment: true

And this is what it looks like when it gets stuck:

Followed by:
The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.

[MPV]

The image is semi-large (and not a target for downsizing at the moment, as it's a GitHub Actions runner image with preinstalled software), weighing in at just above 1.5 GBs.

As shown in the Build image step output:

[...]
#17 sha256:bc5980436d16ba698a1ff2c17f41ebe584924f1011129e328dca4250cf87e7de 1.58GB / 1.58GB 36.1s done
#17 DONE 36.5s

[MPV]

By enabling debugging with...

# [...]
with: 
  # [...]
  debug: true
  verbose-debug: true

...I now get:

      ...Storing image for indexing
  DEBUG copying image to /tmp/docker-scout/sha256/28eea48eb8069e86aaeb059e0c184213495f01f22b705fe50dc78e57c05e4c8e/d43bdf25-9519-41fc-9752-d258de503d3a
  Error: The operation was canceled.

[MPV]

I am trying to reproduce it locally using the docker scout compare CLI command:

$ docker scout compare \
		--ignore-unchanged \
		--to registry://my.regist.ry/my/image:latest \
		local://my.regist.ry/my/image:my-pr-image

...but (while it is slow) it works for me locally (taking around 10 minutes to complete the first time).

On the second local attempt the SBOM seems cached and it only took 12 seconds.

[MPV]

It doesn't seem like I would be exceeding the timeouts of GitHub Actions:

• https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstepstimeout-minutes

[MPV]

When rerunning I am getting different variants, either:

The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.

with:

---

Or also:

The operation was canceled. as per below:

with:

[cdupuis]

@MPV thanks for raising this detailed issue. Unfortunately, I'm not able to re-create this issue myself.

Would it be possible for you to create a sample repo that re-creates this issue?

[MPV]

I have a theory that this kind of error in GHA could potentially happen when the runner runs out of memory (process oom killed) or out of disk. I'll see if I can dig some more.

Did you find any large images to try reproducing it with? I have yet to look and find large (1-2GB+) public ones.

[MPV]

Just to confirm, I've been able to solve/workaround the issues seen above by running on a larger runner.

The default ubuntu-latest runners only have 10 GB disk, so a large-enough image being first built and then Docker Scout'ed did lead to the disk filled up and thus the runner job aborted in the ways seen above.

[MPV]

I am trying to reproduce it locally using the docker scout compare CLI command:

$ docker scout compare \

		--ignore-unchanged \

		--to registry://my.regist.ry/my/image:latest \

		local://my.regist.ry/my/image:my-pr-image

...but (while it is slow) it works for me locally (taking around 10 minutes to complete the first time).

On the second local attempt the SBOM seems cached and it only took 12 seconds.

Is there a way to let Docker Scout check an SBOM instead of making (yet another local disk) copy of the image?

EDIT: it seems docker scout sbom can generate such an SBOM too. Though as I'm already doing this in another tool/step, how might I instead get this Scout action to scan an SBOM instead of (duplicating and using) the image?

I raised a separate upstream issue on that:

• Compare/analyze SBOMs scout-cli#97