[BUG]: Docker Inspect throws exit code 125 due to not finding some layer IDs in final built image.

[webjoaoneto]

New issue checklist

• I searched for existing GitHub issues
• I read pipeline troubleshooting guide
• I checked how to collect logs

Task name

Docker@2

Task version

2.243.0

Issue Description

When docker push after update version to 2.243.0 raises this error on Docker push pipeline

/usr/bin/*** inspect -f {{.RootFS.Layers}}
Error: no names or ids specified
##[error]Error: no names or ids specified
##[error]Unhandled: The process '/usr/bin/***' failed with exit code 125
##[error]Error: The process '/usr/bin/***' failed with exit code 125
    at ExecState._setResult (/opt/app-root/app/_work/_tasks/Docker_e28912f1-0114-4464-802a-a3a35437fd16/2.243.0/node_modules/azure-pipelines-task-lib/toolrunner.js:1249:25)
    at ExecState.CheckComplete (/opt/app-root/app/_work/_tasks/Docker_e28912f1-0114-4464-802a-a3a35437fd16/2.243.0/node_modules/azure-pipelines-task-lib/toolrunner.js:1232:18)
    at ChildProcess.<anonymous> (/opt/app-root/app/_work/_tasks/Docker_e28912f1-0114-4464-802a-a3a35437fd16/2.243.0/node_modules/azure-pipelines-task-lib/toolrunner.js:1160:19)
    at ChildProcess.emit (node:events:513:28)
    at maybeClose (node:internal/child_process:1100:16)
    at Process.ChildProcess._handle.onexit (node:internal/child_process:304:5)

The task pushes the docker image to the right place, but pipeline crashes because the command
docker inspect -f inspect -f {{.RootFS.Layers}} is not passing the image name as an next argument.

Fix:
We back to the version 2.240.2

Environment type (Please select at least one enviroment where you face this issue)

• Self-Hosted
• Microsoft Hosted
• VMSS Pool
• Container

Azure DevOps Server type

Azure DevOps Server (Please specify exact version in the textbox below)

Azure DevOps Server Version (if applicable)

No response

Operation system

ubuntu

Relevant log output

createdAt:2024-07-07T23:32:36Z; layerSize:9.54MB; createdBy:RUN /bin/sh -c set -eux; 	apt-get update; 	apt-get install -y --no-install-recommends 		ca-certificates 		netbase 		tzdata 	; 	rm -rf /var/lib/apt/lists/* # buildkit; layerId:<missing>
createdAt:2024-07-07T23:32:36Z; layerSize:0B; createdBy:ENV LANG=C.UTF-8; layerId:<missing>
createdAt:2024-07-07T23:32:36Z; layerSize:0B; createdBy:ENV PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin; layerId:<missing>
createdAt:2024-07-02T01:25:02Z; layerSize:0B; createdBy:/bin/sh -c #(nop)  CMD ["bash"]; layerId:<missing>
createdAt:2024-07-02T01:25:02Z; layerSize:77.8MB; createdBy:/bin/sh -c #(nop) ADD file:b24689567a7c604de93e4ef1dc87c372514f692556744da43925c575b4f80df6 in / ; layerId:<missing>
/usr/bin/*** inspect -f {{.RootFS.Layers}}
Error: no names or ids specified
##[error]Error: no names or ids specified
##[error]Unhandled: The process '/usr/bin/***' failed with exit code 125
##[error]Error: The process '/usr/bin/***' failed with exit code 125
    at ExecState._setResult (/opt/app-root/app/_work/_tasks/Docker_e28912f1-0114-4464-802a-a3a35437fd16/2.243.0/node_modules/azure-pipelines-task-lib/toolrunner.js:1249:25)
    at ExecState.CheckComplete (/opt/app-root/app/_work/_tasks/Docker_e28912f1-0114-4464-802a-a3a35437fd16/2.243.0/node_modules/azure-pipelines-task-lib/toolrunner.js:1232:18)
    at ChildProcess.<anonymous> (/opt/app-root/app/_work/_tasks/Docker_e28912f1-0114-4464-802a-a3a35437fd16/2.243.0/node_modules/azure-pipelines-task-lib/toolrunner.js:1160:19)
    at ChildProcess.emit (node:events:513:28)
    at maybeClose (node:internal/child_process:1100:16)
    at Process.ChildProcess._handle.onexit (node:internal/child_process:304:5)
Finishing: Docker Push

Full task logs with system.debug enabled

 [REPLACE THIS WITH YOUR INFORMATION] 

Repro steps

No response

[flinox-testes]

Exactly same problem here... the version 2.240.2 it works.

[dcs-adam]

Same issue here. Error was there on 2.240.2, but allowed the task to complete successfully. On 2.243.0, the image is pushed to the registry, but the pipeline fails.

[MarkKharitonov]

Same issue here as well.
Does anyone know if it is possible to invoke the previous version of the task?

[MarkKharitonov]

Fix:
We back to the version 2.240.2

@webjoaoneto - how did you go back to 2.240.2 ?

[chrislanzara]

+1 on this. It's now halted our build pipelines for our projects.

We were on version 2.240.2 up to this afternoon (around midday - 12pm - UK time on the 24th July), then we seem to have gone to 2.243.0 and the stage fails now, but the image is pushed to the ACR. No changes to our docker config, azure pipeline config etc, this seems to be handled by the Docker@2 stage alone.

          - task: Docker@2
            displayName: Push Container to ACR
            continueOnError: false
            inputs:
              command: push
              repository: $(imageName)
              tags: $(tag)
              containerRegistry: dockerRegistryServiceConnection

I did see something like this a while ago (few months back), a very similar issue on the push command, but by the time I went back to it the issue had resolved itself and the stage was successful.

==============================================================================
Task         : Docker
Description  : Build or push Docker images, login or logout, start or stop containers, or run a Docker command
Version      : 2.243.0
Author       : Microsoft Corporation
Help         : https://aka.ms/azpipes-docker-tsg

...

/usr/bin/docker inspect -f {{.RootFS.Layers}}
"docker inspect" requires at least 1 argument.
See 'docker inspect --help'.

Usage:  docker inspect [OPTIONS] NAME|ID [NAME|ID...]

Return low-level information on Docker objects
##[error]"docker inspect" requires at least 1 argument.
##[error]See 'docker inspect --help'.
##[error]Usage:  docker inspect [OPTIONS] NAME|ID [NAME|ID...]
##[error]Return low-level information on Docker objects
##[error]Unhandled: The process '/usr/bin/docker' failed with exit code 1
##[error]Error: The process '/usr/bin/docker' failed with exit code 1
    at ExecState._setResult (/home/vsts/work/_tasks/Docker_e28912f1-0114-4464-802a-a3a35437fd16/2.243.0/node_modules/azure-pipelines-task-lib/toolrunner.js:1249:25)
    at ExecState.CheckComplete (/home/vsts/work/_tasks/Docker_e28912f1-0114-4464-802a-a3a35437fd16/2.243.0/node_modules/azure-pipelines-task-lib/toolrunner.js:1232:18)
    at ChildProcess.<anonymous> (/home/vsts/work/_tasks/Docker_e28912f1-0114-4464-802a-a3a35437fd16/2.243.0/node_modules/azure-pipelines-task-lib/toolrunner.js:1160:19)
    at ChildProcess.emit (node:events:513:28)
    at maybeClose (node:internal/child_process:1100:16)
    at Process.ChildProcess._handle.onexit (node:internal/child_process:304:5)
Finishing: Push Container to ACR

I think the issues around docker inspect are a bit misleading for this directly because looking at our runs from earlier today when 2.240.2 was used, we got the same errors about the docker inspect command but the stage was allowed to complete, whereas under 2.243.0 the same docker inspect errors appear but then the Unhandled message appears and this then causes the stage to fail.

This is from a run where the stage completed successfully:

==============================================================================
Task         : Docker
Description  : Build or push Docker images, login or logout, start or stop containers, or run a Docker command
Version      : 2.240.2
Author       : Microsoft Corporation
Help         : https://aka.ms/azpipes-docker-tsg

...


/usr/bin/docker inspect -f {{.RootFS.Layers}}
"docker inspect" requires at least 1 argument.
See 'docker inspect --help'.

Usage:  docker inspect [OPTIONS] NAME|ID [NAME|ID...]

Return low-level information on Docker objects
##[error]"docker inspect" requires at least 1 argument.
##[error]See 'docker inspect --help'.
##[error]Usage:  docker inspect [OPTIONS] NAME|ID [NAME|ID...]
##[error]Return low-level information on Docker objects
Finishing: Push Container to ACR

Oddly this is only failing for a pipeline building an Angular image. We run 2.243.0 when building a C# API and that runs fine (the underscores are my own to shorten the lines):

Starting: ACR Push
==============================================================================
Task         : Docker
Description  : Build or push Docker images, login or logout, start or stop containers, or run a Docker command
Version      : 2.243.0
Author       : Microsoft Corporation
Help         : https://aka.ms/azpipes-docker-tsg

...

createdAt:2024-07-23T05:24:***5Z; layerSize:74.8MB; createdBy:/bin/sh -c #(nop) ADD file:6c4730e7b**___ebfb56a602 in / ; layerId:<missing>
/usr/bin/docker inspect 02de***9c***4fb___7d6bef3acb5b7*** -f {{.RootFS.Layers}}
[sha256:e078***___4***e***f sha256:2ea3b5___***2f sha256:ad8af89334___07fc206***9fc sha256:855e5***907d3ec93___78a***324c sha256:58fa834ef___9657363ae sha256:698640980e___35b248c5e sha256:5e***ee***___5f***50***20359f sha256:a2980f6c44a___98722208da sha256:e45***75fc8___d44d0a8a7a6206 sha256:f04d0d2___dfe867df***]
Finishing: ACR Push

Anyone able to help or is there any way to force the stage to use the previous 2.240.2 version?

Thanks.

[MarkKharitonov]

OK, found it.
It is actually straightforward to use the older version, just use Docker@2.240.2

[lucasrcorreia]

I had the same problem here and the solution was to force the previous minor version in the yaml, simply by changing the code from:

- task: Docker@2

to:

- task: Docker@2.240.2

[chrislanzara]

OK, found it.

It is actually straightforward to use the older version, just use Docker@2.240.2

This worked! Thank you so much. One to remember for the future too...