getting ssh-add -l : error fetching identities: communication with agent failed (from within devcontainer with common-utils/docker-from-docker)

[hpe-ykoehler]

Hi,

I created a devcontainer using

{
    "dockerComposeFile": "docker-compose.yml",
    "service": "krypton-c-env",
    "workspaceFolder": "/work/libaos",
    "shutdownAction": "stopCompose",
    "remoteUser": "aos-dev",
    "features": {
        "ghcr.io/devcontainers/features/common-utils:2": {
            "username": "aos-dev",
            "uid": "10000",
            "gid": "10001",
            "installZsh": true,
            "installOhMyZsh": true,
            "upgradePackages": true,
            "nonFreePackages": false
        },
        "ghcr.io/devcontainers/features/docker-from-docker:1": {
             "version": "latest",
             "username": "aos-dev"
        }
    },
    "postCreateCommand": "conan profile detect",
    "extensions": [
        "atlassian.atlascode", "ms-vscode.cpptools", "numaru.vscode-ceedling-test-adapter",
        "twxs.cmake", "ms-azuretools.vscode-docker", "ms-vscode.cpptools-extension-pack",
        "ban.troff", "SonkengMaldini.conanlight", "afri-bit.vsconan",
        "ryanluker.vscode-coverage-gutters"
    ]
}

I connect remotely to my linux vm within VSCode and ssh-add -l works, then I go within the devcontainer and ssh-add -l now report the error, causing all my git operation with the remote to fail.

I do see an output for "echo $SSH_AUTH_SOCK" on both ssh remote and within the devcontainer, but likely I am missing a mount of a particular socket as to enable ssh-agent communication between host and the devcontainer environment.

I am unclear if this is a mistake of mine, or something broken by common-utils.

[samruddhikhandale]

Hi 👋

ghcr.io/devcontainers/features/common-utils:2

The major version of the common-utils Feature was bumped as two options were renamed (namely uid to userUid & gid to userGid ) due to a bug. Can you update these options in your dev config and see if it works? thanks!

Details -> https://github.com/devcontainers/features/tree/main/src/common-utils#options

I am unclear if this is a mistake of mine, or something broken by common-utils.

There was a recent refactoring of the common-utils Feature #326. If updating options still does not help, then can you confirm if pinning the version to older version works? ( ie ghcr.io/devcontainers/features/common-utils:1 ).

[hpe-ykoehler]

Hi,

I have renamed the options in my devcontainer.json file but I still, see the same problem. If I create a file, I do see the correct uid/gid assigned to it.

Yet thanks for this info, I had failed to notice the change in the option keyword.

[samruddhikhandale]

I see. According to my above comment, would you mind trying to see if things work when you pin to major version 1 (and you will need to update the options again).

I would like to understand if there was a regression with the refactoring or if it's something else.

Also, can you provide your machine details -

• Machine OS
• VS Code version
• Dev Container Extension Version

[hpe-ykoehler]

I have changed the devcontainer.json to use the common-utils:1

        "ghcr.io/devcontainers/features/common-utils:1": {
            "username": "aos-dev",
            "uid": "10000",
            "gid": "10001",
            "installZsh": false,
            "installOhMyZsh": false,
            "upgradePackages": true,
            "nonFreePackages": false
        },

I am still seeing the ssh-add -l (and git fetch) issue.

So that seems to rule out regression.

I will try to remove the docker-from-docker feature and see if I can repro still.

I am wondering if I need to add any special mount in my docker-compose:

version: '3'
services:
  krypton-c-env:
    build: .

    init: true
    privileged: true
    
    # Mounts the project folder to '/workspace'. While this file is in .devcontainer,
    # mounts are relative to the first file in the list, which is a level up.
    volumes:
    - ..:/work/libaos:cached

    # [Optional] Required for ptrace-based debuggers like C++, Go, and Rust
    cap_add:
      - SYS_PTRACE
    security_opt:
      - seccomp:unconfined

#    user: aos-dev
    command: sleep infinity

I am using VSCode on my Windows 10

1.74.2
e8a3071ea4344d9d48ef8a4df2c097372b0c5161
x64

dev container version is v0.266.1

I connect to SSH to my linux workstation running Ubuntu 20.04 LTS (updated to recent packages)
docker is then run on that linux machine.

ssh-agent is also running only on that linux machine.
the git key is on that machine and accessible to the user ykoehler (different uid/gid), password protected and I did add it to my ssh-agent using ssh-add.

[hpe-ykoehler]

when I connect with "Reopen in SSH folder" I can see both ssh-add -l and git working properly.

[hpe-ykoehler]

oh, scratch that last comment, on that machine with reopen in SSH I have not seeing success. So this may not have anything to do with devcontainer... I have 2 linux machine nearly the same name and may have confuse both.

[hpe-ykoehler]

ok... so I got the ssh-agent stuff to work with this machine, then I was able to use ssh-add -l and git command in "ssh" mode, but still got the same failure with devcontainer.

So back to the initial problem, next step will be to get rid of docker-from-docket and see if it still occurs.

[hpe-ykoehler]

I removed the docker-from-docker feature , still no success, I am unclear where to look next.

[samruddhikhandale]

Thanks for all the details.

Looking at your dev config in #352 (comment), I see finer things which should be changed/added.

	"workspaceFolder": "/workspaces/${localWorkspaceFolderBasename}",

	// Use this environment variable if you need to bind mount your local source code into a new container.
	"remoteEnv": {
		"LOCAL_WORKSPACE_FOLDER": "${localWorkspaceFolder}"
	},

Looking at your docker-compose in #352 (comment)

I don't see either a Dockerfile or an image reference. Also, wondering if the issue is due to a missing entrypoint and a missing volumes:/var/run/docker.sock:/var/run/docker-host.sock

ℹ️ @hpe-ykoehler Have you taken a look at the docker-outside-of-docker-compose Template ? It is a good reference for the scenario you are trying to describe.

[hpe-ykoehler]

it is my understanding that the feature docker-from-docker adds the bind and entrypoint statements, as such I do not need to.

I did the following modification based on your comments:

devcontainer.json:

{
    "dockerComposeFile": "docker-compose.yml",
    "service": "krypton-c-env",
    "workspaceFolder": "/work/libaos",
    "shutdownAction": "stopCompose",
    "remoteUser": "aos-dev",
    "remoteEnv": {
        "LOCAL_WORKSPACE_FOLDER": "${localWorkspaceFolder}"
    },
    "features": {
        "ghcr.io/devcontainers/features/common-utils:1": {
            "username": "aos-dev",
            "uid": "10000",
            "gid": "10001",
            "installZsh": false,
            "installOhMyZsh": false,
            "upgradePackages": true,
            "nonFreePackages": false
        },
        "ghcr.io/devcontainers/features/docker-from-docker:1": {

        }
    },
    "postCreateCommand": "conan profile detect && ./build.sh",
    "extensions": [
        "atlassian.atlascode", "ms-vscode.cpptools", "numaru.vscode-ceedling-test-adapter",
        "twxs.cmake", "ms-azuretools.vscode-docker", "ms-vscode.cpptools-extension-pack",
        "ban.troff", "SonkengMaldini.conanlight", "afri-bit.vsconan",
        "ryanluker.vscode-coverage-gutters"
    ]
}

docker-compose.yml

version: '3'
services:
  krypton-c-env:
    build: .

    init: true
    privileged: true
    
    # Mounts the project folder to '/workspace'. While this file is in .devcontainer,
    # mounts are relative to the first file in the list, which is a level up.
    volumes:
    - ${LOCAL_WORKSPACE_FOLDER:-../}:/work/libaos

    # [Optional] Required for ptrace-based debuggers like C++, Go, and Rust
    cap_add:
      - SYS_PTRACE
    security_opt:
      - seccomp:unconfined

#    user: aos-dev
    command: sleep infinity

Dockerfile

FROM ubuntu:20.04

# Enable new "BUILDKIT" mode for Docker CLI
ENV DOCKER_BUILDKIT=1

# Remove apt-get prompt for timezone
ENV TZ=America/Los_Angeles
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone

RUN export DEBIAN_FRONTEND=noninteractive \
    && apt-get update && apt-get install -y --no-install-recommends \
        build-essential \
        git \
        ruby \
        bash-completion \
        wget \
        ca-certificates \
        python3-pip \
        clang-tidy \
    # Clean up
    && apt-get autoremove -y && apt-get clean -y && rm -rf /var/lib/apt/lists/* /tmp/library-scripts/

# Install kitware repo to get latest CMake (required wget)
RUN wget -qO /etc/apt/trusted.gpg.d/kitware-key.asc https://apt.kitware.com/keys/kitware-archive-latest.asc \
    && echo "deb https://apt.kitware.com/ubuntu/ focal main" | tee /etc/apt/sources.list.d/kitware.list \
    export DEBIAN_FRONTEND=noninteractive \
    && apt-get update && apt-get install -y --no-install-recommends \
        cmake \
    # Clean up
    && apt-get autoremove -y && apt-get clean -y && rm -rf /var/lib/apt/lists/* /tmp/library-scripts/

# Install ninja to get fast build
RUN wget -qO /usr/local/bin/ninja.gz https://github.com/ninja-build/ninja/releases/latest/download/ninja-linux.zip \
    && gunzip /usr/local/bin/ninja.gz \
    && chmod a+x /usr/local/bin/ninja

# Install ceedling (required ruby) to run automated tests
RUN gem install ceedling

# Install conan
RUN pip install conan==2.0.0b6

[samruddhikhandale]

I connect remotely to my linux vm within VSCode and ssh-add -l works, then I go within the devcontainer and ssh-add -l now report the error, causing all my git operation with the remote to fail.

I do see an output for "echo $SSH_AUTH_SOCK" on both ssh remote and within the devcontainer, but likely I am missing a mount of a particular socket as to enable ssh-agent communication between host and the devcontainer environment.

Could you help me out with your exact repro steps with commands used for connecting another container etc? Also, I am trying to understand your scenario and wondering if using docker-in-docker might help as docker-from-docker is used in advance scenarios.

When I run ssh-add -l in the container, I get The agent has no identities. which seems correct as nothing added the ssh keys to it. However, I am missing the build.sh file mentioned in your postCreateCommand.

[hpe-ykoehler]

The build.sh is a bash script that build my project (mkdir build && cd build && cmake .. && make, etc)

As for the docker-in-docker or docker-from-docker so far I can remove that feature and still get the same issue.

The command to connect to the other container? I used the VSCode software which does everything for me, I open up a folder on a remote ssh location and then I click reopen in container.

[samruddhikhandale]

Have you looked at https://code.visualstudio.com/remote/advancedcontainers/develop-remote-host#_connect-using-the-remote-ssh-extension-recommended ?

[hpe-ykoehler]

Yes, this is already in place and that worked nicely, except git access.