`docker-in-docker` fail to start with amd64 image on macOS M1

[zephyros-dev]

Description

• dockerd installed using docker-in-docker features fails to run on macOS M1 machine with amd64 image
• The features works normally using arm64 image

Steps to reproduce

1. Build .devcontainer from the following file:
   devcontainer.json

{
    "name": "DIND container",
    "dockerFile": "Dockerfile",
    "containerUser": "node",
    "remoteUser": "node",
    "features": {
        "ghcr.io/devcontainers/features/docker-from-docker:1": {},
    }
}

Dockerfile

FROM --platform=linux/amd64 mcr.microsoft.com/devcontainers/typescript-node:16

Issues

• dockerd is unable to start. Here's the logs if I run sudo dockerd:

INFO[2022-10-13T03:09:21.395301838Z] Starting up                                  
INFO[2022-10-13T03:09:21.412464546Z] libcontainerd: started new containerd process  pid=3080
INFO[2022-10-13T03:09:21.414531504Z] parsed scheme: "unix"                         module=grpc
INFO[2022-10-13T03:09:21.414705338Z] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2022-10-13T03:09:21.415490796Z] ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}  module=grpc
INFO[2022-10-13T03:09:21.415749879Z] ClientConn switching balancer to "pick_first"  module=grpc
WARN[0000] containerd config version `1` has been deprecated and will be removed in containerd v2.0, please switch to version `2`, see https://github.com/containerd/containerd/blob/main/docs/PLUGINS.md#version-header 
INFO[2022-10-13T03:09:21.657304379Z] starting containerd                           revision=a17ec496a95e55601607ca50828147e8ccaeebf1 version=1.5.13+azure-2
INFO[2022-10-13T03:09:21.775481546Z] loading plugin "io.containerd.content.v1.content"...  type=io.containerd.content.v1
INFO[2022-10-13T03:09:21.776213463Z] loading plugin "io.containerd.snapshotter.v1.aufs"...  type=io.containerd.snapshotter.v1
INFO[2022-10-13T03:09:21.777676171Z] skip loading plugin "io.containerd.snapshotter.v1.aufs"...  error="aufs is not supported (modprobe aufs failed: exec: \"modprobe\": executable file not found in $PATH \"\"): skip plugin" type=io.containerd.snapshotter.v1
INFO[2022-10-13T03:09:21.778019421Z] loading plugin "io.containerd.snapshotter.v1.btrfs"...  type=io.containerd.snapshotter.v1
INFO[2022-10-13T03:09:21.778673296Z] skip loading plugin "io.containerd.snapshotter.v1.btrfs"...  error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs (ext4) must be a btrfs filesystem to be used with the btrfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
INFO[2022-10-13T03:09:21.778758379Z] loading plugin "io.containerd.snapshotter.v1.devmapper"...  type=io.containerd.snapshotter.v1
WARN[2022-10-13T03:09:21.779001879Z] failed to load plugin io.containerd.snapshotter.v1.devmapper  error="devmapper not configured"
INFO[2022-10-13T03:09:21.779105296Z] loading plugin "io.containerd.snapshotter.v1.native"...  type=io.containerd.snapshotter.v1
INFO[2022-10-13T03:09:21.779538629Z] loading plugin "io.containerd.snapshotter.v1.overlayfs"...  type=io.containerd.snapshotter.v1
INFO[2022-10-13T03:09:21.781382129Z] loading plugin "io.containerd.snapshotter.v1.zfs"...  type=io.containerd.snapshotter.v1
INFO[2022-10-13T03:09:21.781861588Z] skip loading plugin "io.containerd.snapshotter.v1.zfs"...  error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
INFO[2022-10-13T03:09:21.781940254Z] loading plugin "io.containerd.metadata.v1.bolt"...  type=io.containerd.metadata.v1
WARN[2022-10-13T03:09:21.782172754Z] could not use snapshotter devmapper in metadata plugin  error="devmapper not configured"
INFO[2022-10-13T03:09:21.782304463Z] metadata content store policy set             policy=shared
INFO[2022-10-13T03:09:21.785518338Z] loading plugin "io.containerd.differ.v1.walking"...  type=io.containerd.differ.v1
INFO[2022-10-13T03:09:21.785782713Z] loading plugin "io.containerd.gc.v1.scheduler"...  type=io.containerd.gc.v1
INFO[2022-10-13T03:09:21.787059838Z] loading plugin "io.containerd.service.v1.introspection-service"...  type=io.containerd.service.v1
INFO[2022-10-13T03:09:21.788375129Z] loading plugin "io.containerd.service.v1.containers-service"...  type=io.containerd.service.v1
INFO[2022-10-13T03:09:21.788565129Z] loading plugin "io.containerd.service.v1.content-service"...  type=io.containerd.service.v1
INFO[2022-10-13T03:09:21.788827463Z] loading plugin "io.containerd.service.v1.diff-service"...  type=io.containerd.service.v1
INFO[2022-10-13T03:09:21.789024129Z] loading plugin "io.containerd.service.v1.images-service"...  type=io.containerd.service.v1
INFO[2022-10-13T03:09:21.789192338Z] loading plugin "io.containerd.service.v1.leases-service"...  type=io.containerd.service.v1
INFO[2022-10-13T03:09:21.789330088Z] loading plugin "io.containerd.service.v1.namespaces-service"...  type=io.containerd.service.v1
INFO[2022-10-13T03:09:21.789434338Z] loading plugin "io.containerd.service.v1.snapshots-service"...  type=io.containerd.service.v1
INFO[2022-10-13T03:09:21.789628213Z] loading plugin "io.containerd.runtime.v1.linux"...  type=io.containerd.runtime.v1
INFO[2022-10-13T03:09:21.790231171Z] loading plugin "io.containerd.runtime.v2.task"...  type=io.containerd.runtime.v2
INFO[2022-10-13T03:09:21.791182546Z] loading plugin "io.containerd.monitor.v1.cgroups"...  type=io.containerd.monitor.v1
INFO[2022-10-13T03:09:21.793136796Z] loading plugin "io.containerd.service.v1.tasks-service"...  type=io.containerd.service.v1
INFO[2022-10-13T03:09:21.793658754Z] loading plugin "io.containerd.internal.v1.restart"...  type=io.containerd.internal.v1
INFO[2022-10-13T03:09:21.795432421Z] loading plugin "io.containerd.grpc.v1.containers"...  type=io.containerd.grpc.v1
INFO[2022-10-13T03:09:21.795857504Z] loading plugin "io.containerd.grpc.v1.content"...  type=io.containerd.grpc.v1
INFO[2022-10-13T03:09:21.796000879Z] loading plugin "io.containerd.grpc.v1.diff"...  type=io.containerd.grpc.v1
INFO[2022-10-13T03:09:21.796269838Z] loading plugin "io.containerd.grpc.v1.events"...  type=io.containerd.grpc.v1
INFO[2022-10-13T03:09:21.796690879Z] loading plugin "io.containerd.grpc.v1.healthcheck"...  type=io.containerd.grpc.v1
INFO[2022-10-13T03:09:21.796906588Z] loading plugin "io.containerd.grpc.v1.images"...  type=io.containerd.grpc.v1
INFO[2022-10-13T03:09:21.797034796Z] loading plugin "io.containerd.grpc.v1.leases"...  type=io.containerd.grpc.v1
INFO[2022-10-13T03:09:21.797202588Z] loading plugin "io.containerd.grpc.v1.namespaces"...  type=io.containerd.grpc.v1
INFO[2022-10-13T03:09:21.797557629Z] loading plugin "io.containerd.internal.v1.opt"...  type=io.containerd.internal.v1
INFO[2022-10-13T03:09:21.799339254Z] loading plugin "io.containerd.grpc.v1.snapshots"...  type=io.containerd.grpc.v1
INFO[2022-10-13T03:09:21.799557879Z] loading plugin "io.containerd.grpc.v1.tasks"...  type=io.containerd.grpc.v1
INFO[2022-10-13T03:09:21.799694629Z] loading plugin "io.containerd.grpc.v1.version"...  type=io.containerd.grpc.v1
INFO[2022-10-13T03:09:21.799756796Z] loading plugin "io.containerd.grpc.v1.introspection"...  type=io.containerd.grpc.v1
INFO[2022-10-13T03:09:21.805491421Z] serving...                                    address=/var/run/docker/containerd/containerd-debug.sock
INFO[2022-10-13T03:09:21.806265588Z] serving...                                    address=/var/run/docker/containerd/containerd.sock.ttrpc
INFO[2022-10-13T03:09:21.807189921Z] serving...                                    address=/var/run/docker/containerd/containerd.sock
INFO[2022-10-13T03:09:21.821722546Z] containerd successfully booted in 0.184633s  
INFO[2022-10-13T03:09:21.887599004Z] parsed scheme: "unix"                         module=grpc
INFO[2022-10-13T03:09:21.887708796Z] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2022-10-13T03:09:21.887778879Z] ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}  module=grpc
INFO[2022-10-13T03:09:21.887836546Z] ClientConn switching balancer to "pick_first"  module=grpc
INFO[2022-10-13T03:09:21.893983879Z] parsed scheme: "unix"                         module=grpc
INFO[2022-10-13T03:09:21.894081379Z] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2022-10-13T03:09:21.894154963Z] ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}  module=grpc
INFO[2022-10-13T03:09:21.894206296Z] ClientConn switching balancer to "pick_first"  module=grpc
INFO[2022-10-13T03:09:21.907646796Z] [graphdriver] using prior storage driver: overlay2 
INFO[2022-10-13T03:09:21.925756838Z] Loading containers: start.                   
WARN[2022-10-13T03:09:21.946484213Z] Running iptables --wait -t nat -L -n failed with message: `iptables v1.8.7 (legacy): can't initialize iptables table `nat': iptables who? (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.`, error: exit status 3 
INFO[2022-10-13T03:09:22.252070713Z] stopping event stream following graceful shutdown  error="<nil>" module=libcontainerd namespace=moby
INFO[2022-10-13T03:09:22.253173921Z] stopping healthcheck following graceful shutdown  module=libcontainerd
INFO[2022-10-13T03:09:22.254884796Z] stopping event stream following graceful shutdown  error="context canceled" module=libcontainerd namespace=plugins.moby
failed to start daemon: Error initializing network controller: error obtaining controller instance: failed to create NAT chain DOCKER: iptables failed: iptables -t nat -N DOCKER: iptables v1.8.7 (legacy): can't initialize iptables table `nat': iptables who? (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.
 (exit status 3)

Information

• macOS version: Monterey 12.6
• Docker Desktop version: 4.12.0 (85629)
• Docker Engine version: 20.10.17

[bamurtaugh]

Thanks for reporting. There seem to have been some transient package issues around this time. Are you still experiencing this issue?

@Chuxel @chrmarti I see containerd in the logs above. Would microsoft/vscode-remote-release#6014 pose issues with using containerd?

[Chuxel]

No this is a problem with mixing architectures - x86 and arm64.

Cross-architecture emulation has a number of limitations. I am guessing that prevents access to the needed underlying devices for it to work. This may not be possible at all.

Here's how docker talks about emulation:

Some images do not support the ARM64 architecture. You can add --platform linux/amd64 to run (or build) an Intel image using emulation.

However, attempts to run Intel-based containers on Apple silicon machines under emulation can crash as qemu sometimes fails to run the container. In addition, filesystem change notification APIs (inotify) do not work under qemu emulation. Even when the containers do run correctly under emulation, they will be slower and use more memory than the native equivalent.

In summary, running Intel-based containers on Arm-based machines should be regarded as “best effort” only. We recommend running arm64 containers on Apple silicon machines whenever possible, and encouraging container authors to produce arm64, or multi-arch, versions of their containers. We expect this issue to become less common over time, as more and more images are rebuilt supporting multiple architectures.

[zephyros-dev]

No this is a problem with mixing architectures - x86 and arm64.

Cross-architecture emulation has a number of limitations. I am guessing that prevents access to the needed underlying devices for it to work. This may not be possible at all.

Here's how docker talks about emulation:

Some images do not support the ARM64 architecture. You can add --platform linux/amd64 to run (or build) an Intel image using emulation.

However, attempts to run Intel-based containers on Apple silicon machines under emulation can crash as qemu sometimes fails to run the container. In addition, filesystem change notification APIs (inotify) do not work under qemu emulation. Even when the containers do run correctly under emulation, they will be slower and use more memory than the native equivalent.

In summary, running Intel-based containers on Arm-based machines should be regarded as “best effort” only. We recommend running arm64 containers on Apple silicon machines whenever possible, and encouraging container authors to produce arm64, or multi-arch, versions of their containers. We expect this issue to become less common over time, as more and more images are rebuilt supporting multiple architectures.

According to this issue docker/for-mac#6284 (comment), it seems like dind will only works on arm64 image since the M1 amd64 emulator does not support ip routing. While this issue is not related to devcontainer, I think adding a note to the README would be useful for future user. What do you think?

[Chuxel]

Yep, that seems reasonable. Adding a NOTES.md to the feature will cause it to appear in the README.

[ci-vamp]

it may be worth updating this note. as of macOS ventura (13.1) and docker desktop 4.16.1 there is an option to emulate using rosetta in the Features in development options of docker desktop.

docker desktop macOS features in development docs

i almost gave up when reading this message and thread but decided to try it anyways (since it was just released a few weeks ago).

its working great! just had to mount my docker socket from host to devcontainer and the rest worked.

reference for others:

Dockerfile

FROM --platform=linux/amd64 mcr.microsoft.com/devcontainers/base:ubuntu

RUN ENABLE_NONROOT_DOCKER=false \
  /bin/bash \
  -c "$(curl -fsSL https://raw.githubusercontent.com/devcontainers/features/main/src/docker-outside-of-docker/install.sh)"

...

devcontainer.json

...
  "mounts": [
    // mount the host docker socket so docker can be used in the devcontainer
    "source=/var/run/docker.sock,target=/var/run/docker-host.sock,type=bind",
  ],
...