Skip to content

Debian 12 : Outside not reachable from rootless docker networks when enabling source IP propagation #434

New issue
New issue
Open
Open
Debian 12 : Outside not reachable from rootless docker networks when enabling source IP propagation#434
@g-azerad

Description

@g-azerad
g-azerad
opened on Apr 29, 2024

Issue

Accordingly with Docker official documentation, I have tried to enable source IP propagation for rootless Docker with the following solutions :

  • slirp4netns RootlessKit port driver
  • pasta RootlessKit network driver, with the implicit port driver

With the default parameters (slirp4netns network driver and builtin port driver), I can access running dockerized applications from the outside, and it is possible to docker pull images from Docker Hub.

When using each of the solutions to enable source IP propagation, neither of the two previous feature are available and we reach timeouts.

Context

  • Server: Hyper-V VM using a NAT network to access the outside
  • OS: Debian 12
  • Docker info
Client: Docker Engine - Community
 Version:    26.0.0
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.13.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.25.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 26.0.0
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: false
  userxattr: true
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  rootless
  cgroupns
 Kernel Version: 6.1.0-18-amd64
 Operating System: Debian GNU/Linux 12 (bookworm)
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 895.2MiB
 Name: vm-debian-2
 ID: 6e6f27e9-0fe4-4d10-8b3e-6ebd12a8594b
 Docker Root Dir: /home/test/.local/share/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No cpuset support
WARNING: No io.weight support
WARNING: No io.weight (per device) support
WARNING: No io.max (rbps) support
WARNING: No io.max (wbps) support
WARNING: No io.max (riops) support
WARNING: No io.max (wiops) support
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
  • slirp4netns version
slirp4netns version 1.2.0
commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
libslirp: 4.7.0
SLIRP_CONFIG_VERSION_MAX: 4
libseccomp: 2.5.4

Logs

  • Systemctl status for rootless Docker with default parameters (correct behaviour, success trying docker run -p 80:80 nginx:1.25.4)
docker.service - Docker Application Container Engine (Rootless)
     Loaded: loaded (/home/test/.config/systemd/user/docker.service; enabled; preset: enabled)
    Drop-In: /home/test/.config/systemd/user/docker.service.d
             └─override.conf
     Active: active (running) since Mon 2024-04-29 15:56:02 CEST; 4min 3s ago
       Docs: https://docs.docker.com/go/rootless/
   Main PID: 2114 (rootlesskit)
      Tasks: 39
     Memory: 122.1M
        CPU: 8.329s
     CGroup: /user.slice/user-1002.slice/[email protected]/app.slice/docker.service
             ├─2114 rootlesskit --state-dir=/run/user/1002/dockerd-rootless --net=slirp4netns --mtu=1500 --slirp4netns-sandbox=auto --slirp4netns-seccomp=auto --disable-host-loopback --port-driver=builtin --copy-up=/etc --copy-up=/run --propagation=rslave /usr/bin/dockerd-rootless.sh
             ├─2123 /proc/self/exe --state-dir=/run/user/1002/dockerd-rootless --net=slirp4netns --mtu=1500 --slirp4netns-sandbox=auto --slirp4netns-seccomp=auto --disable-host-loopback --port-driver=builtin --copy-up=/etc --copy-up=/run --propagation=rslave /usr/bin/dockerd-rootless.sh
             ├─2146 slirp4netns --mtu 1500 -r 3 --disable-host-loopback --enable-sandbox --enable-seccomp 2123 tap0
             ├─2153 dockerd
             └─2167 containerd --config /run/user/1002/docker/containerd/containerd.toml

Apr 29 15:58:10 vm-debian-2 dockerd-rootless.sh[2153]: time="2024-04-29T15:58:10.643268520+02:00" level=error msg="Not continuing with pull after error: context canceled" spanID=374373be0d7759ad traceID=80542cfab1399cb4599eaafa96ec1b61
Apr 29 15:58:50 vm-debian-2 dockerd-rootless.sh[2167]: time="2024-04-29T15:58:50.929155464+02:00" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
Apr 29 15:58:50 vm-debian-2 dockerd-rootless.sh[2167]: time="2024-04-29T15:58:50.929297192+02:00" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
Apr 29 15:58:50 vm-debian-2 dockerd-rootless.sh[2167]: time="2024-04-29T15:58:50.929309516+02:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
Apr 29 15:58:50 vm-debian-2 dockerd-rootless.sh[2167]: time="2024-04-29T15:58:50.930087880+02:00" level=info msg="starting signal loop" namespace=moby path=/run/.ro1423655209/user/1002/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/74195f8b82ffeaa42d7664ac9903962475408be48c8714d50e4
Apr 29 15:58:58 vm-debian-2 dockerd-rootless.sh[2153]: time="2024-04-29T15:58:58.459656831+02:00" level=info msg="ignoring event" container=74195f8b82ffeaa42d7664ac9903962475408be48c8714d50e4ff546aadb9cbd module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Apr 29 15:58:58 vm-debian-2 dockerd-rootless.sh[2167]: time="2024-04-29T15:58:58.460279821+02:00" level=info msg="shim disconnected" id=74195f8b82ffeaa42d7664ac9903962475408be48c8714d50e4ff546aadb9>
Apr 29 15:58:58 vm-debian-2 dockerd-rootless.sh[2167]: time="2024-04-29T15:58:58.460342760+02:00" level=warning msg="cleaning up after shim disconnected" id=74195f8b82ffeaa42d7664ac9903962475408be48c8714d50e4ff546aadb9cbd namespace=moby
Apr 29 15:58:58 vm-debian-2 dockerd-rootless.sh[2167]: time="2024-04-29T15:58:58.460351586+02:00" level=info msg="cleaning up dead shim"
Apr 29 15:58:58 vm-debian-2 dockerd-rootless.sh[2167]: time="2024-04-29T15:58:58.467999692+02:00" level=warning msg="cleanup warnings time=\"2024-04-29T15:58:58+02:00\" level=info msg=\"starting signal loop\" namespace=moby pid=2502 runtime=io.containerd.runc.v2\n"
  • Systemctl status for rootless Docker with pasta network (incorrect behaviour: timeout when trying docker run -p 80:80 nginx:1.25.4)
docker.service - Docker Application Container Engine (Rootless)
     Loaded: loaded (/home/test/.config/systemd/user/docker.service; enabled; preset: enabled)
    Drop-In: /home/test/.config/systemd/user/docker.service.d
             └─override.conf
     Active: active (running) since Mon 2024-04-29 14:58:49 CEST; 48min ago
       Docs: https://docs.docker.com/go/rootless/
   Main PID: 1555 (rootlesskit)
      Tasks: 38
     Memory: 66.9M
        CPU: 9.939s
     CGroup: /user.slice/user-1002.slice/[email protected]/app.slice/docker.service
             ├─1555 rootlesskit --state-dir=/run/user/1002/dockerd-rootless --net=pasta --mtu=1500 --slirp4netns-sandbox=auto --slirp4netns-seccomp=auto --disable-host-loopback --port-driver=implicit --copy-up=/etc --copy-up=/run --propagation=rslave /usr/bin/dockerd-rootless.sh
             ├─1561 /proc/self/exe --state-dir=/run/user/1002/dockerd-rootless --net=pasta --mtu=1500 --slirp4netns-sandbox=auto --slirp4netns-seccomp=auto --disable-host-loopback --port-driver=implicit --copy-up=/etc --copy-up=/run --propagation=rslave /usr/bin/dockerd-rootless.sh
             ├─1590 dockerd
             └─1605 containerd --config /run/user/1002/docker/containerd/containerd.toml

Apr 29 14:58:49 vm-debian-2 dockerd-rootless.sh[1590]: time="2024-04-29T14:58:49.923912488+02:00" level=warning msg="WARNING: No io.max (wiops) support"
Apr 29 14:58:49 vm-debian-2 dockerd-rootless.sh[1590]: time="2024-04-29T14:58:49.923916345+02:00" level=warning msg="WARNING: bridge-nf-call-iptables is disabled"
Apr 29 14:58:49 vm-debian-2 dockerd-rootless.sh[1590]: time="2024-04-29T14:58:49.923920252+02:00" level=warning msg="WARNING: bridge-nf-call-ip6tables is disabled"
Apr 29 14:58:49 vm-debian-2 dockerd-rootless.sh[1590]: time="2024-04-29T14:58:49.923936774+02:00" level=info msg="Docker daemon" commit=8b79278 containerd-snapshotter=false storage-driver=overlay2 version=26.0.0
Apr 29 14:58:49 vm-debian-2 dockerd-rootless.sh[1590]: time="2024-04-29T14:58:49.923976709+02:00" level=info msg="Daemon has completed initialization"
Apr 29 14:58:49 vm-debian-2 dockerd-rootless.sh[1590]: time="2024-04-29T14:58:49.954817534+02:00" level=info msg="API listen on /run/user/1002/docker.sock"
Apr 29 14:58:49 vm-debian-2 systemd[542]: Started docker.service - Docker Application Container Engine (Rootless).
Apr 29 15:00:18 vm-debian-2 dockerd-rootless.sh[1590]: time="2024-04-29T15:00:18.176877834+02:00" level=warning msg="Error getting v2 registry: Get \"https://registry-1.docker.io/v2/\": dial tcp: lookup registry-1.docker.io on 10.0.2.3:53: read udp 10.0.2.100:33416->10.0.2.3:53: i/o timeout" spanID=8a7239eb7e0df459 traceID=bfcd1227bf9ed59e94bc041f209f12db
Apr 29 15:00:18 vm-debian-2 dockerd-rootless.sh[1590]: time="2024-04-29T15:00:18.177037334+02:00" level=info msg="Attempting next endpoint for pull after error: Get \"https://registry-1.docker.io/v2\": dial tcp: lookup registry-1.docker.io on 10.0.2.3:53: read udp 10.0.2.100:33416->10.0.2.3:53: i/o timeout" spanID=8a7239eb7e0df459 traceID=bfcd1227bf9ed59e94bc041f209f12db
Apr 29 15:00:18 vm-debian-2 dockerd-rootless.sh[1590]: time="2024-04-29T15:00:18.180708447+02:00" level=error msg="Handler for POST /v1.45/images/create returned error: Get \"https://registry-1.docker.io/v2/\": dial tcp: lookup registry-1.docker.io on 10.0.2.3:53: read udp 10.0.2.100:33416->10.0.2.3:53: i/o timeout" spanID=8a7239eb7e0df459 traceID=bfcd1227bf9ed59e94bc041f209f12db
  • Systemctl status for rootless Docker with slirp4netns port driver.
    Incorrect behaviour with docker run -p 80:80 nginx:1.25.4 command : image correctly pulled but error running the container :
 docker run -p 80:80 nginx:1.25.4
Unable to find image 'nginx:1.25.4' locally
1.25.4: Pulling from library/nginx
13808c22b207: Pull complete
6fcdffcd79f0: Pull complete
fbf231d461b3: Pull complete
c9590dd9c988: Pull complete
b4033143d859: Pull complete
abaefc5fcbde: Pull complete
bcef83155b8b: Pull complete
Digest: sha256:9ff236ed47fe39cf1f0acf349d0e5137f8b8a6fd0b46e5117a401010e56222e1
Status: Downloaded newer image for nginx:1.25.4
docker: Error response from daemon: driver failed programming external connectivity on endpoint determined_payne (f1712cedcd2703778e6d6635d7e34d8270b93712a9f47f1900fac9057d61712b): Error starting userland proxy: error while calling PortManager.AddPort(): reply.Error: map[desc:bad request: add_hostfwd: slirp_add_hostfwd failed].

Systemctl status :

docker.service - Docker Application Container Engine (Rootless)
     Loaded: loaded (/home/test/.config/systemd/user/docker.service; enabled; preset: enabled)
    Drop-In: /home/test/.config/systemd/user/docker.service.d
             └─override.conf
     Active: active (running) since Mon 2024-04-29 18:59:39 CEST; 4min 3s ago
       Docs: https://docs.docker.com/go/rootless/
   Main PID: 1047 (rootlesskit)
      Tasks: 33
     Memory: 290.9M
        CPU: 9.035s
     CGroup: /user.slice/user-1002.slice/[email protected]/app.slice/docker.service
             ├─1047 rootlesskit --state-dir=/run/user/1002/dockerd-rootless --net=slirp4netns --mtu=1500 --slirp4netns-sandbox=auto --slirp4netns-seccomp=auto --disable-host-loopback --port-driver=slirp4netns --copy-up=/etc --copy-up=/run --propagation=rslave /usr/bin/dockerd-rootless.sh
             ├─1056 /proc/self/exe --state-dir=/run/user/1002/dockerd-rootless --net=slirp4netns --mtu=1500 --slirp4netns-sandbox=auto --slirp4netns-seccomp=auto --disable-host-loopback --port-driver=slirp4netns --copy-up=/etc --copy-up=/run --propagation=rslave /usr/bin/dockerd-rootless.sh
             ├─1078 slirp4netns --mtu 1500 -r 3 --disable-host-loopback --api-socket /run/user/1002/dockerd-rootless/.s4nn.sock --enable-sandbox --enable-seccomp 1056 tap0
             ├─1085 dockerd
             └─1097 containerd --config /run/user/1002/docker/containerd/containerd.toml

Apr 29 18:59:39 vm-debian-2 dockerd-rootless.sh[1085]: time="2024-04-29T18:59:39.243567478+02:00" level=warning msg="WARNING: No io.max (riops) support"
Apr 29 18:59:39 vm-debian-2 dockerd-rootless.sh[1085]: time="2024-04-29T18:59:39.243571236+02:00" level=warning msg="WARNING: No io.max (wiops) support"
Apr 29 18:59:39 vm-debian-2 dockerd-rootless.sh[1085]: time="2024-04-29T18:59:39.243574993+02:00" level=warning msg="WARNING: bridge-nf-call-iptables is disabled"
Apr 29 18:59:39 vm-debian-2 dockerd-rootless.sh[1085]: time="2024-04-29T18:59:39.243578940+02:00" level=warning msg="WARNING: bridge-nf-call-ip6tables is disabled"
Apr 29 18:59:39 vm-debian-2 dockerd-rootless.sh[1085]: time="2024-04-29T18:59:39.243593818+02:00" level=info msg="Docker daemon" commit=8b79278 containerd-snapshotter=false storage-driver=overlay2 version=26.0.0
Apr 29 18:59:39 vm-debian-2 dockerd-rootless.sh[1085]: time="2024-04-29T18:59:39.243637611+02:00" level=info msg="Daemon has completed initialization"
Apr 29 18:59:39 vm-debian-2 dockerd-rootless.sh[1085]: time="2024-04-29T18:59:39.272398570+02:00" level=info msg="API listen on /run/user/1002/docker.sock"
Apr 29 18:59:39 vm-debian-2 systemd[536]: Started docker.service - Docker Application Container Engine (Rootless).
Apr 29 19:00:53 vm-debian-2 dockerd-rootless.sh[1085]: time="2024-04-29T19:00:53.965465725+02:00" level=warning msg="Failed to allocate and map port 80-80: Error starting userland proxy: error while calling PortManager.AddPort(): reply.Error: map[desc:bad request: add_hostfwd: slirp_add_hostfwd failed]"
Apr 29 19:00:54 vm-debian-2 dockerd-rootless.sh[1085]: time="2024-04-29T19:00:54.023005263+02:00" level=error msg="Handler for POST /v1.45/containers/f66bb23f97b265f47cc08a8b3eb60e3fe9924e9a15bc7e1f39f3da7d3fa0dd30/start returned error: driver failed programming external connectivity on endpoint determined_payne (f1712cedcd2703778e6d6635d7e34d8270b93712a9f47f1900fac9057d61712b): Error starting userland proxy: error while calling PortManager.AddPort(): reply.Error: map[desc:bad request add_hostfwd: slirp_add_hostfwd failed]" spanID=7aa2e869843348b8 traceID=3436c4616371df198db7a605f64a840a

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions