Alternate port_handler that keeps the source ip for user-defined rootless networks

[usury]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Alternate "port_handler=slirt4netns" provided in PR 6965 (#6965) for "only 127.0.0.1 within containers" not implemented for user-defined rootless cni networks

• specifying "port_handler=slirp4netns" when connecting a container to a already-created user-defined rootless cni network does NOT error out and does NOT implement the behavior provided in PR 6965
• as a result, not possible to see accurate remote address within containers connected to a user-defined rootless cni network (only "127.0.0.1")

Steps to reproduce the issue:

normaluser@containerhost $> podman network create myCNI

normaluser@containerhost $> podman run --name myNginx1 --publish 8081:80 --network=myCNI -d nginx:alpine
normaluser@containerhost $> podman run --name myNginx2 --publish 8082:80 --network=myCNI:port_handler=slirp4netns -d nginx:alpine
normaluser@containerhost $> podman run --name myNginx3 --publish 8083:80 --network=slirp4netns:port_handler=slirp4netns -d nginx:alpine

normaluser@containerhost $>  podman inspect myNginx1 | grep -i ipaddress
                "IPAddress": "10.89.0.2",
normaluser@containerhost $>  podman inspect myNginx2 | grep -i ipaddress
                "IPAddress": "",
normaluser@containerhost $>  podman inspect myNginx3 | grep -i ipaddress
                "IPAddress": "",

normaluser@containerhost $> podman exec myNginx1 ifconfig
                ## shows loopback and **eth0**, as expected
normaluser@containerhost $> podman exec myNginx2 ifconfig
                ## absolutely blank, not expected
normaluser@containerhost $> podman exec myNginx3 ifconfig
                ## shows loopback and **tap0**, as expected

otherhost $> curl --head http://_containerhost_:8081
otherhost $> curl --head http://_containerhost_:8082
`        curl: (7) Failed to connect to _containerhost_ port 8082: Connection refused`
otherhost $> curl --head http://_containerhost_:8083

normaluser@containerhost $> podman logs myNginx1 | grep HEAD
        127.0.0.1 - - [29/Oct/2020:15:51:25 -0700] "HEAD / HTTP/1.1" 200 0 "-" "curl/7.69.1" "-"
normaluser@containerhost $> podman logs myNginx2 | grep HEAD
        ## nothing to see, no connection succeeded
normaluser@containerhost $> podman logs myNginx3|grep HEAD
        192.168.0.116 - - [29/Oct/2020:15:51:31 -0700] "HEAD / HTTP/1.1" 200 0 "-" "curl/7.69.1" "-"

Describe the results you received:
"myNginx1" container has only 127.0.0.1 as remote address (confusing), though explained in PR 6965
"myNginx2" container launches without error however port not opened and no ip address assigned
"myNginx3" container - no problem - everything makes sense

Describe the results you expected:
"myNginx1:" expected to see correct remote address wihin container (though explainded in PR 6965)
"myNginx2:" expected successful ip assignment and port to open using "port_handler=slirp4netns", or contaier launch to fail
"myNginx3" container - no problem - everything makes sense

Additional information you deem important (e.g. issue happens only occasionally):
happens consistently and reproducably

Output of podman version:

        Version:      2.1.1
        API Version:  2.0.0
        Go Version:   go1.14
        Built:        Wed Dec 31 16:00:00 1969
        OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.16.1
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.20, commit: '
  cpus: 4
  distribution:
    distribution: debian
    version: "10"
  eventLogger: journald
  hostname: arachne
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 2000
      size: 1
    - container_id: 1
      host_id: 100001
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 2000
      size: 1
    - container_id: 1
      host_id: 100001
      size: 65536
  kernel: 4.19.0-10-amd64
  linkmode: dynamic
  memFree: 369057792
  memTotal: 2091732992
  ociRuntime:
    name: runc
    package: 'runc: /usr/sbin/runc'
    path: /usr/sbin/runc
    version: |-
      runc version 1.0.0~rc6+dfsg1
      commit: 1.0.0~rc6+dfsg1-3
      spec: 1.0.1
  os: linux
  remoteSocket:
    path: /run/user/2000/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.1.4
      commit: unknown
      libslirp: 4.3.1-git
      SLIRP_CONFIG_VERSION_MAX: 3
  swapFree: 746057728
  swapTotal: 1073737728
  uptime: 184h 14m 26.29s (Approximately 7.67 days)
registries:
  search:
  - docker.io
  - quay.io
version:
  APIVersion: 2.0.0
  Built: 0
  BuiltTime: Wed Dec 31 16:00:00 1969
  GitCommit: ""
  GoVersion: go1.14
  OsArch: linux/amd64
  Version: 2.1.1

Package info (e.g. output of rpm -q podman or apt list podman):

podman/unknown,now 2.1.1~2 amd64 [installed]
podman/unknown 2.1.1~2 arm64
podman/unknown 2.1.1~2 armhf
podman/unknown 2.1.1~2 ppc64el

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):
Container host is a VirtualBox VM running on Fedora 32
podman packages installed from OpenSuse repo
$> cat "/etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list"

## "Kubic" repo from "OpenSuse" for "podman" packages since they aren't in Debian 10 (buster) repos
deb https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/Debian_10/ /

$> cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 10 (buster)"

[usury]

Since the "port_handler=slirp4netns" solution from PR 6965 was implemented to address the "only 127.0.0.1" problem within rootless containers ("port_handler=rootlesskit" by default), and since that solution is not implemented for user-defined rooteless cni networks, perhaps revisiting the "only 127.0.0.1" problem with "port_handler=rootlesskit" is in order before simply extending the solution from PR 6965 to user-defined rootless cni networks.

Furthermore, the default behavior of "only 127.0.0.1" within rootless containers, regardless of user-defined rootless cni networking, is confusing for users who are new to containers/podman but not new to webserver/application configuration, like myself.

• For hours, I chased what I thought was a self-inflicted configuration problem brought about by my own container/podman ignorance. Instead, "127.0.0.1" in my container logs was a consequence of an implementation decision for rootless podman to use "port_handler=rootlesskit" by default, not my inexperience with containers/podman.

Additionally, the default behvavior of "only 127.0.0.1" within rootless containers, regardless of user-defined rootless cni networking, breaks potential remote-address-based authentication mechanisms a developer could devise for rootless container services.

Background as I know it
PR 6965 provided a mechanism for specifying "port_handler=[rootlesskit|slirp4netns]" as an extension to the "--network" commandline arg when launching a rootless container. That option provides a work-around for the issue of seeing only "127.0.0.1" as the remote address for requests reaching rootless containers via published host ports

PR 6965 allows users to specify an alternate mechanism for rootless port binding which successfully preserves the remote address for packets reaching rootless containers.

PR 6965 did not fix the "only 127.0.0.1" issue stemming from the use of "port_handler=rootlesskit". Instead, PR 6965 made it possible to select "port_handler=slirp4netns" which does not exhibit the "only 127.0.0.1" behavior of "port_handler=rootlesskit" (the default).

Since the alternate "port_handler" work-around for "only 127.0.0.1" provided in PR 6965 is not implemented for user-defined rootless cni networks, perhaps a more general or clearer way to handle "rootlesskit" vs "slirp4netns" port_handler is required.

That clearer way to handle "rootlesskit" vs "slirp4netns" port_handler ought to support "slirp4netns" network options when creating and/or connecting to user-defined rootless cni networks (whichever is more appropriate) to address the issues identifed at the top of this post.

There may be future options required for current/future underlying network subsystems. A non-convoluted mechanism to easily support such subsystems/options could be useful or help to avoid some future headaches.

  Idea ONLY - not an actual working mechanism
  A separate "--netopt" commandline arg supported by "podman run" and/or "podman network create"
  I read a similar suggestion when learning about PR 6965

    ## NO user-defined rootless cni network
    ## specify --netopt when CONTAINER is created
    ## container uses default network and "port_handler=slirp4netns"
    "podman run --name myApp0 --netopt port_handler=slirp4netns"

    ## WITH user-defined rootless cni network
    ## specify --netopt when NETWORK is created
    ## container automatically uses "port_handler=slirp4netns" by virture of joining "myCNI1"
    ## !! There may be very good reasons preventing this per-NETWORK approach !!
    ## !! Maybe each containers on a network needs/should support independent "port_handler" !!
    "podman network create --name myCNI1 --netopt port_handler=slirp4netns"
    "podman run --name myApp1 --network myCNI1"

    ## WITH user-defined rootless cni network
    ## specify --netopt when CONTAINER is created
    ## continer uses "myCNI2" network and "port_handler=slirp4netns"
    ## !! There may be very good reasons preventing this per-CONTAINER approach when joining a network !!
    ## !! Maybe all containers on a network need the same "port_handler" !!
    "podman network create --name myCNI2"
    "podman run --name myApp2 --network myCNI2 --netopt port_handler=slirp4netns"

---

***** User-defined rootless network limitations *****

---

## Setup
    normaluser@containerhost $> podman rm -fa       # new set of examples
    normaluser@containerhost $> podman network create myCNI
    normaluser@containerhost $> podman run --name myNginx1 --publish 8081:80 --network=myCNI -d nginx:alpine
    normaluser@containerhost $> podman run --name myNginx2 --publish 8082:80 --network=myCNI:port_handler=slirp4netns -d nginx:alpine

## only "myNginx1" (port 8081) succeeded in opening a port
## Listening Ports
    normaluser@containerhost $> netstat -tnlp | grep 808[0-9]
    ## -t tcp ports, -n numeric addresses, -l listening, -p process
        tcp6       0      0 :::8081                 :::*                    LISTEN      26135/containers-ro 

## still, both containers started successfully (expected "myNginx2" to fail since it did not actually open port 8082)
## Containers
    normaluser@containerhost $> podman ps
        CONTAINER ID  IMAGE                             COMMAND               CREATED             STATUS                 PORTS                 NAMES
        844616df3c71  docker.io/library/nginx:alpine    nginx -g daemon o...  About a minute ago  Up About a minute ago  0.0.0.0:8082->80/tcp  myNginx2
        964801ccf46d  docker.io/library/nginx:alpine    nginx -g daemon o...  About a minute ago  Up About a minute ago  0.0.0.0:8081->80/tcp  myNginx1
        0aaa632fe8cc  quay.io/libpod/rootless-cni-infra sleep infinity        About a minute ago  Up About a minute ago                        rootless-cni-infra

## "127.0.0.1" behavior consistent with examples to follow
    normaluser@otherhost     $> curl --head http://containerhost:8081
    normaluser@containerhost $> curl --head http://containerhost:8081
    normaluser@containerhost $> podman logs myNginx1 | grep HEAD
        127.0.0.1 - - [29/Oct/2020:13:12:32 -0700] "HEAD / HTTP/1.1" 200 0 "-" "curl/7.69.1" "-"
        127.0.0.1 - - [29/Oct/2020:13:12:50 -0700] "HEAD / HTTP/1.1" 200 0 "-" "curl/7.64.0" "-"

## cannot connect on port 8082 (it was not successfully published - as confirmed by netstat, above)
    normaluser@otherhost     $> curl --head http://containerhost:8082
    normaluser@containerhost $> curl --head http://containerhost:8081
        curl: (7) Failed to connect to 192.168.0.22 port 8082: Connection refused
    normaluser@containerhost $> podman logs myNginx2 | grep HEAD
        <nothing to see>

    normaluser@containerhost $> podman inspect myNginx1 | grep -i ipaddress
                        "IPAddress": "10.89.0.2",
    normaluser@containerhost $> podman inspect myNginx2 | grep -i ipaddress
                "IPAddress": "",

    normaluser@containerhost $> podman exec myNginx2 ifconfig
        ## absolutely blank, not even loopback - not expected
    normaluser@containerhost $> podman exec myNginx2 ip addr
        1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN qlen 1000

    normaluser@containerhost $> podman exec myNginx2 netstat -tln
        tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      
        tcp        0      0 :::80                   :::*                    LISTEN      

Oddly:
"myNginx2" launches without error but does not get an ip address on "myCNI" (but "myNginx1" does as expected)
"myNginx2" launches without error but does not open a port
"myNginx2" thinks it is listening internally on its port 80 even though ifconfig within container reports nothing
expected "podman run --name myNginx2 ..." to either
- fail for not opening the port,
- fail for not having an ip address
- successfully receive ip address and open the port using "port_handlier=slirp4netns" network option

[usury]

---

***** potentially informative "port_handler=rootlesskit" behavior *****

---

Unexpected IP6 address binding behavior of "port_handler=rootlesskit"

Alternate "port_handler" provided in PR 6965 helps to reveal ...

• rootless "podman run" with "--network=slirpnetns:port_handler=rootlesskit" (default)
  -- binds to local IP6 address unless non-zero IP4 address specified in "--publish" arg
  --- container sees only "127.0.0.1" as remote address (regardless of actual remote address)
  -- binds to requested non-zero IP4 address when specified in "--publish" arg
  --- container still sees only "127.0.0.1" as remote address (regardless of actual remote address)

• rootless "podman run" with "--network=slirpnetns:port_handler=slirp4netns"
  -- binds to IP4 address
  -- container sees correct remote address
  -- these are good things

## Setup
    normaluser@containerhost $> podman rm -fa       # new set of examples
    normaluser@containerhost $> podman run --name myNginx1 --publish 8081:80 -d nginx:alpine                # no IP addr
    normaluser@containerhost $> podman run --name myNginx2 --publish 0.0.0.0:8082:80 -d nginx:alpine        # "any" IP4 addr
    normaluser@containerhost $> podman run --name myNginx2 --publish 192.168.0.22:8083:80 -d nginx:alpine   # container host IP4 addr
    normaluser@containerhost $> podman run --name myNginx4 --publish 8084:80 --network=slirp4netns:port_handler=slirp4netns -d nginx:alpine     # port_handler=slirp4netns

## Container logs always show "127.0.0.1" via "port_handler=rootlesskit" (default) "--publish 8081:80" (no ip addr specified)
    normaluser@otherhost     $> curl --head http://containerhost:8081
    normaluser@containerhost $> curl --head http://containerhost:8081
    normaluser@containerhost $> podman logs myNginx1 | grep HEAD
        127.0.0.1 - - [29/Oct/2020:10:18:25 -0700] "HEAD / HTTP/1.1" 200 0 "-" "curl/7.69.1" "-"
        127.0.0.1 - - [29/Oct/2020:10:18:45 -0700] "HEAD / HTTP/1.1" 200 0 "-" "curl/7.64.0" "-"

## Container logs always show "127.0.0.1" via "port_handler=rootlesskit" (default) "--publish 0.0.0.0:8082:80" (the "any" ip4 addr)
    normaluser@otherhost     $> curl --head http://containerhost:8082
    normaluser@containerhost $> curl --head http://containerhost:8082
    normaluser@containerhost $> podman logs myNginx2 | grep HEAD
        127.0.0.1 - - [29/Oct/2020:10:18:26 -0700] "HEAD / HTTP/1.1" 200 0 "-" "curl/7.69.1" "-"
        127.0.0.1 - - [29/Oct/2020:10:18:46 -0700] "HEAD / HTTP/1.1" 200 0 "-" "curl/7.64.0" "-"

## Container logs always show "127.0.0.1" via "port_handler=rootlesskit" (default) "--publish 192.168.0.22:8083:80" (container host ip4 addr)
    normaluser@otherhost     $> curl --head http://containerhost:8083
    normaluser@containerhost $> curl --head http://containerhost:8083
    normaluser@containerhost $> podman logs myNginx3 | grep HEAD
        127.0.0.1 - - [29/Oct/2020:10:18:27 -0700] "HEAD / HTTP/1.1" 200 0 "-" "curl/7.69.1" "-"
        127.0.0.1 - - [29/Oct/2020:10:18:47 -0700] "HEAD / HTTP/1.1" 200 0 "-" "curl/7.64.0" "-"

## Correct remote address via "port_handler=slirp4netns"
    normaluser@otherhost     $> curl --head http://containerhost:8084
    normaluser@containerhost $> curl --head http://containerhost:8084
    normaluser@containerhost $> podman logs myNginx4
        192.168.0.116 - - [29/Oct/2020:10:18:30 -0700] "HEAD / HTTP/1.1" 200 0 "-" "curl/7.69.1" "-"
        192.168.0.22 - - [29/Oct/2020:10:18:52 -0700] "HEAD / HTTP/1.1" 200 0 "-" "curl/7.64.0" "-"

## Listening Ports
    normaluser@containerhost $> netstat -tnlp | grep 808[0-9]
    ## -t tcp ports, -n numeric addresses, -l listening, -p process
        tcp        0      0 192.168.0.22:8083       0.0.0.0:*               LISTEN      26387/containers-ro 
        tcp        0      0 0.0.0.0:8084            0.0.0.0:*               LISTEN      26470/slirp4netns   
        tcp6       0      0 :::8081                 :::*                    LISTEN      26135/containers-ro 
        tcp6       0      0 :::8082                 :::*                    LISTEN      26221/containers-ro 

Rather Odd Observations:

• When no ip address or when "0.0.0.0" is specified in "--publish" arg, "port_handler=rootlesskit" (default) binds to localhost IP6 address (the ones on port 8081 and 8082)
• When a non-zero IP4 address is specified in "--publish" arg (ip addr of containerhost in examples, above), "port_handler=rootlesskit" (default) binds to specified IP4 address (the one on port 8083)
• Regardless of IP4 vs IP6 address binding, when "port_handler=rootlesskit" only 127.0.0.1 is available within container
• Container using "port_handler=slirp4netns" binds to the "any" IP4 address (the one on port 8084)

I do not know what underlying IP4/IP6 linux mechanism allows connections like $> curl http://IP4 address:8081 (and 8082) to reach IP6 address:8081 (and 8082) where "rootlesskit" opened the ports. Perhaps that is an important mechanism in the behavior observed.

Still, the IP4 vs IP6 address bindings provide a clue regarding why "127.0.0.1" is the source address within containers launched with "port_handler=rootlesskit" (default) network option

Perhaps "rootlesskit" must recreate each packet it receives?

• When bound to host IP6 address, does "rootlesskit" recreate each packet for destination IP4 address of the container?
• If that is the case, it makes sense for source address to be "127.0.0.1" because the packet reaching container was generated on localhost (by "rootlesskit") from IP6 packets it somehow received
  -- still, it is unhelpful to see "127.0.0.1" within destination container by default
• However, that does not explain "myNginx3" example, which is bound to IP4 192.168.0.22:8083
  -- only "127.0.0.1" within that container, as well

Work-arounds on other forums suggest diabling ip6 on container host. A better approach may be to actually have "rootlesskit" bind to container host IP4 address, assuming that is actually part of the problem. There still remains the problem of seeing only "127.0.0.1" regardless of IP4 vs IP6 address binding ("myNginx2" and "myNginx3" examples)

[rhatdan]

@AkihiroSuda WDYT?

[AkihiroSuda]

--netopt

SGTM

Perhaps "rootlesskit" must recreate each packet it receives?

If we do that, it will be slowed down as in slirp4netns.

Probably we can just spoof srcIP using IP_TRANSPARENT socket, but I'm not familiar with IP_TRNASPARENT.

[usury]

SGTM

Perhaps "rootlesskit" must recreate each packet it receives?

If we do that, it will be slowed down as in slirp4netns.

My apologies for the confusion - it wasn't a suggestion for rootlesskit to recreate packets. It was a hypothesis as to why 127.0.0.1 is the remote address within a container when the packet passed through "rootlesskit". I should have said "Perhaps rootlesskit recreates each packet it receives?" The subpoints under that statement in original comment, above, make more sense in that clarified context.

[usury]

I know I muddied the topic by hypothesizing. However, the initial bug remains. It is not possible to specify "port_handler=slirp4netns" for user-defined rootless CNI networks, which means it is not currently possible to determine the correct remote address for containers using such a network. The step-by-step section at the beginning of this thread describes how to reproduce the behavior.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

I take it this is still an issue.