Unable to connect to public IP of host/reverse proxy container

[cadaniel]

Issue Description

I have several hosts behind a traefik revers proxy, which include

• matrix server -> matrix.houseofnuts.ca
• matrix auth server -. auth.matrix.houseofnuts.ca
• authentik server -> auth.houseofnuts.ca

I'm unable to curl to these urls from the containers, while curl to a known good url (google.com) works just fine. The host is also able to properly curl the urls

Steps to reproduce the issue

Steps to reproduce the issue

1. podman exec -it matrix-service curl -vvv https://auth.matrix.houseofnuts.ca

Describe the results you received

16:17:42.435981 [0-x] == Info: [READ] client_reset, clear readers
16:17:42.437302 [0-0] == Info: Host auth.matrix.houseofnuts.ca:443 was resolved.
16:17:42.437678 [0-0] == Info: IPv6: (none)
16:17:42.437885 [0-0] == Info: IPv4: 69.165.169.36
16:17:42.438187 [0-0] == Info: [HTTPS-CONNECT] adding wanted h2
16:17:42.438492 [0-0] == Info: [HTTPS-CONNECT] added
16:17:42.438743 [0-0] == Info: [HTTPS-CONNECT] connect, init
16:17:42.439097 [0-0] == Info:   Trying 69.165.169.36:443...
16:17:42.439610 [0-0] == Info: [HTTPS-CONNECT] connect -> 0, done=0
16:17:42.439953 [0-0] == Info: [HTTPS-CONNECT] Curl_conn_connect(block=0) -> 0, done=0
16:17:42.440418 [0-0] == Info: [HTTPS-CONNECT] adjust_pollset -> 1 socks
16:17:42.440764 [0-0] == Info: [HTTPS-CONNECT] connect -> 0, done=0
16:17:42.441088 [0-0] == Info: [HTTPS-CONNECT] Curl_conn_connect(block=0) -> 0, done=0
16:17:42.441462 [0-0] == Info: [HTTPS-CONNECT] adjust_pollset -> 1 socks
16:17:43.442975 [0-0] == Info: [HTTPS-CONNECT] connect -> 0, done=0
16:17:43.443424 [0-0] == Info: [HTTPS-CONNECT] Curl_conn_connect(block=0) -> 0, done=0
16:17:43.443870 [0-0] == Info: [HTTPS-CONNECT] adjust_pollset -> 1 socks

Describe the results you expected

Successfull https connection

podman info output

Client:        Podman Engine
Version:       5.4.2
API Version:   5.4.2
Go Version:    go1.24.4
Built:         Sun Dec 21 11:42:01 2025
Build Origin:  Debian
OS/Arch:       linux/amd64

host:
  arch: amd64
  buildahVersion: 1.39.3
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon_2.1.12-4_amd64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.12, commit: unknown'
  cpuUtilization:
    idlePercent: 83.46
    systemPercent: 3.62
    userPercent: 12.93
  cpus: 16
  databaseBackend: sqlite
  distribution:
    codename: trixie
    distribution: debian
    version: "13"
  eventLogger: journald
  freeLocks: 2014
  hostname: extapps
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.12.74+deb13+1-amd64
  linkmode: dynamic
  logDriver: journald
  memFree: 3260633088
  memTotal: 23089668096
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns_1.14.0-3_amd64
      path: /usr/lib/podman/aardvark-dns
      version: aardvark-dns 1.14.0
    package: netavark_1.14.0-2_amd64
    path: /usr/lib/podman/netavark
    version: netavark 1.14.0
  ociRuntime:
    name: crun
    package: crun_1.21-1_amd64
    path: /usr/bin/crun
    version: |-
      crun version 1.21
      commit: 10269840aa07fb7e6b7e1acff6198692d8ff5c88
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt_0.0~git20250503.587980c-2+deb13u1_amd64
    version: ""
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns_1.2.1-1.1_amd64
    version: |-
      slirp4netns version 1.2.1
      commit: 09e31e92fa3d2a1d3ca261adaeb012c8d75a8194
      libslirp: 4.8.0
      SLIRP_CONFIG_VERSION_MAX: 5
      libseccomp: 2.6.0
  swapFree: 8149528576
  swapTotal: 8149528576
  uptime: 71h 59m 28.00s (Approximately 2.96 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries: {}
store:
  configFile: /home/debian/.config/containers/storage.conf
  containerStore:
    number: 24
    paused: 0
    running: 24
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/debian/.local/share/containers/storage
  graphRootAllocated: 148325314560
  graphRootUsed: 41983725568
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 75
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/debian/.local/share/containers/storage/volumes
version:
  APIVersion: 5.4.2
  BuildOrigin: Debian
  Built: 1766335321
  BuiltTime: Sun Dec 21 11:42:01 2025
  GitCommit: ""
  GoVersion: go1.24.4
  Os: linux
  OsArch: linux/amd64
  Version: 5.4.2

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

No

Additional environment details

sudo firewall-cmd --list-all
public (default, active)
  target: default
  ingress-priority: 0
  egress-priority: 0
  icmp-block-inversion: no
  interfaces: 
  sources: 
  services: dhcpv6-client ssh
  ports: 5432/tcp 2456/udp 2457/udp 2458/udp 33080/tcp 10000/tcp 10000/udp 33080/udp 3478/udp 3478/tcp
  protocols: 
  forward: yes
  masquerade: yes
  forward-ports: 
        port=80:proto=tcp:toport=8000:toaddr=
        port=443:proto=tcp:toport=4430:toaddr=
  source-ports: 
  icmp-blocks: 
  rich rules: 

Additional information

# matrix-auth-service-matrix-auth-service.container
[Unit]
Requires=matrix-auth-db.service matrix-auth-pod.service
After=matrix-auth-db.service matrix-auth-pod.service

[Container]
ContainerName=matrix-auth-service
Image=ghcr.io/element-hq/matrix-authentication-service:latest
AutoUpdate=registry
Pod=matrix-auth.pod
Network=runite.network
AddHost=auth.houseofnuts.ca:host-gateway
AddHost=matrix.houseofnuts.ca:host-gateway

Volume=/home/debian/containers/matrix-auth/config:/app/config:z
Secret=postgres-debian-pwd,type=env,target=POSTGRES_PASSWORD
Secret=bouncer-api-key,type=env,target=CrowdsecLapiKey
Secret=bouncer-api-key,target=bouncer-api-key,mode=0777

Environment=MAS_CONFIG=/app/config/config.yaml

Label=traefik.enable=true 
Label=traefik.http.routers.ma-http.rule=Host(`auth.matrix.houseofnuts.ca`) 
Label=traefik.http.routers.ma-http.entrypoints=web
Label=traefik.http.routers.ma-http.middlewares=https-redirect
Label=traefik.http.middlewares.https-redirect.redirectscheme.scheme=https
Label=traefik.http.middlewares.https-redirect.redirectscheme.permanent=true
Label=traefik.http.routers.ma-https.rule=Host(`auth.matrix.houseofnuts.ca`) 
Label=traefik.http.routers.ma-https.entrypoints=websecure 
Label=traefik.http.services.ma-https.loadbalancer.server.port=8080 
Label=traefik.http.routers.ma-https.tls.certresolver=letsencrypt 

Label=traefik.http.routers.ma-https.middlewares=crowdsec@file 

[Service]
Restart=always

[Install]
WantedBy=default.target
Restart=always

[Luap99]

69.165.169.36

Is that the main ip address of the server you are running podman on?
If you use pasta it copies the default ip address of the system into the container namespace, so any connection to said ip will stay within the container namespace from within and not reach the host.

In particular to reach the host use host.containers.internal or remap your own domain names with --add-host=service1.example.com:host-gateway
#26502

[cadaniel]

69.165.169.36

Is that the main ip address of the server you are running podman on? If you use pasta it copies the default ip address of the system into the container namespace, so any connection to said ip will stay within the container namespace from within and not reach the host.

In particular to reach the host use host.containers.internal or remap your own domain names with --add-host=service1.example.com:host-gateway #26502

No it's not the main IP of the server. that would be 192.168.50.22

My router forwards ports to the server.

I have tried AddHost=auth.houseofnuts.ca:host-gateway to no avail.

Forwarding all traffic to the local host of the machine would also not be routed by the reverse proxy.

[Luap99]

No it's not the main IP of the server. that would be 192.168.50.22

So basically you tried to connect from container A to the public ip of the router which then (did you verify this) NATs the request back to your server and then should be firewall remapped to port 4430 (per your rule) on the server and from there again redirected by rootlessport (our p[odman port forwarder when used with custom networks)?

A lot to unwrap, ideally add packet captures at all places and see where the packets get lost/rejected.

For the add-host case it is more direct as it should be from container to server back to other container. IF you connect from the host to the main address 192.168.50.22:433 does this work? AFAIK firewalld has some special behaviors regarding the port forwarding with yones where local connecting might not get port remapped to 4430.

Again packet captures would be helpful.

---

And as general thing we only support the latest version upstream, so please test with the latest podman/pasta versions, passt_0.0~git20250503.587980c-2+deb13u1_amd64 is close to a year old it seems.