Podman fails to clean dnat rules for restarted/recreated containers

[eduelias]

Issue Description

We have around 1100 machines running podman and some of them present this problem.

After a container having a port exposure (like 3001) restarts, the DNAT rule for the previous version of the container is still left behind and, when the new container spins up, the DNAT rules are added at the bottom, being evaluated last.

Then the port-forwarding fails as the DNAT rules get evaluated top-down and then the the packages are forwarded to the previous IP address, not the current one.

Broken machine:

$ nft list ruleset | grep "tcp dport 3001"
		tcp dport 3001 jump nv_2f259bab_10_88_0_0_nm16_dnat
		tcp dport 3001 dnat ip to 10.88.0.20:3001 <-- Previous IP address
		ip saddr 10.88.0.0/16 tcp dport 3001 jump NETAVARK-HOSTPORT-SETMARK
		ip saddr 127.0.0.1 tcp dport 3001 jump NETAVARK-HOSTPORT-SETMARK
		tcp dport 3001 dnat ip to 10.88.0.28:3001

Working machine:

$ nft list ruleset | grep "tcp dport 3001"
		tcp dport 3001 jump nv_2f259bab_10_88_0_0_nm16_dnat
		ip saddr 10.88.0.0/16 tcp dport 3001 jump NETAVARK-HOSTPORT-SETMARK
		ip saddr 127.0.0.1 tcp dport 3001 jump NETAVARK-HOSTPORT-SETMARK
		tcp dport 3001 dnat ip to 10.88.0.51:3001

I tried to find this issue reported but I didn't. We found something at netavark here but nothing referenced in this repo.

Steps to reproduce the issue

Steps to reproduce the issue

1. Run a container exposing the port 3001
2. Restart several times
3. nft list ruleset | grep "tcp dport 3001"

Describe the results you received

You should see two IPs pointing to the same 3001 port eventually.

Describe the results you expected

Only one IP should point to the same port, ever.

podman info output

# podman info
host:
  arch: arm64
  buildahVersion: 1.38.1
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - pids
  - rdma
  - misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.12-3.fc41.aarch64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.12, commit: '
  cpuUtilization:
    idlePercent: 85.99
    systemPercent: 6.69
    userPercent: 7.32
  cpus: 4
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: iot
    version: "41"
  eventLogger: journald
  freeLocks: 2041
  hostname: 218776-00000000c343d085
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.12.10-200.fc41.aarch64
  linkmode: dynamic
  logDriver: journald
  memFree: 70037504
  memTotal: 965046272
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.13.1-1.fc41.aarch64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.13.1
    package: netavark-1.13.1-1.fc41.aarch64
    path: /usr/libexec/podman/netavark
    version: netavark 1.13.1
  ociRuntime:
    name: crun
    package: crun-1.19.1-1.fc41.aarch64
    path: /usr/bin/crun
    version: |-
      crun version 1.19.1
      commit: 3e32a70c93f5aa5fea69b50256cca7fd4aa23c80
      rundir: /run/user/0/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20250121.g4f2c8e7-2.fc41.aarch64
    version: |
      pasta 0^20250121.g4f2c8e7-2.fc41.aarch64-pasta
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.3.1-1.fc41.aarch64
    version: |-
      slirp4netns version 1.3.1
      commit: e5e368c4f5db6ae75c2fce786e31eef9da6bf236
      libslirp: 4.8.0
      SLIRP_CONFIG_VERSION_MAX: 5
      libseccomp: 2.5.5
  swapFree: 676511744
  swapTotal: 964685824
  uptime: 173h 45m 44.00s (Approximately 7.21 days)
  variant: v8
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 7
    paused: 0
    running: 6
    stopped: 1
  graphDriverName: overlay
  graphOptions:
    overlay.imagestore: /usr/lib/containers/storage
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 14694191104
  graphRootUsed: 10351955968
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "true"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 11
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 5.3.2
  Built: 1737504000
  BuiltTime: Wed Jan 22 00:00:00 2025
  GitCommit: ""
  GoVersion: go1.23.4
  Os: linux
  OsArch: linux/arm64
  Version: 5.3.2

Podman in a container

No

Privileged Or Rootless

Privileged

Upstream Latest Release

No

Additional environment details

Additional environment details

Additional information

It happens to ~10% of our fleet of 1100 machines.

[Luap99]

Please provide an exact reproducer, ideally a script we can run to reproduce or at least provide proper podman commands you use.

Because

Run a container exposing the port 3001
Restart several times

can mean a lot of things.

Second you are using outdated versions, upstream only support the latest version so make to to reproduce with that. Fedora has generally up to date packages, did you never update your system? Otherwise I don't see why you have such old packages.

As general thing if stuff leaks than this is likely to failed cleanup so check the full journal for any podman/netavark related errors

[github-actions]

A reviewer has determined we need more information to understand the reported issue. A comment on what is missing should be provided. Be certain you:

• provide an exact reproducer where possible
• verify you have provided all relevant information - minimum is podman info
• answer any follow up questions

If no response to the needs-info is provided in 30 days, this issue may be closed by our stale bot.

For more information on reporting issues on this repository, consult our issue guide.

[TaylorRichberger]

Are you using a Quadlet systemd service with LogNamespace set? I actually have been running into an issue that sounds exactly like this, and I can reproduce it as shown below.

The long and short of it is that LogNamespace does not play nicely with Quadlet, and breaks not just mount but netavark cleanup as well. I'm posting this here in the hopes that another person who hits this issue in the same way finds this via an internet search instead of coming up mostly empty-handed like I did. There are other issues mentioning LogNamespace, but not with these symptoms that I could find. In particular, network: netavark: setns: IO error: Invalid argument (os error 22) finds nothing relevant.

# cat /etc/containers/systemd/test-dnat.container
[Unit]
Description=DNAT rule issue test container

[Container]
Image=docker.io/python:3.14-slim
Exec=python -mhttp.server -d / 80
PublishPort=127.0.0.1:8123:80

[Install]
WantedBy=multi-user.target default.target

# systemctl restart test-dnat
# systemctl restart test-dnat
# systemctl restart test-dnat
# iptables-save | grep 8123
-A NETAVARK-DN-1D8721804F16F -s 10.88.0.0/16 -d 127.0.0.1/32 -p tcp -m tcp --dport 8123 -j NETAVARK-HOSTPORT-SETMARK
-A NETAVARK-DN-1D8721804F16F -s 127.0.0.1/32 -d 127.0.0.1/32 -p tcp -m tcp --dport 8123 -j NETAVARK-HOSTPORT-SETMARK
-A NETAVARK-DN-1D8721804F16F -d 127.0.0.1/32 -p tcp -m tcp --dport 8123 -j DNAT --to-destination 10.88.0.65:80
-A NETAVARK-HOSTPORT-DNAT -p tcp -m tcp --dport 8123 -m comment --comment "dnat name: podman id: 9da1f4f088f7da2cc3388b2ccad74354b657fcdadf5fcfbb214ad8a5cdc00585" -j NETAVARK-DN-1D8721804F16F

# systemctl stop test-dnat
# vim /etc/containers/systemd/test-dnat.container
# cat /etc/containers/systemd/test-dnat.container
[Unit]
Description=DNAT rule issue test container

[Service]
LogNamespace=testdnat

[Container]
Image=docker.io/python:3.14-slim
Exec=python -mhttp.server -d / 80
PublishPort=127.0.0.1:8123:80

[Install]
WantedBy=multi-user.target default.target

# systemctl daemon-reload
# systemctl restart test-dnat
# systemctl restart test-dnat
# systemctl restart test-dnat
# iptables-save | grep 8123
-A NETAVARK-DN-1D8721804F16F -s 10.88.0.0/16 -d 127.0.0.1/32 -p tcp -m tcp --dport 8123 -j NETAVARK-HOSTPORT-SETMARK
-A NETAVARK-DN-1D8721804F16F -s 127.0.0.1/32 -d 127.0.0.1/32 -p tcp -m tcp --dport 8123 -j NETAVARK-HOSTPORT-SETMARK
-A NETAVARK-DN-1D8721804F16F -d 127.0.0.1/32 -p tcp -m tcp --dport 8123 -j DNAT --to-destination 10.88.0.66:80
-A NETAVARK-DN-1D8721804F16F -d 127.0.0.1/32 -p tcp -m tcp --dport 8123 -j DNAT --to-destination 10.88.0.67:80
-A NETAVARK-DN-1D8721804F16F -d 127.0.0.1/32 -p tcp -m tcp --dport 8123 -j DNAT --to-destination 10.88.0.68:80
-A NETAVARK-DN-1D8721804F16F -d 127.0.0.1/32 -p tcp -m tcp --dport 8123 -j DNAT --to-destination 10.88.0.69:80
-A NETAVARK-HOSTPORT-DNAT -p tcp -m tcp --dport 8123 -m comment --comment "dnat name: podman id: 45ccc52d15c905a4b9b88d56bbc8a636b1c835c1d9003edac8ab9e34c8e54760" -j NETAVARK-DN-1D8721804F16F
-A NETAVARK-HOSTPORT-DNAT -p tcp -m tcp --dport 8123 -m comment --comment "dnat name: podman id: 78e805b1d547449807c21f271fdeeac113cd1b24ea729a80ef114d3f4da21dfb" -j NETAVARK-DN-1D8721804F16F
-A NETAVARK-HOSTPORT-DNAT -p tcp -m tcp --dport 8123 -m comment --comment "dnat name: podman id: b9f809775c627a8854d15f03114cc174d8e1c587ec77249b647c9988736e5e45" -j NETAVARK-DN-1D8721804F16F
-A NETAVARK-HOSTPORT-DNAT -p tcp -m tcp --dport 8123 -m comment --comment "dnat name: podman id: cde262538f820d27e73dbbb68469efdfe8ba44ed2cd2dfaff2ab030231042eda" -j NETAVARK-DN-1D8721804F16F

# journalctl -u test-dnat --namespace testdnat | tail
Nov 21 15:50:54 my-hostname test-dnat[9905]: b9f809775c627a8854d15f03114cc174d8e1c587ec77249b647c9988736e5e45
Nov 21 15:51:10 my-hostname test-dnat[9963]: time="2025-11-21T15:51:10-05:00" level=warning msg="StopSignal SIGTERM failed to stop container systemd-test-dnat in 10 seconds, resorting to SIGKILL"
Nov 21 15:51:10 my-hostname podman[9963]: 2025-11-21 15:51:10.49921425 -0500 EST m=+10.053561495 container died b9f809775c627a8854d15f03114cc174d8e1c587ec77249b647c9988736e5e45 (image=docker.io/library/python:3.14-slim, name=systemd-test-dnat, PODMAN_SYSTEMD_UNIT=test-dnat.service)
Nov 21 15:51:10 my-hostname podman[9963]: 2025-11-21 15:51:10.542560148 -0500 EST m=+10.096907384 container remove b9f809775c627a8854d15f03114cc174d8e1c587ec77249b647c9988736e5e45 (image=docker.io/library/python:3.14-slim, name=systemd-test-dnat, PODMAN_SYSTEMD_UNIT=test-dnat.service)
Nov 21 15:51:10 my-hostname test-dnat[9963]: Error: cleaning up container b9f809775c627a8854d15f03114cc174d8e1c587ec77249b647c9988736e5e45: removing container b9f809775c627a8854d15f03114cc174d8e1c587ec77249b647c9988736e5e45 network: netavark: setns: IO error: Invalid argument (os error 22)
Nov 21 15:51:10 my-hostname podman[9995]: 2025-11-21 15:51:10.641602145 -0500 EST m=+0.031949725 container create cde262538f820d27e73dbbb68469efdfe8ba44ed2cd2dfaff2ab030231042eda (image=docker.io/library/python:3.14-slim, name=systemd-test-dnat, PODMAN_SYSTEMD_UNIT=test-dnat.service)
Nov 21 15:51:10 my-hostname podman[9995]: 2025-11-21 15:51:10.629351926 -0500 EST m=+0.019699522 image pull 88fe63acba12eb55b9f21a121536ad36b2fdbd13c46e08159eaad7a1f3adcfcc docker.io/python:3.14-slim
Nov 21 15:51:10 my-hostname podman[9995]: 2025-11-21 15:51:10.753091325 -0500 EST m=+0.143438934 container init cde262538f820d27e73dbbb68469efdfe8ba44ed2cd2dfaff2ab030231042eda (image=docker.io/library/python:3.14-slim, name=systemd-test-dnat, PODMAN_SYSTEMD_UNIT=test-dnat.service)
Nov 21 15:51:10 my-hostname podman[9995]: 2025-11-21 15:51:10.758138019 -0500 EST m=+0.148485605 container start cde262538f820d27e73dbbb68469efdfe8ba44ed2cd2dfaff2ab030231042eda (image=docker.io/library/python:3.14-slim, name=systemd-test-dnat, PODMAN_SYSTEMD_UNIT=test-dnat.service)
Nov 21 15:51:10 my-hostname test-dnat[9995]: cde262538f820d27e73dbbb68469efdfe8ba44ed2cd2dfaff2ab030231042eda
#

[calvinbui]

I sometimes experience this same issue with containers started using podman run and then restarted using podman restart. No quadlets involved, so can't be related to LogNamespace.

I'm using macvlan containers which get IP addresses via DHCP through netavark.

When I restart the container, sometimes Traefik (web proxy) can't get the new IP address and fails to serve it. When i try to access the private IP directly, there is no response either. If I exec into the container and curl the port its serving, I do get a response. At this point, I delete the container and recreate it again and it works.

[RomLecat]

I have the same issue and it can be consistently reproduced.
I have a fairly "stock" Rocky Linux 10 with Podman 5.6.0 and firewalld enabled.

I use quadlets to manage podman containers, such as:

[Unit]
Description=Zabbix-proxy
After=network-online.target

[Container]
ContainerName=zabbix-proxy
Image=registry.internal/zabbix/zabbix-proxy-mysql:7.4-alpine-latest
AutoUpdate=registry
Volume=/mnt/data/zabbix-proxy/tls:/etc/zabbix/tls:z
Volume=/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem:/etc/ssl/certs/ca-certificates.crt:ro,z
Network=zabbix.network
PublishPort=0.0.0.0:10051:10051
PublishPort=[::]:10051:10051
Environment="TZ=Europe/Paris"
Environment="DB_SERVER_HOST=mariadb"
Environment="MYSQL_USER=zabbix-proxy"
Environment="MYSQL_PASSWORD=zabbix-proxy"
Environment="MYSQL_DATABASE=zabbix-proxy"
Environment="ZBX_CACHESIZE=1G"
Environment="ZBX_STARTPINGERS=12"
Environment="ZBX_STARTIPMIPOLLERS=8"
Environment="ZBX_TLSCONNECT=cert"
Environment="ZBX_TLSCAFILE=/etc/zabbix/tls/ca.pem"
Environment="ZBX_TLSCERTFILE=/etc/zabbix/tls/cert.pem"
Environment="ZBX_TLSKEYFILE=/etc/zabbix/tls/cert.key.pem"
AddCapability=net_raw
PodmanArgs=--umask=0002

[Service]
Environment=REGISTRY_AUTH_FILE=/etc/registry/auth.json
Restart=always

[Install]
WantedBy=default.target

If I restarted the container, I have the following errors in logs:

Dec 14 17:10:15 podman[1668]: 2025-12-14 17:10:15.350859658 +0000 UTC m=+0.793438245 container died e9334c36f992f82ffeadcf9f88aa10259ec5118f8ae8d9335ac93ae33ec754f6 (image=registr>
Dec 14 17:10:15 zabbix-proxy[1714]: internal:0:0-0: Error: Could not process rule: No such file or directory
Dec 14 17:10:15 zabbix-proxy[1714]: internal:0:0-0: Error: Could not process rule: No such file or directory
Dec 14 17:10:15 zabbix-proxy[1714]: internal:0:0-0: Error: Could not process rule: No such file or directory
Dec 14 17:10:15 zabbix-proxy[1714]: internal:0:0-0: Error: Could not process rule: No such file or directory
Dec 14 17:10:15 zabbix-proxy[1714]: internal:0:0-0: Error: Could not process rule: No such file or directory
Dec 14 17:10:15 zabbix-proxy[1714]: internal:0:0-0: Error: Could not process rule: No such file or directory
Dec 14 17:10:15 podman[1668]: 2025-12-14 17:10:15.417202267 +0000 UTC m=+0.859780842 container remove e9334c36...

And duplicate rules are created in nftables.

A workaround is to use firewall_driver=iptables (in /etc/containers/containers.conf) and iptables-nft.

[chirayuk]

Reproduces on Fedora 43 / netavark 1.17.2 — without LogNamespace

I'm seeing this same issue on a fresh Fedora 43 (Server Edition) install with current packages:

• Podman: 5.7.1
• Netavark: 1.17.2-1.fc43
• Kernel: 6.18.13-200.fc43.x86_64
• Arch: x86_64

My setup uses Podman Quadlet (systemd-managed containers) on a custom bridge network (10.90.0.0/16). The Forgejo container publishes two ports: 0.0.0.0:222:22/tcp (SSH)
and 127.0.0.1:3000:3000/tcp (HTTP).

Importantly, my Quadlet does NOT use LogNamespace. The [Service] section is minimal:

[Service]
Restart=always
TimeoutStartSec=90
RestartSec=5

After a single systemctl restart, the old DNAT rules remain and new ones are appended. Since nftables evaluates top-down, port 222 traffic hits the first (stale) rule
and gets forwarded to a dead container IP:

chain nv_a87cef0e_10_90_0_0_nm16_dnat {
    # First restart — stale, points to dead IP
    tcp dport 222 dnat ip to 10.90.128.1:22
    ...
    # Second restart — also stale
    tcp dport 222 dnat ip to 10.90.128.3:22
    ...
    # Third restart — also stale
    tcp dport 222 dnat ip to 10.90.128.5:22
    ...
    # Current container — correct but never reached
    tcp dport 222 dnat ip to 10.90.128.6:22
}

No errors in journalctl -u ck-forgejo.service during the stop/start cycle — netavark exits cleanly but silently leaves the old rules behind.

[Societus]

Same issue here — additional data point from an LXC environment on iptables backend

Environment:

• Podman 5.4.1 (rootful), managed via Quadlet
• Netavark 1.12.1, aardvark-dns 1.12.2
• Ubuntu 25.04 (Plucky), kernel 6.17.13-2-pve
• Running inside an unprivileged Proxmox LXC container (may be relevant — see below)
• Storage: overlay on ZFS (Native Overlay Diff: true, metacopy: false)
• Runtime: crun 1.20
• Firewall driver: iptables (distro default, not manually configured)
• 2 vCPU, 2GB RAM

Not using LogNamespace in the [Service] section, confirming chirayuk's finding that this bug exists independently of that trigger.

Minor note: pasta segfaults on version check (/usr/bin/pasta --version → signal: segmentation fault). Not related to this issue but noting it in case it affects netavark's rootless fallback path on this distro/kernel combo.

Reproducer

Two-container setup (app + valkey cache) on a custom Quadlet bridge network. Issue manifests after 3-5 systemctl restart cycles.

Network definition (/etc/containers/systemd/searxng.network):

[Network]
NetworkName=searxng-net

Cache container (/etc/containers/systemd/searxng-valkey.container):

[Unit]
Description=SearXNG Valkey Cache

[Container]
ContainerName=searxng-valkey
Image=docker.io/valkey/valkey:9-alpine
Exec=valkey-server --save 30 1 --loglevel warning
Volume=/root/valkey-data:/data:Z
Network=searxng.network

[Service]
Restart=always

[Install]
WantedBy=default.target

App container (/etc/containers/systemd/searxng-core.container):

[Unit]
Description=SearXNG Core Service
After=searxng-valkey.service
Requires=searxng-valkey.service

[Container]
ContainerName=searxng-core
Image=docker.io/searxng/searxng:latest
PublishPort=8080:8080
EnvironmentFile=/root/core-config/.env
Volume=/root/core-config/:/etc/searxng/:ro,Z
Volume=/root/core-data:/var/cache/searxng/:Z
Network=searxng.network

[Service]
Restart=unless-stopped

[Install]
WantedBy=default.target

Steps to reproduce:

systemctl enable --now searxng-network.service searxng-valkey.service searxng-core.service
# Verify it works
curl -s -o /dev/null -w "%{http_code}" http://localhost:8080/
# Restart 3-5 times
systemctl restart searxng-core
systemctl restart searxng-core
systemctl restart searxng-core
# Now broken
curl -v http://localhost:8080/
# Expected: 200 OK
# Actual: * Trying [::1]:8080... Connection refused
#          * Trying 127.0.0.1:8080... No route to host

Rule accumulation (iptables-save format)

After 3-5 restarts:

:NETAVARK-DN-BFF2EFB0060F1 - [0:0]
-A NETAVARK-DN-BFF2EFB0060F1 -s 10.89.0.0/24 -d 10.3.0.14/32 -p tcp -m tcp --dport 8080 -j NETAVARK-HOSTPORT-SETMARK
-A NETAVARK-DN-BFF2EFB0060F1 -s 10.3.0.14/32 -d 10.3.0.14/32 -p tcp -m tcp --dport 8080 -j NETAVARK-HOSTPORT-SETMARK
-A NETAVARK-DN-BFF2EFB0060F1 -d 10.3.0.14/32 -p tcp -m tcp --dport 8080 -j DNAT --to-destination 10.89.0.98:8080
-A NETAVARK-DN-BFF2EFB0060F1 -d 10.3.0.14/32 -p tcp -m tcp --dport 8080 -j DNAT --to-destination 10.89.0.142:8080
-A NETAVARK-DN-BFF2EFB0060F1 -d 10.3.0.14/32 -p tcp -m tcp --dport 8080 -j DNAT --to-destination 10.89.0.201:8080
-A NETAVARK-DN-BFF2EFB0060F1 -d 10.3.0.14/32 -p tcp -m tcp --dport 8080 -j DNAT --to-destination 10.89.0.243:8080

Only the last IP (10.89.0.243) is the live container. The first DNAT rule catches all traffic and sends it to a dead IP → ICMP "No route to host" from the bridge.

Three orphaned veth interfaces (veth0, veth1, veth3) also remained attached to the podman1 bridge from previous runs.

Journal output during stop/start

No netavark errors in journalctl during the stop/start cycle. The teardown silently failed to clean up old rules while the setup silently appended new ones. This is consistent with chirayuk's observation (Fedora 43, netavark 1.17.2). I did not see the Error: Could not process rule: No such file or directory that RomLecat reported on Rocky Linux 10.

podman info

host:
  arch: amd64
  buildahVersion: 1.39.3
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon_2.1.12-4_amd64
    version: 'conmon version 2.1.12, commit: unknown'
  cpus: 2
  distribution:
    codename: plucky
    distribution: ubuntu
    version: "25.04"
  eventLogger: journald
  kernel: 6.17.13-2-pve
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns_1.12.2-2_amd64
      version: aardvark-dns 1.12.2
    package: netavark_1.12.1-9_amd64
    version: netavark 1.12.1
  ociRuntime:
    name: crun
    package: crun_1.20-1syncable1_amd64
    version: crun version 1.20
  os: linux
  rootless: false
  seccompEnabled: true
  selinuxEnabled: false
store:
  graphDriverName: overlay
  graphStatus:
    Backing Filesystem: zfs
    Native Overlay Diff: "true"
    Supports shifting: "false"
    Using metacopy: "false"
version:
  APIVersion: 5.4.1
  Version: 5.4.1
  GoVersion: go1.24.1

Additional observations

firewall_driver=iptables workaround does not help here. RomLecat suggested switching to firewall_driver=iptables in /etc/containers/containers.conf as a workaround. I'm already on the iptables backend (distro default) and still hitting this bug, so that workaround is not universal.

LXC namespace nesting may be a factor. Inside an unprivileged LXC, the container's network namespace is nested inside the LXC's own namespace. Netfilter teardown hooks may behave differently in a nested namespace — the process may lack netlink capabilities to fully tear down chains in a parent namespace. I'd expect cleanup to work correctly either way, but the nesting could explain why teardown silently fails here while working on bare metal for some users.

Diagnostic tools give false reassurance. podman port reports the mapping as correct. ss -tlnp shows conmon listening on 0.0.0.0:8080. Both appear healthy even when the service is completely unreachable. The only reliable diagnostic:

iptables -t nat -L NETAVARK-DN-BFF2EFB0060F1 -n --line-numbers

Persistent workaround

For anyone hitting this, a one-time cleanup works but doesn't prevent recurrence. I'm running an ExecStopPost= hook in the Quadlet .container unit that automatically cleans stale state after the container stops:

/usr/local/sbin/cleanup-netavark.sh:

#!/bin/bash
# Workaround for: https://github.com/containers/podman/issues/27516
# Cleans stale netavark DNAT rules and orphaned veth interfaces after container stop.
set -euo pipefail

# Remove orphaned veth interfaces from podman bridges
for bridge in $(ip -br link show type bridge | awk '{print $1}'); do
  ip link show type veth master "$bridge" -br 2>/dev/null | \
    awk '{print $1}' | while read -r veth; do
      [ -n "$veth" ] && ip link del "$veth" 2>/dev/null || true
    done
done

# Flush all netavark iptables DNAT chains
iptables -t nat -L -n 2>/dev/null | awk '/^Chain NETAVARK-DN-/ {print $2}' | while read -r chain; do
  iptables -t nat -F "$chain" 2>/dev/null || true
done

# Flush all netavark iptables POSTROUTING chains
iptables -t nat -L -n 2>/dev/null | awk '/^Chain NETAVARK-POST-/ {print $2}' | while read -r chain; do
  iptables -t nat -F "$chain" 2>/dev/null || true
done

# If using nftables backend, flush netavark tables too
if command -v nft &>/dev/null && nft list tables 2>/dev/null | grep -q netavark; then
  nft flush table inet netavark 2>/dev/null || true
fi

Added to the Quadlet unit:

[Service]
ExecStopPost=/usr/local/sbin/cleanup-netavark.sh
Restart=unless-stopped

This ensures stale rules and orphaned interfaces are cleaned before the next start. Netavark recreates a single clean DNAT rule on start.

For one-time manual cleanup:

systemctl stop searxng-core.service
ip link show type veth master podman1 -br | awk '{print $1}' | xargs -r ip link del
iptables -t nat -L -n | awk '/^Chain NETAVARK-DN-/ {print $2}' | xargs -I{} iptables -t nat -F {}
iptables -t nat -L -n | awk '/^Chain NETAVARK-POST-/ {print $2}' | xargs -I{} iptables -t nat -F {}
systemctl start searxng-core.service

Happy to provide additional logs, packet captures, or test any proposed fix.

[tpapaioa]

Hi @Luap99 , I'm seeing this issue on Fedora 44, updated to the latest available packages. With a basic httpd quadlet configuration that publishes both IPv4 and IPv6 ports:

$ rpm -q netavark podman
netavark-1.17.2-1.fc44.x86_64
podman-5.8.2-1.fc44.x86_64

$ cat /etc/containers/systemd/httpd.network 
[Network]
NetworkName=httpd
IPv6=true

$ cat /etc/containers/systemd/httpd.container 
[Container]
ContainerName=httpd
Network=httpd.network
Image=registry.access.redhat.com/ubi10/httpd-24
PublishPort=80:8080
PublishPort=[::]:80:8080

[Unit]
StartLimitIntervalSec=0

[Service]
Restart=always
RestartSec=5s

[Install]
WantedBy=default.target

Every restart of the container appends to the DNAT rules in nft, without removing the ones for the previous IP address:

$ sudo nft list ruleset
[...]
	chain nv_4420a6eb_10_89_0_0_nm24_dnat {
		ip saddr 10.89.0.0/24 tcp dport 80 jump NETAVARK-HOSTPORT-SETMARK
		ip saddr 127.0.0.1 tcp dport 80 jump NETAVARK-HOSTPORT-SETMARK
		tcp dport 80 dnat ip to 10.89.0.2:8080
		ip saddr 10.89.0.0/24 tcp dport 80 jump NETAVARK-HOSTPORT-SETMARK
		ip saddr 127.0.0.1 tcp dport 80 jump NETAVARK-HOSTPORT-SETMARK
		tcp dport 80 dnat ip to 10.89.0.3:8080
		ip saddr 10.89.0.0/24 tcp dport 80 jump NETAVARK-HOSTPORT-SETMARK
		ip saddr 127.0.0.1 tcp dport 80 jump NETAVARK-HOSTPORT-SETMARK
		tcp dport 80 dnat ip to 10.89.0.4:8080
	}

	chain nv_4420a6eb_fd3e-f0ab-bd12-d646--_nm64_dnat {
		ip6 saddr fd3e:f0ab:bd12:d646::/64 tcp dport 80 jump NETAVARK-HOSTPORT-SETMARK
		tcp dport 80 dnat ip6 to [fd3e:f0ab:bd12:d646::2]:8080
		ip6 saddr fd3e:f0ab:bd12:d646::/64 tcp dport 80 jump NETAVARK-HOSTPORT-SETMARK
		tcp dport 80 dnat ip6 to [fd3e:f0ab:bd12:d646::2]:8080
		ip6 saddr fd3e:f0ab:bd12:d646::/64 tcp dport 80 jump NETAVARK-HOSTPORT-SETMARK
		tcp dport 80 dnat ip6 to [fd3e:f0ab:bd12:d646::3]:8080
		ip6 saddr fd3e:f0ab:bd12:d646::/64 tcp dport 80 jump NETAVARK-HOSTPORT-SETMARK
		tcp dport 80 dnat ip6 to [fd3e:f0ab:bd12:d646::3]:8080
		ip6 saddr fd3e:f0ab:bd12:d646::/64 tcp dport 80 jump NETAVARK-HOSTPORT-SETMARK
		tcp dport 80 dnat ip6 to [fd3e:f0ab:bd12:d646::4]:8080
		ip6 saddr fd3e:f0ab:bd12:d646::/64 tcp dport 80 jump NETAVARK-HOSTPORT-SETMARK
		tcp dport 80 dnat ip6 to [fd3e:f0ab:bd12:d646::4]:8080
	}
[...]

Interestingly, if I have the exact same setup, except without the IPv6 port published (commenting out PublishPort=[::]:80:8080 but leaving the rest of the quadlet configuration the same), then I don't see this issue, and the rules for the old IP addresses are removed after every container restart:

$ sudo nft list ruleset
[...]
	chain nv_4420a6eb_10_89_0_0_nm24_dnat {
		ip saddr 10.89.0.0/24 tcp dport 80 jump NETAVARK-HOSTPORT-SETMARK
		ip saddr 127.0.0.1 tcp dport 80 jump NETAVARK-HOSTPORT-SETMARK
		tcp dport 80 dnat ip to 10.89.0.3:8080
	}

	chain nv_4420a6eb_fd3e-f0ab-bd12-d646--_nm64_dnat {
		ip6 saddr fd3e:f0ab:bd12:d646::/64 tcp dport 80 jump NETAVARK-HOSTPORT-SETMARK
		tcp dport 80 dnat ip6 to [fd3e:f0ab:bd12:d646::3]:8080
	}
[...]

$ sudo systemctl restart httpd
$ sudo systemctl restart httpd

$ sudo nft list ruleset
[...]
	chain nv_4420a6eb_10_89_0_0_nm24_dnat {
		ip saddr 10.89.0.0/24 tcp dport 80 jump NETAVARK-HOSTPORT-SETMARK
		ip saddr 127.0.0.1 tcp dport 80 jump NETAVARK-HOSTPORT-SETMARK
		tcp dport 80 dnat ip to 10.89.0.5:8080
	}

	chain nv_4420a6eb_fd3e-f0ab-bd12-d646--_nm64_dnat {
		ip6 saddr fd3e:f0ab:bd12:d646::/64 tcp dport 80 jump NETAVARK-HOSTPORT-SETMARK
		tcp dport 80 dnat ip6 to [fd3e:f0ab:bd12:d646::5]:8080
	}
}
[...]

I've also seen this same behavior on Red Hat Enterprise Linux 10.1 (podman-5.6.0) and 10.2 (podman-5.8.2). I hope that helps. I'm happy to provide any further info you need.