accept events seemingly with reversed client and server IP/ports

[dnwe]

Describe the bug

Periodically we see a number of inbound accept events where the client and server IP and port in the connection tuple seem like they are the wrong way around and the static listener port (e.g., 443, 2040, 8081, 9090, 9091 etc.) is tracked as the client port (fd.cport) and the ephemeral range port is tracked as the server port (fd.sport).

Here's a dump of sysdig --modern-bpf -j 'evt.type in (accept,accept4,listen) and evt.dir=< and fd.sport>=32768' using the same bpf level as falco that seems to show these coming from the bpf driver:

{"evt.cpu":0,"evt.dir":"<","evt.info":"fd=36(<4t>172.20.0.1:2040->172.20.0.1:48693) tuple=172.20.0.1:2040->172.20.0.1:48693 queuepct=0 queuelen=0 queuemax=0","evt.num":1120567,"evt.outputtime":1750075498103591233,"evt.type":"accept","proc.name":"kubelet","proc.pid":8594,"thread.tid":8594}
{"evt.cpu":0,"evt.dir":"<","evt.info":"fd=68(<4t>172.30.31.146:9090->172.30.35.245:37786) tuple=172.30.31.146:9090->172.30.35.245:37786 queuepct=0 queuelen=0 queuemax=0","evt.num":4179273,"evt.outputtime":1750077601098514978,"evt.type":"accept","proc.name":"prometheus","proc.pid":105796,"thread.tid":105941}
{"evt.cpu":0,"evt.dir":"<","evt.info":"fd=47(<4t>172.30.31.148:9091->172.30.35.245:52250) tuple=172.30.31.148:9091->172.30.35.245:52250 queuepct=0 queuelen=0 queuemax=0","evt.num":4631781,"evt.outputtime":1750077911077389135,"evt.type":"accept","proc.name":"prometheus","proc.pid":105796,"thread.tid":105951}
{"evt.cpu":0,"evt.dir":"<","evt.info":"fd=48(<4t>172.30.102.133:8081->172.30.35.245:34032) tuple=172.30.102.133:8081->172.30.35.245:34032 queuepct=0 queuelen=0 queuemax=0","evt.num":4642001,"evt.outputtime":1750077918462537590,"evt.type":"accept","proc.name":"prometheus","proc.pid":105796,"thread.tid":105954}
{"evt.cpu":1,"evt.dir":"<","evt.info":"fd=7(<4t>172.21.0.1:443->172.30.35.234:54098) tuple=172.21.0.1:443->172.30.35.234:54098 queuepct=0 queuelen=0 queuemax=0","evt.num":5123972,"evt.outputtime":1750078248974904459,"evt.type":"accept","proc.name":"pod_nanny","proc.pid":12358,"thread.tid":12378}
{"evt.cpu":0,"evt.dir":"<","evt.info":"fd=33(<4t>172.21.225.209:9090->172.30.35.219:48134) tuple=172.21.225.209:9090->172.30.35.219:48134 queuepct=0 queuelen=0 queuemax=0","evt.num":5686812,"evt.outputtime":1750078633237538685,"evt.type":"accept","proc.name":"customer-metric","proc.pid":35913,"thread.tid":35918}
{"evt.cpu":0,"evt.dir":"<","evt.info":"fd=8(<4t>172.30.222.85:9090->172.30.35.245:51912) tuple=172.30.222.85:9090->172.30.35.245:51912 queuepct=0 queuelen=0 queuemax=0","evt.num":6424831,"evt.outputtime":1750079137273125101,"evt.type":"accept","proc.name":"prometheus","proc.pid":105796,"thread.tid":105951}
{"evt.cpu":1,"evt.dir":"<","evt.info":"fd=8(<4t>172.30.222.85:9090->172.30.35.245:55938) tuple=172.30.222.85:9090->172.30.35.245:55938 queuepct=0 queuelen=0 queuemax=0","evt.num":8058449,"evt.outputtime":1750080253497731940,"evt.type":"accept","proc.name":"prometheus","proc.pid":105796,"thread.tid":105941}
{"evt.cpu":1,"evt.dir":"<","evt.info":"fd=55834574866(<4t>172.30.12.69:9195->172.30.35.222:34134) tuple=172.30.12.69:9195->172.30.35.222:34134 queuepct=0 queuelen=0 queuemax=0","evt.num":8959420,"evt.outputtime":1750080868580866165,"evt.type":"accept4","proc.name":"hyperion","proc.pid":38060,"thread.tid":38060}
{"evt.cpu":0,"evt.dir":"<","evt.info":"fd=62(<4t>172.30.12.74:9090->172.30.35.245:57548) tuple=172.30.12.74:9090->172.30.35.245:57548 queuepct=0 queuelen=0 queuemax=0","evt.num":9430541,"evt.outputtime":1750081189555311594,"evt.type":"accept","proc.name":"prometheus","proc.pid":105796,"thread.tid":105941}
{"evt.cpu":0,"evt.dir":"<","evt.info":"fd=8(<4t>172.30.110.6:9091->172.30.35.245:55826) tuple=172.30.110.6:9091->172.30.35.245:55826 queuepct=0 queuelen=0 queuemax=0","evt.num":10722260,"evt.outputtime":1750082070794044987,"evt.type":"accept","proc.name":"prometheus","proc.pid":105796,"thread.tid":105951}

In falco terms these trigger our unexpected network traffic rules. Do you have any idea of what could be the cause of this behaviour?

How to reproduce it

Unknown

Expected behaviour

I'd expect the accept events to have the server port of the listener

Screenshots

N/A

Environment

• Falco version: {"default_driver_version":"8.1.0+driver","driver_api_version":"8.0.0","driver_schema_version":"3.6.0","engine_version":"50","engine_version_semver":"0.50.0","falco_version":"0.41.0","libs_version":"0.21.0","plugin_api_version":"3.11.0"}

• System info: Linux version 6.8.0-60-generic (buildd@lcy02-amd64-054) (x86_64-linux-gnu-gcc-13 (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0, GNU ld (GNU Binutils for Ubuntu) 2.42) #63-Ubuntu SMP PREEMPT_DYNAMIC Tue Apr 15 19:04:15 UTC 2025

• Cloud provider or hardware configuration:
• OS: Ubuntu 24.04.2 LTS (Noble Numbat)

• Kernel: 6.8.0-60-generic

• Installation method: from source

Additional context

N/A

[dnwe]

Hmm looking into the modern_bpf code, it seems like it purposefully does this flipped ordering of the tuple when packing?

driver/modern_bpf/helpers/store/auxmap_store_params.h#L710-L720

Although actually now I've re-read that code it actually looks correct and for an accept event it is definitely calling that with INBOUND here

[dnwe]

I did also test with the pre-compiled release of 0.41.2 (just to rule out any compiler optimization causing the issue), but still see the problem

Another example would be this event, which seemingly tracks as an 'accept' into an ephemeral port within the prometheus pod:

  "output": "2025-07-02T09:14:35.010298653+0000: Warning Inbound network traffic to pod: prometheus-1 in container: prometheus on unexpected port: 37610 from Client IP: 172.30.222.222 to Server IP: 172.30.216.244. Protocol: tcp, Client Port: 9090, Event: type=accept res=SUCCESS rawres=8 name=172.30.222.222:9090->172.30.216.244:37610 connected=true is_server=true, container_id=d23569ab7615 container_name=prometheus container_image_repository=example.com/ns/prometheus container_image_tag=1721 k8s_pod_name=prometheus-1 k8s_ns_name=default",
  "output_fields": {
    "fd.name": "172.30.222.222:9090->172.30.216.244:37610",
    "evt.rawres": 8,
    "fd.l4proto": "tcp",
    "container.image.tag": "1721",
    "evt.type": "accept",
    "proc.cmdline": "prometheus --config.file=/etc/prometheus/prometheus.yaml --enable-feature=auto-gomaxprocs --log.format=json --log.level=warn --storage.tsdb.path=/prometheus/storage --storage.tsdb.retention.time=672h --web.console.libraries=/usr/share/prometheus/console_libraries --web.console.templates=/usr/share/prometheus/consoles --web.enable-lifecycle",
    "fd.sport": 37610,
    "fd.sip": "172.30.216.244",
    "evt.time.iso8601": 1751447675010298624,
    "k8s.ns.name": "default",
    "container.id": "d23569ab7615",
    "fd.connected": true,
    "fd.is_server": true,
    "container.name": "prometheus",
    "k8s.pod.name": "prometheus-1",
    "container.image.repository": "example.com/ns/prometheus",
    "fd.cport": 9090,
    "evt.res": "SUCCESS",
    "fd.cip": "172.30.222.222"
  },

But, 172.30.222.222:9090 is the metrics endpoint on the zookeeper-4 pod, so it seems much more likely this event was in the opposite direction and should have tracked as an outbound connect from prometheus-1:37610 to zookeeper-4:9090

NAME                  READY  STATUS   RESTARTS  AGE        IP
default/prometheus-1  4/4    Running  2         6d18h      172.30.216.244
default/zookeeper-4   2/2    Running  0         17h33m49s  172.30.222.222

[dnwe]

Originally I'd been reporting the issue as "accept event has the IP and port numbers around the wrong way", but now I actually wonder if it's instead a problem with misrepresenting a connect event as an accept event somehow?

[dnwe]

@leogr any thoughts on this one? Happy to collect more info, but still a little mystified by the underlying cause at the moment

[leogr]

Hey @dnwe

I have no clue at the moment. I vaguely recall that a related issue was addressed. Btw, if you notice any discrepancy between Falco 0.41 and sysdig, it might be since the latter is using an older libs version.

@cc @falcosecurity/libs-maintainers for visibility.

We will investigate.

/milestone 0.42.0

[dnwe]

@leogr I intentionally patched my sysdig source so I could build and use the same libs version as falco for testing

[leogr]

@leogr I intentionally patched my sysdig source so I could build and use the same libs version as falco for testing

Got it. Thanks for the clarification!

[leogr]

Any update on this?

Unfortunatelly we have had no cycles to work on this, so
/milestone 0.43.0

[poiana]

Issues go stale after 90d of inactivity.

Mark the issue as fresh with /remove-lifecycle stale.

Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle stale

[leogr]

/remove-lifecycle stale