Syslog forwarding with UDP

[m1575]

Our current firewall only supports syslog over UDP with port 514. I attempted to follow the Syslog Forwarding guidance in the documentation, exchanging TCP for UDP and specifying port 514/udp in the PublishPort line, but it doesn't seem to be working.
Specific things I changed:

• Added integration for Custom UDP Logs instead of Custom TCP Logs
• While configuring Custom UDP Logs, specified a listen port of 514 and a dataset name of udp.syslog
• In /etc/containers/systemd/lme-fleet-server.container, I changed the PublishPort directive to PublishPort=8220:8220,514:514/udp
• Created an rsyslog configuration /etc/rsyslog.d/60-forward-udp.conf with a forwarding directive of . @@my-lme-ip:514

Am I missing something? I'm not super familiar with rsyslog, podman, or elastic.

[NVivero]

Based on what you shared, your setup looks correct for enabling syslog forwarding over UDP. There are a few things to confirm to make sure the updates you made take effect.

1. If you haven't already done so, restart the container service to ensure the port mapping takes effect. You can use the following commands:

sudo systemctl daemon-reexec
sudo systemctl daemon-reload
sudo systemctl restart podman-lme-fleet-server.service

2. You will also need to check for any port conflicts to ensure nothing (like rsyslog) is already bound to UDP 514. If that's the case, you can have rsyslog receive on 514 and forward to an alternate UDP port that the container can listen on (e.g., 1514).

3. Confirm traffic is hitting the host with sudo tcpdump -i any port 514

Let us know what you see and if you run into anymore issues.

[m1575]

Thanks for the advice.

I changed the syslog forwarding to port 1514 in the Custom UDP Logs integration, the PublishPort directive, and in the rsyslog configuration file 60-forward-udp.conf. I confirmed that my firewall is sending packets to my-lme-ip:1514. I also ran your recommended commands for restarting the container service.

When I run sudo tcpdump -i any port 514, these are my results:

Unfortunately, I'm still not seeing anything from my firewall IP or on port 1514 within the Kibana server.

[NVivero]

The tcpdump you ran was still checking port 514 and not 1514. Can you try sudo tcpdump -i any port 1514 and see if that works?

[m1575]

Here's the results from sudo tcpdump -i any port 1514:

[NVivero]

Do you see the logs in the Discover UI?

[m1575]

No, I still see nothing coming to port 1514.

[NVivero]

Did you restart Fleet after updating PublishPort to 1514? The container won’t bind to the new port until it’s restarted so that may be the issue. You can restart the service with the following commands:

sudo systemctl daemon-reexec
sudo systemctl daemon-reload
sudo systemctl restart podman-lme-fleet-server.service

If that’s already been done and/or logs still aren’t showing up after restarting, run this to check if the agent inside the container is starting the UDP input or hitting errors:
podman logs lme-fleet-server | grep -i input

[m1575]

That worked. Thanks again for your help!