Connection errors with filebeat as agent and logstash source

[camilisette]

We currently have Filebeat agents picking up logs from different hosts and sending logs to Vector using the logstash source. It works fine for a few days, but after some days of running, we start to see an increase in connection errors in the Filebeat logs.

Errors like:

• "Failed to publish events caused by: read tcp XXX.XX.XXX.XX:XXXX->XXX.XX.XXX.XX:XXXX: i/o timeout"
• "Failed to publish events caused by: write tcp XXX.XX.XXX.XX:XXXX->XXX.XX.XXX.XX:XXXX: use of closed network connection"
• "Failed to publish events caused by: client is not connected"
• "Failed to publish events caused by: EOF filesdf"
• "File cleanup state update error: input outlet closed"
• "Failed to publish events caused by: invalid sequence number received (seq=1024, expected=6)"

Some of the things we noticed:

• With an increase of these errors, we are seeing duplicate logs, we suspect it's because filebeat resends them because either Vector didn't ack or didn't ack in time.
• Not seeing any utilization issues, infact utilization is low. Checked cpu, memory, file descriptor limit, network i/o, storage i/o, etc.
• Important: Restarting the Vector services gets rid of the errors for a couple days, then it resurfaces after it's been running for some days.

Details:
Agent: Filebeat sending logs using the logstash output
Vector: Receives logs using the logstash source
Vector version: 0.42.0, running on EC2

Any ideas on what we could check? Could there be some internal limit in Vector that we're hitting?

[pront]

Hi @camilisette,

It sounds like a stale TCP connection issue. It's hard to diagnose such issues that arise after long running operations. I would use available debugging tools (see here) to get an overview of Vector's internal metrics around the time the errors start appearing vs the metrics during normal operation.