Skip to content

loki.source.syslog leaks memory when processing UDP packets #6207

@cpr-nwright

Description

@cpr-nwright

Component(s)

loki.source.syslog

What's wrong?

The UDP code for loki.source.syslog is written without considering that UDP is a connectionless protocol. In
internal/component/loki/source/syslog/internal/syslogtarget/transport.go, the acceptPackets function has the following code:

	for {
		...
		n, addr, err = t.udpConn.ReadFrom(buf)
		if n <= 0 && err != nil {
			level.Warn(t.logger).Log("msg", "failed to read packets", "addr", addr, "err", err)
			continue
		}

		stream, ok := streams[addr.String()]
		if !ok {
			stream = NewConnPipe(addr)
			streams[addr.String()] = stream
			t.openConnections.Add(1)
			go t.handleRcv(stream)
		}
	}

handleRcv has the following loop:

	for {
		datagram := make([]byte, t.maxMessageLength())
		n, err := c.Read(datagram)
		if err != nil {
			if err == io.EOF {
				break
			}

			level.Warn(t.logger).Log("msg", "error reading from pipe", "err", err)
			continue
		}
		...
	}

Because UDP is connectionless, c.Read will never return EOF under normal circumstances. This means handleRcv is an infinite loop, so alloy will hold onto this goroutine, with datagram allocated with t.maxMessageLength(), forever.

Since UDP is connectionless, clients are likely to use different source ports to send syslog messages over time, meaning alloy will have hundreds to thousands of these handleRecv goroutines for each syslog client.

See attached pprof for evidence of memory leak

Steps to reproduce

  1. Start alloy with a loki.source.syslog component with a UDP listener.
  2. Send UDP syslog messages to the listener from a large number of IP:source port pairs
  3. Observe memory leak

System information

No response

Software version

No response

Configuration


Logs


Tip

React with 👍 if this issue is important to you.

Metadata

Metadata

Assignees

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions