Alloy consumes all available RAM when accidentally scraping a "large" sparse file

[dylannorthrup]

Component(s)

loki.source.file

What's wrong?

This is a repost of a 2+ year old unaddressed promtail bug. It is posted here in response to @JStickler's closing of that issue and saying promtail would receive no future support or updates

This report is an updated version of the original posted by @tiimwsuqld on 15 Nov 2023

Describe the bug Alloy consumes all RAM (doesn't start swapping) and causes the VM to freeze. OOM doesn't appear to kick in. This is caused when /var/log/lastlog ends up in the pattern match, which is a massively sparse file with almost no data in it. Alloy shouldn't consume all memory to read large files.

Expected behavior Alloy should limit used memory (even at startup) so it can't consume everything on the machine causing it to trigger an oom-kill. Yes, this can be avoided by excluding the lastlog file from being processed, but we should have limits on memory usage.

Environment:
* Infrastructure: Google Cloud VM (e2-standard-2, 8GB RAM)
* Deployment tool: docker-compose

Screenshots, Alloy config, or terminal output If applicable, add any output to help explain your problem.

This is a system exhibiting the "lastlog looks really big but really isn't" issue:

hostname> /bin/ls -sh /var/log/lastlog
76K /var/log/lastlog
hostname> /bin/ls -lh /var/log/lastlog
-rw-rw-r--. 1 root utmp 1.2T Apr  8 14:45 /var/log/lastlog
hostname>

Yes, this can be mitigated by not monitoring lastlog or using __path_exclude__ for globs that would include it, but alloy really shouldn't crash systems if it points to a sparse file.

Steps to reproduce

1. Configure a user on a Linux system with a very large UID (1,000,000+ or so). Run ls -l /var/log/lastlog to verify the file size is reported to be at least as high as the amount of RAM on the system.
2. Have a stanza in your config.alloy watching /var/log/lastlog
3. Start alloy.
4. Watch memory yo-yo between normal and 100% while alloy is oom-killed, then restarted by systemd.

System information

Linux 5.14.0-611.27.1 x86_64

Software version

Grafana Alloy 1.13.0

Configuration

loki.write "logs_base" {
    endpoint {
      url = "https://loki.example.com/loki/api/v1/push"

      basic_auth {
        username = "loki_writer"
        password = "loki_password"
      }

    }
    external_labels = {}
  }

  local.file_match "logs_base_lastlog" {
    path_targets = [{
      __address__       = "hostname.example.com",
      __path__          = "/var/log/lastlog",
    }]
  }

  loki.source.file "logs_base_lastlog" {
    targets    = local.file_match.logs_base_lastlog.targets
    forward_to = [loki.write.logs_base.receiver]
  }

Logs

Tip

_{React with 👍 if this issue is important to you.}

[kalleep]

Interesting, not too familiar with /var/log/lastlog but I think I see what the issue is in general.

When we consume lines we read until we hit a newline without any cap, so you could reproduce this by creating a really large file without any newlines and try to read it. The obvious fix would be to cap reading and flush it as a line if we reach the cap.

We don't have anything like that today and it should probably be configurable. Either we have the default value as uncapped to be sure not to break anything or we pick a pretty large default.

[dylannorthrup]

@kalleep, my first comment on the old repo has two links in it. The first is to some (likely naive) go code that reliably demonstrates the two calculation methods. The second link is to a [9+ year old issue in the golang/go repo](https://github.com/golang/go/issues/22735] about sparse files with a focus on tar files. There is a lot of good info there, but what might be of particular interest are the last two comments from @glycerine with links to some code they created. In particular, @glycerine discussions differences between Linux file systems and APFS on the mac.

Cheers!

[kalleep]

@dylannorthrup thanks for the links that clears up some things for sure.

Okay so the draft pr I linked will make it so sparse files won't OOM alloy but we will read a lot of unnecessary data and forward them as lines. I need to think a bit if we could detect sparse files and handle them somehow.