file source with AWS EFS (NFS)

[czerasz]

Question

Trying to get logs from a path which mounts EFS.
With the provided configuration it's not working 🙁

Does Vector file type source support this?

Vector Config

sources:
  my_logs:
    type: file
    include:
      - '/var/log/some.log' # works with file on EBS
      - '/mnt/some/path/on/efs/some.log' # does not work with file on EFS
    read_from: end

Vector Logs

No response

[thomasqueirozb]

The file source just uses regular file open calls (and in a Unix system where/how the fs is mounted shouldn't matter). Are you sure this file exists?

[czerasz]

yes, I tested with an existing /mnt/some/path/on/efs/some.log file

[czerasz]

Inotify does not work on EFS... Could this be related?

Recently discovered that other tools distinguish between "inotify" or "poll"...

[thomasqueirozb]

inotify is not used with the current file source implementation (see #22803), the filesystem is polled every glob_minimum_cooldown_ms. This is the line that fetches paths from a glob path. I can only suspect that some fs call from within the glob crate is failing and glob::glob_with's iterator is not returning the EFS path. It may also be that the Vector process doesn't have enough permissions, but as I mentioned before you would expect regular file reads to come from vector. If you verify that the EFS path is not returned by glob::glob_with (which I highly suspect is the case) then I suggest opening an issue in their repo.