Log files with same inode number in different block Device caused missing log file

[mioyui88]

Describe the bug

I have two log files in different block device. Two log files have same file name and same inode number.

The first file with inode 17 in device fh72h/64882d 
stat /var/lib/kubelet/pods/pod-uid-1/volumes/kubernetes.io~csi/local-pv-id-1/mount/xx/xx/file.log
       Device: fh72h/64882d                 Inode:17 

The second file with inode 17 in device fd53h/64851d 
stat /var/lib/kubelet/pods/pod-uid-2/volumes/kubernetes.io~csi/local-pv-id-2/mount/xx/xx/file.log
       Device: fd53h/64851d                 Inode:17 

When I want collect the two file.log, I find fluentd only collect one file, the other one doesn't show the fluentd tail target and existing list.

tailing paths: target =  /var/lib/kubelet/pods/pod-uid-1/volumes/kubernetes.io~csi/local-pv-id-1/mount/xx/xx/file.log |
  existing = /var/lib/kubelet/pods/pod-uid-1/volumes/kubernetes.io~csi/local-pv-id-1/mount/xx/xx/file.log

To Reproduce

1. mount two block device to different mountpoint
2. cd mountpoint, make new file, I think default inode number is same
3. start fluentd , set config follow_inodes true

Expected behavior

Fluentd can identify inode number in different block device, and collect log.

Your Environment

- Fluentd version:1.16.2
- TD Agent version:
- Operating system: Centos7
- Kernel version:3.10.0-1160.92.1.el7.x86_64

Your Configuration

<source>
  @type tail
  read_from_head true
  skip_refresh_on_startup true
  path /var/lib/kubelet/pods/*/volumes/kubernetes.io~csi/*/mount/namespace/*/*/*.log
  pos_file /var/log/csi_namespace.log.pos
  refresh_interval 5s
  tag "csi_namespace"
  limit_recently_modified 7d
  format /?<message>.*/
</source>

Your Error Log

There is no error log, just can't get the log. And Fluentd print existing just include one log file.

Additional context

No response

[mioyui88]

In addition, I use config
follow_inodes false, enable_stat_watcher true, enable_watch_timer true, open_on_every_update true , still have some problem.
When I mount new volume to /var/lib/kubelet/pods/pod-new/volumes/kubernetes.io~csi/local-pv-new/mount ,
and config tail path is /var/lib/kubelet/pods/*/volumes/kubernetes.io~csi/*/mount/namespace/*/*/*.log
Fluentd can't collect new log in /var/lib/kubelet/pods/pod-new/volumes/kubernetes.io~csi/local-pv-new/mount/namespace/*/*/file.log.
After I restart Fluentd agent, then the log can be collected to Fluentd.

So how can I collect the log file that stored in new mounted path without restarting Fluentd agent.

**Update**
There is no problem with config follow_inodes false, enable_stat_watcher true, enable_watch_timer true, open_on_every_update true.
The reason caused log file missing is because K8S pod get the node mountpoint need config
mountPropagation: HostToContainer

[mioyui88]

@daipom @ashie @repeatedly

[ashie]

Thanks for your report!
Hmm, it seems a specification bug.
The ideal solution is adding device ID to pos file, but doing it is not so easy with keeping backward compatibility.

Possible work around is separating <source> section per devices, although it might be hard to use in this case since mount point is determined dynamically.