Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -10,5 +10,5 @@ applies_to:

In the default behaviour, Filebeat opens the files and keeps them open until it reaches the end of them. In situations when the configured output is blocked (e.g. {{es}} or {{ls}} is unavailable) for a long time, this can cause Filebeat to keep file handlers to files that were deleted from the file system in the mean time. As long as Filebeat keeps the deleted files open, the operating system doesn’t free up the space on disk, which can lead to increase disk utilisation or even out of disk situations.

To mitigate this issue, you can set the [`close_timeout`](/reference/filebeat/filebeat-input-log.md#filebeat-input-log-close-timeout) setting to `5m`. This will ensure every file handler is closed once every 5 minutes, regardless of whether it reached EOF or not. Note that this option can lead to data loss if the file is deleted before Filebeat reaches the end of the file.
To mitigate this issue, you can set the [`close.reader.after_interval`](/reference/filebeat/filebeat-input-filestream.md#filebeat-input-filestream-close-timeout) setting to `5m`. This will ensure every file handler is closed once every 5 minutes, regardless of whether it reached EOF or not. Note that this option can lead to data loss if the file is deleted before Filebeat reaches the end of the file.

2 changes: 1 addition & 1 deletion docs/reference/filebeat/filebeat-cpu.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,5 +8,5 @@

# Filebeat is using too much CPU [filebeat-cpu]

Filebeat might be configured to scan for files too frequently. Check the setting for `scan_frequency` in the `filebeat.yml` config file. Setting `scan_frequency` to less than 1s may cause Filebeat to scan the disk in a tight loop.
Filebeat might be configured to scan for files too frequently. Check the setting for [`prospector.scanner.check_interval`](/reference/filebeat/filebeat-input-filestream.md#filebeat-input-filestream-scan-frequency)) in the `filebeat.yml` config file. Setting `prospector.scanner.check_interval` to less than 1s may cause Filebeat to scan the disk in a tight loop.

Check notice on line 11 in docs/reference/filebeat/filebeat-cpu.md

View workflow job for this annotation

GitHub Actions / build / vale

Elastic.WordChoice: Consider using 'can, might' instead of 'may', unless the term is in the UI.

2 changes: 1 addition & 1 deletion docs/reference/filebeat/filebeat-not-collecting-lines.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
Filebeat might be incorrectly configured or unable to send events to the output. To resolve the issue:

* If using modules, make sure the `var.paths` setting points to the file. If configuring an input manually, make sure the `paths` setting is correct.
* Verify that the file is not older than the value specified by [`ignore_older`](/reference/filebeat/filebeat-input-log.md#filebeat-input-log-ignore-older). `ignore_older` is disable by default so this depends on the value you have set. You can change this behavior by specifying a different value for [`ignore_older`](/reference/filebeat/filebeat-input-log.md#filebeat-input-log-ignore-older).
* Verify that the file is not older than the value specified by [`ignore_older`](/reference/filebeat/filebeat-input-filestream.md#filebeat-input-filestream-ignore-older). `ignore_older` is disabled by default so this depends on the value you have set. You can change this behavior by specifying a different value for [`ignore_older`](/reference/filebeat/filebeat-input-filestream.md#filebeat-input-filestream-ignore-older).

Check notice on line 14 in docs/reference/filebeat/filebeat-not-collecting-lines.md

View workflow job for this annotation

GitHub Actions / build / vale

Elastic.WordChoice: Consider using 'deactivated, deselected, hidden, turned off, unavailable' instead of 'disabled', unless the term is in the UI.
* Make sure that Filebeat is able to send events to the configured output. Run Filebeat in debug mode to determine whether it’s publishing events successfully:

```sh
Expand Down
15 changes: 7 additions & 8 deletions docs/reference/filebeat/open-file-handlers.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,17 +8,16 @@

# Too many open file handlers [open-file-handlers]

Filebeat keeps the file handler open in case it reaches the end of a file so that it can read new log lines in near real time. If Filebeat is harvesting a large number of files, the number of open files can become an issue. In most environments, the number of files that are actively updated is low. The `close_inactive` configuration option should be set accordingly to close files that are no longer active.
Filebeat keeps the file handler open in case it reaches the end of a file so that it can read new log lines in near real time. If Filebeat is harvesting a large number of files, the number of open files can become an issue. In most environments, the number of files that are actively updated is low. The [`close.on_state_change.inactive`](/reference/filebeat/filebeat-input-filestream.md#filebeat-input-filestream-close-inactive) configuration option should be set accordingly to close files that are no longer active.

Check notice on line 11 in docs/reference/filebeat/open-file-handlers.md

View workflow job for this annotation

GitHub Actions / build / vale

Elastic.Wordiness: Consider using 'many' instead of 'a large number of'.

There are additional configuration options that you can use to close file handlers, but all of them should be used carefully because they can have side effects. The options are:

* [`close_renamed`](/reference/filebeat/filebeat-input-log.md#filebeat-input-log-close-renamed)
* [`close_removed`](/reference/filebeat/filebeat-input-log.md#filebeat-input-log-close-removed)
* [`close_eof`](/reference/filebeat/filebeat-input-log.md#filebeat-input-log-close-eof)
* [`close_timeout`](/reference/filebeat/filebeat-input-log.md#filebeat-input-log-close-timeout)
* [`harvester_limit`](/reference/filebeat/filebeat-input-log.md#filebeat-input-log-harvester-limit)
* [`close.on_state_change.renamed`](/reference/filebeat/filebeat-input-filestream.md#filebeat-input-filestream-close-renamed)
* [`close.on_state_change.removed`](/reference/filebeat/filebeat-input-filestream.md#filebeat-input-filestream-close-removed)
* [`close.reader.on_eof`](/reference/filebeat/filebeat-input-filestream.md#filebeat-input-filestream-close-eof)
* [`close.reader.after_interval`](/reference/filebeat/filebeat-input-filestream.md#filebeat-input-filestream-close-timeout)
* [`harvester_limit`](/reference/filebeat/filebeat-input-filestream.md#filebeat-input-filestream-harvester-limit)

The `close_renamed` and `close_removed` options can be useful on Windows to resolve issues related to file rotation. See [Open file handlers cause issues with Windows file rotation](/reference/filebeat/windows-file-rotation.md). The `close_eof` option can be useful in environments with a large number of files that have only very few entries. The `close_timeout` option is useful in environments where closing file handlers is more important than sending all log lines. For more details, see [Inputs](/reference/filebeat/configuration-filebeat-options.md).
The `close.on_state_change.renamed` and `close.on_state_change.removed` options can be useful on Windows to resolve issues related to file rotation. See [Open file handlers cause issues with Windows file rotation](/reference/filebeat/windows-file-rotation.md). The `close.reader.on_eof` option can be useful in environments with a large number of files that have only very few entries. The `close.reader.after_interval` option is useful in environments where closing file handlers is more important than sending all log lines. For more details, see [Inputs](/reference/filebeat/configuration-filebeat-options.md).

Check notice on line 21 in docs/reference/filebeat/open-file-handlers.md

View workflow job for this annotation

GitHub Actions / build / vale

Elastic.Wordiness: Consider using 'many' instead of 'a large number of'.

Make sure that you read the documentation for these configuration options before using any of them.

4 changes: 2 additions & 2 deletions docs/reference/filebeat/reduce-registry-size.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@

# Registry file is too large [reduce-registry-size]

Filebeat keeps the state of each file and persists the state to disk in the registry file. The file state is used to continue file reading at a previous position when Filebeat is restarted. If a large number of new files are produced every day, the registry file might grow to be too large. To reduce the size of the registry file, there are two configuration options available: [`clean_removed`](/reference/filebeat/filebeat-input-log.md#filebeat-input-log-clean-removed) and [`clean_inactive`](/reference/filebeat/filebeat-input-log.md#filebeat-input-log-clean-inactive).
Filebeat keeps the state of each file and persists the state to disk in the registry file. The file state is used to continue file reading at a previous position when Filebeat is restarted. If a large number of new files are produced every day, the registry file might grow to be too large. To reduce the size of the registry file, there are two configuration options available: [`clean_removed`](/reference/filebeat/filebeat-input-filestream.md#filebeat-input-filestream-clean-removed) and [`clean_inactive`](/reference/filebeat/filebeat-input-filestream.md#filebeat-input-filestream-clean-inactive).

Check notice on line 11 in docs/reference/filebeat/reduce-registry-size.md

View workflow job for this annotation

GitHub Actions / build / vale

Elastic.Wordiness: Consider using 'many' instead of 'a large number of'.

For old files that you no longer touch and are ignored (see [`ignore_older`](/reference/filebeat/filebeat-input-log.md#filebeat-input-log-ignore-older)), we recommended that you use `clean_inactive`. If old files get removed from disk, then use the `clean_removed` option.
For old files that you no longer touch and are ignored (see [`ignore_older`](/reference/filebeat/filebeat-input-filestream.md#filebeat-input-filestream-ignore-older)), we recommended that you use `clean_inactive`. If old files get removed from disk, then use the `clean_removed` option.

4 changes: 1 addition & 3 deletions docs/reference/filebeat/windows-file-rotation.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,8 @@ applies_to:

# Open file handlers cause issues with Windows file rotation [windows-file-rotation]

On Windows, you might have problems renaming or removing files because Filebeat keeps the file handlers open. This can lead to issues with the file rotating system. To avoid this issue, you can use the [`close_removed`](/reference/filebeat/filebeat-input-log.md#filebeat-input-log-close-removed) and [`close_renamed`](/reference/filebeat/filebeat-input-log.md#filebeat-input-log-close-renamed) options together.
On Windows, you might have problems renaming or removing files because Filebeat keeps the file handlers open. This can lead to issues with the file rotating system. To avoid this issue, you can use the [`close.on_state_change.removed`](/reference/filebeat/filebeat-input-filestream.md#filebeat-input-filestream-close-removed) and [`close.on_state_change.renamed`](/reference/filebeat/filebeat-input-filestream.md#filebeat-input-filestream-close-renamed) options together.

::::{important}
When you configure these options, files may be closed before the harvester has finished reading the files. If the file cannot be picked up again by the input and the harvester hasn’t finish reading the file, the missing lines will never be sent to the output.
::::


Loading