Skip to content

Commit 61237d2

Browse files
andrzej-stencelmergify[bot]
authored andcommitted
docs(filebeat): update troubleshooting docs to link to Filestream options (#51788)
(cherry picked from commit 39668eb)
1 parent 0e4d9d0 commit 61237d2

6 files changed

Lines changed: 13 additions & 16 deletions

File tree

docs/reference/filebeat/faq-deleted-files-are-not-freed.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,5 +10,5 @@ applies_to:
1010

1111
In the default behaviour, Filebeat opens the files and keeps them open until it reaches the end of them. In situations when the configured output is blocked (e.g. {{es}} or {{ls}} is unavailable) for a long time, this can cause Filebeat to keep file handlers to files that were deleted from the file system in the mean time. As long as Filebeat keeps the deleted files open, the operating system doesn’t free up the space on disk, which can lead to increase disk utilisation or even out of disk situations.
1212

13-
To mitigate this issue, you can set the [`close_timeout`](/reference/filebeat/filebeat-input-log.md#filebeat-input-log-close-timeout) setting to `5m`. This will ensure every file handler is closed once every 5 minutes, regardless of whether it reached EOF or not. Note that this option can lead to data loss if the file is deleted before Filebeat reaches the end of the file.
13+
To mitigate this issue, you can set the [`close.reader.after_interval`](/reference/filebeat/filebeat-input-filestream.md#filebeat-input-filestream-close-timeout) setting to `5m`. This will ensure every file handler is closed once every 5 minutes, regardless of whether it reached EOF or not. Note that this option can lead to data loss if the file is deleted before Filebeat reaches the end of the file.
1414

docs/reference/filebeat/filebeat-cpu.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,5 +8,5 @@ applies_to:
88

99
# Filebeat is using too much CPU [filebeat-cpu]
1010

11-
Filebeat might be configured to scan for files too frequently. Check the setting for `scan_frequency` in the `filebeat.yml` config file. Setting `scan_frequency` to less than 1s may cause Filebeat to scan the disk in a tight loop.
11+
Filebeat might be configured to scan for files too frequently. Check the setting for [`prospector.scanner.check_interval`](/reference/filebeat/filebeat-input-filestream.md#filebeat-input-filestream-scan-frequency)) in the `filebeat.yml` config file. Setting `prospector.scanner.check_interval` to less than 1s may cause Filebeat to scan the disk in a tight loop.
1212

docs/reference/filebeat/filebeat-not-collecting-lines.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ applies_to:
1111
Filebeat might be incorrectly configured or unable to send events to the output. To resolve the issue:
1212

1313
* If using modules, make sure the `var.paths` setting points to the file. If configuring an input manually, make sure the `paths` setting is correct.
14-
* Verify that the file is not older than the value specified by [`ignore_older`](/reference/filebeat/filebeat-input-log.md#filebeat-input-log-ignore-older). `ignore_older` is disable by default so this depends on the value you have set. You can change this behavior by specifying a different value for [`ignore_older`](/reference/filebeat/filebeat-input-log.md#filebeat-input-log-ignore-older).
14+
* Verify that the file is not older than the value specified by [`ignore_older`](/reference/filebeat/filebeat-input-filestream.md#filebeat-input-filestream-ignore-older). `ignore_older` is disabled by default so this depends on the value you have set. You can change this behavior by specifying a different value for [`ignore_older`](/reference/filebeat/filebeat-input-filestream.md#filebeat-input-filestream-ignore-older).
1515
* Make sure that Filebeat is able to send events to the configured output. Run Filebeat in debug mode to determine whether it’s publishing events successfully:
1616

1717
```sh

docs/reference/filebeat/open-file-handlers.md

Lines changed: 7 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -8,17 +8,16 @@ applies_to:
88

99
# Too many open file handlers [open-file-handlers]
1010

11-
Filebeat keeps the file handler open in case it reaches the end of a file so that it can read new log lines in near real time. If Filebeat is harvesting a large number of files, the number of open files can become an issue. In most environments, the number of files that are actively updated is low. The `close_inactive` configuration option should be set accordingly to close files that are no longer active.
11+
Filebeat keeps the file handler open in case it reaches the end of a file so that it can read new log lines in near real time. If Filebeat is harvesting a large number of files, the number of open files can become an issue. In most environments, the number of files that are actively updated is low. The [`close.on_state_change.inactive`](/reference/filebeat/filebeat-input-filestream.md#filebeat-input-filestream-close-inactive) configuration option should be set accordingly to close files that are no longer active.
1212

1313
There are additional configuration options that you can use to close file handlers, but all of them should be used carefully because they can have side effects. The options are:
1414

15-
* [`close_renamed`](/reference/filebeat/filebeat-input-log.md#filebeat-input-log-close-renamed)
16-
* [`close_removed`](/reference/filebeat/filebeat-input-log.md#filebeat-input-log-close-removed)
17-
* [`close_eof`](/reference/filebeat/filebeat-input-log.md#filebeat-input-log-close-eof)
18-
* [`close_timeout`](/reference/filebeat/filebeat-input-log.md#filebeat-input-log-close-timeout)
19-
* [`harvester_limit`](/reference/filebeat/filebeat-input-log.md#filebeat-input-log-harvester-limit)
15+
* [`close.on_state_change.renamed`](/reference/filebeat/filebeat-input-filestream.md#filebeat-input-filestream-close-renamed)
16+
* [`close.on_state_change.removed`](/reference/filebeat/filebeat-input-filestream.md#filebeat-input-filestream-close-removed)
17+
* [`close.reader.on_eof`](/reference/filebeat/filebeat-input-filestream.md#filebeat-input-filestream-close-eof)
18+
* [`close.reader.after_interval`](/reference/filebeat/filebeat-input-filestream.md#filebeat-input-filestream-close-timeout)
19+
* [`harvester_limit`](/reference/filebeat/filebeat-input-filestream.md#filebeat-input-filestream-harvester-limit)
2020

21-
The `close_renamed` and `close_removed` options can be useful on Windows to resolve issues related to file rotation. See [Open file handlers cause issues with Windows file rotation](/reference/filebeat/windows-file-rotation.md). The `close_eof` option can be useful in environments with a large number of files that have only very few entries. The `close_timeout` option is useful in environments where closing file handlers is more important than sending all log lines. For more details, see [Inputs](/reference/filebeat/configuration-filebeat-options.md).
21+
The `close.on_state_change.renamed` and `close.on_state_change.removed` options can be useful on Windows to resolve issues related to file rotation. See [Open file handlers cause issues with Windows file rotation](/reference/filebeat/windows-file-rotation.md). The `close.reader.on_eof` option can be useful in environments with a large number of files that have only very few entries. The `close.reader.after_interval` option is useful in environments where closing file handlers is more important than sending all log lines. For more details, see [Inputs](/reference/filebeat/configuration-filebeat-options.md).
2222

2323
Make sure that you read the documentation for these configuration options before using any of them.
24-

docs/reference/filebeat/reduce-registry-size.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ applies_to:
88

99
# Registry file is too large [reduce-registry-size]
1010

11-
Filebeat keeps the state of each file and persists the state to disk in the registry file. The file state is used to continue file reading at a previous position when Filebeat is restarted. If a large number of new files are produced every day, the registry file might grow to be too large. To reduce the size of the registry file, there are two configuration options available: [`clean_removed`](/reference/filebeat/filebeat-input-log.md#filebeat-input-log-clean-removed) and [`clean_inactive`](/reference/filebeat/filebeat-input-log.md#filebeat-input-log-clean-inactive).
11+
Filebeat keeps the state of each file and persists the state to disk in the registry file. The file state is used to continue file reading at a previous position when Filebeat is restarted. If a large number of new files are produced every day, the registry file might grow to be too large. To reduce the size of the registry file, there are two configuration options available: [`clean_removed`](/reference/filebeat/filebeat-input-filestream.md#filebeat-input-filestream-clean-removed) and [`clean_inactive`](/reference/filebeat/filebeat-input-filestream.md#filebeat-input-filestream-clean-inactive).
1212

13-
For old files that you no longer touch and are ignored (see [`ignore_older`](/reference/filebeat/filebeat-input-log.md#filebeat-input-log-ignore-older)), we recommended that you use `clean_inactive`. If old files get removed from disk, then use the `clean_removed` option.
13+
For old files that you no longer touch and are ignored (see [`ignore_older`](/reference/filebeat/filebeat-input-filestream.md#filebeat-input-filestream-ignore-older)), we recommended that you use `clean_inactive`. If old files get removed from disk, then use the `clean_removed` option.
1414

docs/reference/filebeat/windows-file-rotation.md

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -8,10 +8,8 @@ applies_to:
88

99
# Open file handlers cause issues with Windows file rotation [windows-file-rotation]
1010

11-
On Windows, you might have problems renaming or removing files because Filebeat keeps the file handlers open. This can lead to issues with the file rotating system. To avoid this issue, you can use the [`close_removed`](/reference/filebeat/filebeat-input-log.md#filebeat-input-log-close-removed) and [`close_renamed`](/reference/filebeat/filebeat-input-log.md#filebeat-input-log-close-renamed) options together.
11+
On Windows, you might have problems renaming or removing files because Filebeat keeps the file handlers open. This can lead to issues with the file rotating system. To avoid this issue, you can use the [`close.on_state_change.removed`](/reference/filebeat/filebeat-input-filestream.md#filebeat-input-filestream-close-removed) and [`close.on_state_change.renamed`](/reference/filebeat/filebeat-input-filestream.md#filebeat-input-filestream-close-renamed) options together.
1212

1313
::::{important}
1414
When you configure these options, files may be closed before the harvester has finished reading the files. If the file cannot be picked up again by the input and the harvester hasn’t finish reading the file, the missing lines will never be sent to the output.
1515
::::
16-
17-

0 commit comments

Comments
 (0)