Merge branch 'main' into fix/s3-poller-backup-delete

Author: MichaelKatsoulis

NOTICE.txt

@@ -12840,11 +12840,11 @@ OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 
 --------------------------------------------------------------------------------
 Dependency : github.com/elastic/go-quark
-Version: v0.3.0
+Version: v0.6.0
 Licence type (autodetected): Apache-2.0
 --------------------------------------------------------------------------------
 
-Contents of probable licence file $GOMODCACHE/github.com/elastic/go-quark@v0.3.0/LICENSE.txt:
+Contents of probable licence file $GOMODCACHE/github.com/elastic/go-quark@v0.6.0/LICENSE.txt:
 
 
                                  Apache License

changelog/fragments/1776242399-filestream-include-file-fingerprint-opt-in.yaml

@@ -0,0 +1,12 @@
+kind: enhancement
+summary: "Filestream: include_file_fingerprint controls log.file.fingerprint in events."
+description: |
+  Add an `include_file_fingerprint` option (default: `true`) to the filestream
+  input. By default `log.file.fingerprint` is included in published events when
+  `file_identity.fingerprint` is configured, preserving the previous behaviour.
+  Set `include_file_fingerprint: false` to omit the field and reduce network,
+  indexing, and storage costs at scale. A still-growing fingerprint is never
+  added to events; only the completed SHA-256 is published.
+component: filebeat
+pr: https://github.com/elastic/beats/pull/50129
+issue: https://github.com/elastic/beats/issues/50724

changelog/fragments/1776242399-filestream-include-file-fingerprint.yaml

@@ -0,0 +1,13 @@
+kind: enhancement
+summary: Add Filestream harvester metrics to monitoring logs
+description: |
+  The following Filestream metrics are added to the logs and the `/stats` HTTP endpoint:
+   - files_ingested_percent_100
+   - files_ingested_percent_95_99
+   - files_ingested_percent_lt_95
+  
+  These gauges are aggregated across all running Filestream inputs and
+  count active plain-file harvesters only.
+
+component: filebeat
+issue: https://github.com/elastic/beats/issues/36653

changelog/fragments/1783018351-add-filestream-harvester-metrics.yaml

@@ -0,0 +1,3 @@
+kind: enhancement
+summary: update quark to 0.6.0
+component: auditbeat

changelog/fragments/1783413804-auditbeat-quark-0.6.yaml

@@ -10,5 +10,5 @@ applies_to:
 
 In the default behaviour, Filebeat opens the files and keeps them open until it reaches the end of them.  In situations when the configured output is blocked (e.g. {{es}} or {{ls}} is unavailable) for a long time, this can cause Filebeat to keep file handlers to files that were deleted from the file system in the mean time. As long as Filebeat keeps the deleted files open, the operating system doesn’t free up the space on disk, which can lead to increase disk utilisation or even out of disk situations.
 
-To mitigate this issue, you can set the [`close_timeout`](/reference/filebeat/filebeat-input-log.md#filebeat-input-log-close-timeout) setting to `5m`. This will ensure every file handler is closed once every 5 minutes, regardless of whether it reached EOF or not. Note that this option can lead to data loss if the file is deleted before Filebeat reaches the end of the file.
+To mitigate this issue, you can set the [`close.reader.after_interval`](/reference/filebeat/filebeat-input-filestream.md#filebeat-input-filestream-close-timeout) setting to `5m`. This will ensure every file handler is closed once every 5 minutes, regardless of whether it reached EOF or not. Note that this option can lead to data loss if the file is deleted before Filebeat reaches the end of the file.
 

docs/reference/filebeat/faq-deleted-files-are-not-freed.md

@@ -8,5 +8,5 @@ applies_to:
 
 # Filebeat is using too much CPU [filebeat-cpu]
 
-Filebeat might be configured to scan for files too frequently. Check the setting for `scan_frequency` in the `filebeat.yml` config file. Setting `scan_frequency` to less than 1s may cause Filebeat to scan the disk in a tight loop.
+Filebeat might be configured to scan for files too frequently. Check the setting for [`prospector.scanner.check_interval`](/reference/filebeat/filebeat-input-filestream.md#filebeat-input-filestream-scan-frequency)) in the `filebeat.yml` config file. Setting `prospector.scanner.check_interval` to less than 1s may cause Filebeat to scan the disk in a tight loop.
 

docs/reference/filebeat/filebeat-cpu.md

@@ -441,14 +441,14 @@ file_identity.fingerprint: ~
 
     `growing` {applies_to}`stack: ga 9.5.0+`
   :   When `true` (default), files smaller than the fingerprint size (`offset` + `length`) are tracked using the bytes available so far, instead of being skipped until they grow large enough. Once a file reaches the fingerprint size, it's automatically migrated to the regular SHA-256 fingerprint, with no data duplication. Refer to [Enhanced fingerprint](/reference/filebeat/file-identity.md#file-identity-fingerprint-growing) for details.
-      
+
       Set to `false` to restore the pre-9.5 behavior.
 
       ```yaml
       file_identity.fingerprint:
         growing: true
       ```
-    
+
 
 **`native`**
 :   Differentiates between files using their inodes and device IDs. This is the default file identity in Filebeat versions prior to 9.0.0.
@@ -723,7 +723,7 @@ This option is not supported on Windows.
 ```yaml {applies_to}
 stack: ga 9.5.0
 ```
-Controls whether `log.file.fingerprint` is added to published events. Only takes effect when `file_identity.fingerprint` is configured. Defaults to `false`. The file path (`log.file.path`) is always present in events regardless of this setting.
+Controls whether `log.file.fingerprint` is added to published events. Only takes effect when `file_identity.fingerprint` is configured. Defaults to `true`. The file path (`log.file.path`) is always present in events regardless of this setting.
 
 ### `exclude_lines` [filebeat-input-filestream-exclude-lines]
 
@@ -1371,22 +1371,28 @@ Note: Each metric listed has a corresponding gzip_* counterpart (e.g.,
 the same data but exclusively for GZIP compressed files. The original metrics
 provide the total count, including both plain and GZIP files.
 
-### Harvester metrics [_harvester_metrics]
+### Scanner and harvester metrics [_harvester_metrics]
 
 ```{applies_to}
 stack: ga 9.5+
 ```
 
-The `filestream` input also exposes scanner metrics under
+The `filestream` input also exposes scanner and harvester progress metrics under
 `.monitoring.metrics.filebeat.filestream` in monitoring logs and under
 `filebeat.filestream` in the `/stats` HTTP endpoint output. These metrics are
 aggregate gauges across all running `filestream` inputs. They are updated after
-each scanner pass and reset when inputs stop.
+each scanner pass and reset when inputs stop. Harvester progress metrics measure
+how much data active plain-file harvesters have read from their files. They do
+not measure output publishing or acknowledgment progress. GZIP files and files
+ignored by `filestream` settings or state are excluded.
 
 | Metric | Description |
 | --- | --- |
 | `files_empty` | Number of matched files that are empty. |
 | `files_ignored` | Number of matched files ignored by `filestream` settings or state, such as `prospector.scanner.exclude_files`, `ignore_older`, or `ignore_inactive`. |
+| `files_ingested_percent_100` | Number of active plain-file harvesters whose read offset is at or beyond the scanner-observed file size. |
+| `files_ingested_percent_95_99` | Number of active plain-file harvesters whose read offset is at least 95% and less than 100% of the scanner-observed file size. |
+| `files_ingested_percent_lt_95` | Number of active plain-file harvesters whose read offset is less than 95% of the scanner-observed file size. |
 | `files_matched` | Number of filesystem path matches returned by the configured `paths` globs before duplicate, ignore, and ingestibility filtering. |
 | `files_no_ingest_target` | Number of matched non-empty files that did not produce an ingest target, such as duplicate matches, files that are too small to fingerprint or symlinks to already known files. |
 | `files_unique` | Number of unique files that produced ingest targets after scanner filtering and de-duplication. |

docs/reference/filebeat/filebeat-input-filestream.md

@@ -11,7 +11,7 @@ applies_to:
 Filebeat might be incorrectly configured or unable to send events to the output. To resolve the issue:
 
 * If using modules, make sure the `var.paths` setting points to the file. If configuring an input manually, make sure the `paths` setting is correct.
-* Verify that the file is not older than the value specified by [`ignore_older`](/reference/filebeat/filebeat-input-log.md#filebeat-input-log-ignore-older). `ignore_older` is disable by default so this depends on the value you have set. You can change this behavior by specifying a different value for [`ignore_older`](/reference/filebeat/filebeat-input-log.md#filebeat-input-log-ignore-older).
+* Verify that the file is not older than the value specified by [`ignore_older`](/reference/filebeat/filebeat-input-filestream.md#filebeat-input-filestream-ignore-older). `ignore_older` is disabled by default so this depends on the value you have set. You can change this behavior by specifying a different value for [`ignore_older`](/reference/filebeat/filebeat-input-filestream.md#filebeat-input-filestream-ignore-older).
 * Make sure that Filebeat is able to send events to the configured output. Run Filebeat in debug mode to determine whether it’s publishing events successfully:
 
     ```sh

docs/reference/filebeat/filebeat-not-collecting-lines.md

@@ -8,17 +8,16 @@ applies_to:
 
 # Too many open file handlers [open-file-handlers]
 
-Filebeat keeps the file handler open in case it reaches the end of a file so that it can read new log lines in near real time. If Filebeat is harvesting a large number of files, the number of open files can become an issue. In most environments, the number of files that are actively updated is low. The `close_inactive` configuration option should be set accordingly to close files that are no longer active.
+Filebeat keeps the file handler open in case it reaches the end of a file so that it can read new log lines in near real time. If Filebeat is harvesting a large number of files, the number of open files can become an issue. In most environments, the number of files that are actively updated is low. The [`close.on_state_change.inactive`](/reference/filebeat/filebeat-input-filestream.md#filebeat-input-filestream-close-inactive) configuration option should be set accordingly to close files that are no longer active.
 
 There are additional configuration options that you can use to close file handlers, but all of them should be used carefully because they can have side effects. The options are:
 
-* [`close_renamed`](/reference/filebeat/filebeat-input-log.md#filebeat-input-log-close-renamed)
-* [`close_removed`](/reference/filebeat/filebeat-input-log.md#filebeat-input-log-close-removed)
-* [`close_eof`](/reference/filebeat/filebeat-input-log.md#filebeat-input-log-close-eof)
-* [`close_timeout`](/reference/filebeat/filebeat-input-log.md#filebeat-input-log-close-timeout)
-* [`harvester_limit`](/reference/filebeat/filebeat-input-log.md#filebeat-input-log-harvester-limit)
+* [`close.on_state_change.renamed`](/reference/filebeat/filebeat-input-filestream.md#filebeat-input-filestream-close-renamed)
+* [`close.on_state_change.removed`](/reference/filebeat/filebeat-input-filestream.md#filebeat-input-filestream-close-removed)
+* [`close.reader.on_eof`](/reference/filebeat/filebeat-input-filestream.md#filebeat-input-filestream-close-eof)
+* [`close.reader.after_interval`](/reference/filebeat/filebeat-input-filestream.md#filebeat-input-filestream-close-timeout)
+* [`harvester_limit`](/reference/filebeat/filebeat-input-filestream.md#filebeat-input-filestream-harvester-limit)
 
-The `close_renamed` and `close_removed` options can be useful on Windows to resolve issues related to file rotation. See [Open file handlers cause issues with Windows file rotation](/reference/filebeat/windows-file-rotation.md). The `close_eof` option can be useful in environments with a large number of files that have only very few entries. The `close_timeout` option is useful in environments where closing file handlers is more important than sending all log lines. For more details, see [Inputs](/reference/filebeat/configuration-filebeat-options.md).
+The `close.on_state_change.renamed` and `close.on_state_change.removed` options can be useful on Windows to resolve issues related to file rotation. See [Open file handlers cause issues with Windows file rotation](/reference/filebeat/windows-file-rotation.md). The `close.reader.on_eof` option can be useful in environments with a large number of files that have only very few entries. The `close.reader.after_interval` option is useful in environments where closing file handlers is more important than sending all log lines. For more details, see [Inputs](/reference/filebeat/configuration-filebeat-options.md).
 
 Make sure that you read the documentation for these configuration options before using any of them.
-