[9.5](backport #51793) filestream: default include_file_fingerprint to true

[mergify]

Proposed commit message

filestream: default include_file_fingerprint to true

Soft-revert of #50129, which made log.file.fingerprint in published
events opt-in via include_file_fingerprint (default false). Removing the
field by default is technically a breaking change that we don't
understand fully yet.

Users who want to save network, indexing, and storage costs at scale can
set include_file_fingerprint to false. A still-growing fingerprint is
never published.

After further benchmarks we can re-consider if we want to make the
breaking change and flip the default to false.

Reclassify the changelog fragment from breaking-change to enhancement.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works. Where relevant, I have used the stresstest.sh script to run them under stress conditions and race detector to verify their stability.
• I have added an entry in ./changelog/fragments using the changelog tool.

Disruptive User Impact

None. This PR removes a disruption rather than introducing one.

How to test this PR locally

Integration test

Asserts log.file.fingerprint is present by default, present with include_file_fingerprint: true, and absent with false:

go test -v -tags integration -count=1 -run TestFilestreamIncludeFileFingerprint ./filebeat/tests/integration/

A still-growing fingerprint is never published (regardless of the flag)

A file smaller than the fingerprint length is tracked via a growing fingerprint whose SHA-256 is not yet computed, so log.file.fingerprint must never be added to its events even when include_file_fingerprint: true. Reproduced end to end with the real binary and real config:

cd filebeat && go build -o /tmp/fb .
mkdir -p /tmp/fp/{home,out,in-growing,in-complete}
# growing: 75 bytes, stays below the 1024-byte fingerprint length
printf 'growing line 1\ngrowing line 2\ngrowing line 3\ngrowing line 4\ngrowing line 5\n' > /tmp/fp/in-growing/small.log
# complete: > 1024 bytes, so the fingerprint completes
for i in $(seq 1 60); do printf 'complete log line number %02d with some padding text to add bytes\n' "$i"; done > /tmp/fp/in-complete/big.log

Config at /tmp/fp/fb.yml (vary paths between the two input dirs and include_file_fingerprint between true/false across the four runs):

path.home: /tmp/fp/home
filebeat.inputs:
  - type: filestream
    id: fp-repro
    paths: [ /tmp/fp/in-growing/*.log ]   # or /tmp/fp/in-complete/*.log
    file_identity.fingerprint:
      growing: true
    prospector.scanner.fingerprint.enabled: true
    include_file_fingerprint: true        # or false
output.file: { path: /tmp/fp/out, filename: output, rotate_on_startup: false }
queue.mem.flush.timeout: 0

Run each combination and count events carrying the field:

rm -rf /tmp/fp/home /tmp/fp/out; mkdir -p /tmp/fp/home /tmp/fp/out
timeout 15s /tmp/fb -e -c /tmp/fp/fb.yml --strict.perms=false
echo "with fingerprint: $(jq -c 'select(.log.file.fingerprint != null)' /tmp/fp/out/* | wc -l) / $(wc -l < /tmp/fp/out/*)"

Observed:

file	include_file_fingerprint	events	with log.file.fingerprint
growing (75 B)	true	5	0
growing (75 B)	false	5	0
complete (3840 B)	true	60	60
complete (3840 B)	false	60	0

The growing file never carries log.file.fingerprint under either flag, while the complete file carries it only when the flag is true. Sample log.file for the growing input with include_file_fingerprint: true (no fingerprint key):

{"device_id":"36","inode":"1512763","path":"/tmp/fp/in-growing/small.log"}

Sample log.file for the complete input with include_file_fingerprint: true (fingerprint present):

{"device_id":"36","fingerprint":"bab6a068cc46b9059ae1fa1b654cbc4dff7b4adca3866854d4b2b31058074e79","inode":"1512764","path":"/tmp/fp/in-complete/big.log"}

Related issues

• Relates [filebeat] Filestream: fingerprint in events opt-in via include_file_fingerprint #50129
• Relates [Filebeat] Filestream: make fingerprint on event configurable #50724
• Relates [Filebeat] Filestream: make file identity metadata configurable #50115
• Relates [Filebeat] Filestream: Remove the limitation of not ingesting files smaller than 1kB #44780

---

This is an automatic backport of pull request #51793 done by [Mergify](https://mergify.com).

[github-actions]

🤖 GitHub comments

Just comment with:

• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)
• /test : Run the Buildkite pipeline.

[infra-vault-gh-plugin-prod]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[github-actions]

🔍 Preview links for changed docs

• docs/reference/filebeat/filebeat-input-filestream.md

[github-actions]

✅ Elastic Docs Style Checker (Vale)

No issues found on modified lines!

---

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.