x-pack/filebeat/input/awss3: Add lexicographical polling mode

[kcreddy]

Proposed commit message

x-pack/filebeat/input/awss3: Add lexicographical polling mode

Adds lexicographical ordering mode for S3 bucket polling. 
The input uses the S3 "StartAfter" parameter to resume listing.
For buckets with a large number of objects where object keys 
are naturally ordered (ex: timestamp-prefixed), this significantly 
reduces the number of objects listed [1] and memory footprint.

The persisted tail key tracks in-flight objects for crash resistance,
ensuring no objects are skipped after an unexpected restart.

Adds 2 new configuration options "lexicographical_ordering" and 
"lexicographical_lookback_keys":
- lexicographical_ordering: When set to "true", enables 
  lexicographical ordering mode. Default is "false".
- lexicographical_lookback_keys: Maintains a bounded lookback 
  buffer of the N most recently processed keys (default: 100). The 
  start-after parameter uses the lexicographically oldest key in 
  the buffer, creating a sliding window that catches late-arriving 
  objects.

[1] https://github.com/elastic/beats/issues/47926#issuecomment-3713381444

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works. Where relevant, I have used the stresstest.sh script to run them under stress conditions and race detector to verify their stability.
• I have added an entry in ./changelog/fragments using the changelog tool.

Disruptive User Impact

How to test this PR locally

Related issues

• Closes [Filebeat]: aws-s3 input - Add Lexicographical Ordering Mode #47926

Use cases

[github-actions]

🤖 GitHub comments

Just comment with:

• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @kcreddy? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[github-actions]

🔍 Preview links for changed docs

• docs/reference/filebeat/filebeat-input-aws-s3.md

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[Kavindu-Dodan]

@kcreddy thanks again for the update and working on this.

I left few more remarks and we are almost there 🙌

[github-actions]

✅ Vale Linting Results

No issues found on modified lines!

---

The Vale linter checks documentation changes against the Elastic Docs style guide.

To use Vale locally or report issues, refer to Elastic style guide for Vale.

[Kavindu-Dodan]

Overall looks good, but please check the current lint failures to get a green build - https://github.com/elastic/beats/actions/runs/21706571563/job/62599113313?pr=48310#step:5:38

[efd6]

The remaining lint complaints are all "Error return value is not checked (errcheck)". This because of this. I do not believe that option should be true; there are numerous cases (for example to cases that are flagged here) where a direct panic from (runtime.panicdottypeE or runtime.panicnildottype) is the correct action for the program to take because of a failed type assertion. To work around the repo policy, I think it would be best to add a //nolint:errcheck // some reason to the relevant lines.

[Kavindu-Dodan]

Unfortunately, the actual runtime state registry expects a map or an object as state value. So we must fix this and validate if beats input work end to end.

[efd6]

nits/queries only

[efd6]

Thanks

[Kavindu-Dodan]

I have this concern which must be checked and added - #48310 (comment)

Otherwise, my local run went fine and I can clearly see the benefit of this feature. I am approving to unblock once above concern is fixed.