Skip to content

x-pack/filebeat/input/awss3: exclude backup-prefixed keys from polling#51912

Open
efd6 wants to merge 1 commit into
elastic:mainfrom
efd6:51852-awss3
Open

x-pack/filebeat/input/awss3: exclude backup-prefixed keys from polling#51912
efd6 wants to merge 1 commit into
elastic:mainfrom
efd6:51852-awss3

Conversation

@efd6

@efd6 efd6 commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Proposed commit message

x-pack/filebeat/input/awss3: exclude backup-prefixed keys from polling

When same-bucket backup is configured and bucket_list_prefix is empty
(the default), the S3 listing returns backup objects alongside source
objects. Because each backup has a new key, ETag, and LastModified, the
state registry never recognises it as already processed. This causes
infinite reprocessing: each poll cycle processes its own prior backup
output and creates a deeper copy.

Exclude keys that start with the configured backup prefix from the
listing results when the backup destination is the same bucket and the
prefix falls within the listing scope. This applies to both the v1
poller (s3_input.go) and v2 poller (v2_polling.go).

Fixes #51852

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works. Where relevant, I have used the stresstest.sh script to run them under stress conditions and race detector to verify their stability.
  • I have added an entry in ./changelog/fragments using the changelog tool.

Disruptive User Impact

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

When same-bucket backup is configured and bucket_list_prefix is empty
(the default), the S3 listing returns backup objects alongside source
objects. Because each backup has a new key, ETag, and LastModified, the
state registry never recognises it as already processed. This causes
infinite reprocessing: each poll cycle processes its own prior backup
output and creates a deeper copy.

Exclude keys that start with the configured backup prefix from the
listing results when the backup destination is the same bucket and the
prefix falls within the listing scope. This applies to both the v1
poller (s3_input.go) and v2 poller (v2_polling.go).

Fixes elastic#51852
@efd6 efd6 self-assigned this Jul 14, 2026
@efd6 efd6 added Filebeat Filebeat Team:obs-ds-hosted-services Label for the Observability Hosted Services team bugfix Team:Security-Service Integrations Security Service Integrations Team backport-8.19 Automated backport to the 8.19 branch backport-9.3 Automated backport to the 9.3 branch backport-9.4 backport-9.5 labels Jul 14, 2026
@botelastic botelastic Bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jul 14, 2026
@github-actions

Copy link
Copy Markdown
Contributor

🤖 GitHub comments

Just comment with:

  • run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)
  • /test : Run the Buildkite pipeline.

@efd6 efd6 marked this pull request as ready for review July 14, 2026 02:29
@efd6 efd6 requested a review from a team as a code owner July 14, 2026 02:29
@infra-vault-gh-plugin-prod

Copy link
Copy Markdown

Pinging @elastic/obs-ds-hosted-services (Team:obs-ds-hosted-services)

@infra-vault-gh-plugin-prod

Copy link
Copy Markdown

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport-8.19 Automated backport to the 8.19 branch backport-9.3 Automated backport to the 9.3 branch backport-9.4 backport-9.5 bugfix Filebeat Filebeat Team:obs-ds-hosted-services Label for the Observability Hosted Services team Team:Security-Service Integrations Security Service Integrations Team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[bug-hunter] AWS S3 same-bucket backups are recursively reprocessed

1 participant