[8.19](backport #51701) [x-pack/filebeat/input/azureblobstorage] - Harden blob listing against transient failures and implement configurable retries (#51787)

* [x-pack/filebeat/input/azureblobstorage] -  Harden blob listing against transient failures and implement configurable retries (#51701)

x-pack/filebeat/input/azureblobstorage: harden blob listing against transient failures

Transient Azure Storage failures (HTTP 503 ServerBusy, HTTP 429
throttling, or brief network timeouts) during blob listing (pagination)
could exhaust the small, non-configurable default retries and fatally
stop the input with no recovery, leaving long ingestion gaps.

This change:

- Exposes a retry configuration block (max_retries, initial_retry_delay,
  max_retry_delay) wired into the Azure SDK pipeline retry policy. Because
  the policy lives in the client request pipeline, it covers blob listing
  as well as downloads, and the download stream reader's retry count is
  also driven by max_retries. Defaults mirror the SDK's own policy (3
  retries, 800ms base delay, 60s ceiling), so configs that omit the block
  behave exactly as before.
- Makes a transient listing failure non-fatal while polling: the input is
  marked Degraded and the listing is retried on the next poll interval, so
  it rides out longer outages and recovers on its own. No blobs are
  skipped, because each poll re-lists from scratch and the checkpoint only
  advances for processed blobs. Permanent failures (a missing container or
  an authentication error) still stop the input, and one-shot (non-polling)
  runs are unchanged.
- Reports Running again after a successful listing pass that schedules no
  jobs, so a poll that recovers but has no new work (no new blobs, or
  everything filtered out or already past the checkpoint) no longer leaves
  the input stuck in Degraded.
- Migrates the package tests to localized logptest loggers (dropping the
  deprecated logp.TestingSetup) and documents the stack versions the retry
  attribute is available in.

Assisted-By: Cursor (Claude Opus 4.8)
(cherry picked from commit 0f86bf1)

# Conflicts:
#	docs/reference/filebeat/filebeat-input-azure-blob-storage.md
#	x-pack/filebeat/input/azureblobstorage/config.go
#	x-pack/filebeat/input/azureblobstorage/input.go
#	x-pack/filebeat/input/azureblobstorage/input_stateless.go
#	x-pack/filebeat/input/azureblobstorage/types.go

* x-pack/filebeat/input/azureblobstorage: clean up botched retry backport

The 8.19 backport of #51701 (configurable retries) was committed with
unresolved merge-conflict markers in config.go, input.go,
input_stateless.go, types.go and the input reference docs, so the
package did not compile.

Resolving the conflict had also pulled in the unrelated path_prefix
feature, which exists only on main and was never backported to 8.19: a
PathPrefix config field, its Source plumbing, and a documented
path_prefix option. With the supporting filtering logic absent on 8.19
this was dead config.

Remove the conflict markers, drop the stray path_prefix additions from
both the code and the docs, and keep only the retry changes that #51701
actually introduced. Also drop the v9-only {applies_to} version
directives and the batch_size entry from the docs, which the 8.19
reference does not use.

Assisted-By: Cursor (Claude Opus 4.8)

* x-pack/filebeat/input/azureblobstorage: document retry in the asciidoc

On 8.19.x the input reference documentation is published from the
Asciidoc sources under x-pack/filebeat/docs/inputs, not the Markdown
files under docs/reference, which are irrelevant on this branch. The
previous cleanup had left the new retry documentation in the Markdown
file.

Add the retry attribute to input-azure-blob-storage.asciidoc (a
supported-attributes list entry and a new retry section describing
max_retries, initial_retry_delay and max_retry_delay), and revert the
Markdown file to its base state so the branch does not touch it.

Assisted-By: Cursor (Claude Opus 4.8)

* Update x-pack/filebeat/docs/inputs/input-azure-blob-storage.asciidoc

Co-authored-by: Visha Angelova <91186315+vishaangelova@users.noreply.github.com>

---------

Co-authored-by: Shourie Ganguly <shourie.ganguly@elastic.co>
Co-authored-by: Visha Angelova <91186315+vishaangelova@users.noreply.github.com>

Author: 3 people

changelog/fragments/1783054266-azureblobstorage-configurable-retries.yaml

@@ -0,0 +1,41 @@
+# Kind can be one of:
+# - breaking-change: a change to previously-documented behavior
+# - deprecation: functionality that is being removed in a later release
+# - bug-fix: fixes a problem in a previous version
+# - enhancement: extends functionality but does not break or fix existing behavior
+# - feature: new functionality
+# - known-issue: problems that we are aware of in a given version
+# - security: impacts on the security of a product or a user’s deployment.
+# - upgrade: important information for someone upgrading from a prior version
+# - other: does not fit into any of the other categories
+kind: enhancement
+
+# Change summary; a 80ish characters long description of the change.
+summary: Add a configurable retry policy to the Azure Blob Storage input.
+
+# Long description; in case the summary is not enough to describe the change
+# this field accommodate a description without length limits.
+description: >
+  The `azure-blob-storage` input now exposes a `retry` configuration block
+  (`max_retries`, `initial_retry_delay` and `max_retry_delay`) that tunes the
+  Azure SDK retry policy. Because the policy lives in the client request
+  pipeline, it now covers blob listing (pagination) in addition to downloads;
+  previously a transient error such as an HTTP 503 `ServerBusy` during listing
+  could exhaust the small, non-configurable default retries and stop the input.
+  The settings apply per storage account. Defaults are unchanged when the block
+  is omitted. Additionally, when polling is enabled, a transient blob-listing
+  failure that outlives the retries (503, 429 or a network timeout) is now
+  non-fatal: the input is marked degraded and retries on the next poll interval
+  instead of exiting, so longer outages are ridden out. Permanent failures such
+  as a missing container still stop the input.
+
+# Affected component; usually one of "elastic-agent", "fleet-server", "filebeat", "metricbeat", "auditbeat", "all", etc.
+component: "filebeat"
+
+# PR URL; optional; the PR number that added the changeset.
+# If not present is automatically filled by the tooling finding the PR where this changelog fragment has been added.
+# pr: https://github.com/elastic/beats/pull/0000
+
+# Issue URL; optional; the GitHub issue related to this changeset (either closes or is part of).
+# If not present is automatically filled by the tooling with the issue linked to the PR number.
+issue: https://github.com/elastic/beats/issues/44629

x-pack/filebeat/docs/inputs/input-azure-blob-storage.asciidoc

@@ -140,6 +140,7 @@ Now let's explore the configuration attributes a bit more elaborately.
    11. <<attrib-file_selectors,file_selectors>>
    12. <<attrib-expand_event_list_from_field,expand_event_list_from_field>>
    13. <<attrib-timestamp_epoch,timestamp_epoch>>
+   14. <<attrib-retry,retry>>
 
 
 [id="attrib-account-name"]
@@ -393,6 +394,40 @@ filebeat.inputs:
 
 The Azure Blob Storage APIs don't provide a direct way to filter files based on timestamp, so the input will download all the files and then filter them based on the timestamp. This can cause a bottleneck in processing if the number of files are very high. It is recommended to use this attribute only when the number of files are limited or ample resources are available.
 
+[id="attrib-retry"]
+[float]
+==== `retry`
+
+This attribute controls how transient Azure Storage failures are retried. Azure occasionally returns throttling responses such as `HTTP 429` or `HTTP 503` (`ServerBusy`), and brief network problems can also interrupt requests. The retry settings feed the Azure SDK's retry policy, which sits in the client request pipeline, so they apply to *every* request the input makes — both for listing blobs (pagination) and for downloading them.
+
+The `retry` attribute contains the following sub-attributes:
+
+* `max_retries`: The number of times a failed request is retried after the first attempt. A negative value disables retries. Defaults to `3`.
+* `initial_retry_delay`: The base delay used for the exponential backoff between attempts. Each subsequent retry waits longer, up to `max_retry_delay`. Defaults to `800ms`.
+* `max_retry_delay`: The upper bound on the backoff, so that during a long outage the input keeps retrying at a steady interval instead of drifting towards ever larger waits. Defaults to `60s`.
+
+Every sub-attribute is optional. Omitting the `retry` attribute (or any of its sub-attributes) keeps the Azure SDK defaults, so existing configurations continue to behave as before. The settings are applied per storage account and, therefore, affect all configured containers.
+
+When polling is enabled, a transient blob-listing failure that outlives these retries (for example, an HTTP 503 `ServerBusy`, an HTTP 429, or a network timeout) no longer stops the input. Instead, the input is marked `degraded` and the listing is retried on the next <<attrib-poll_interval,poll_interval>>. This allows the input to ride out longer outages rather than exiting. Permanent failures, such as a missing container or an authentication error, still stop the input.
+
+["source","yaml",subs="attributes"]
+----
+filebeat.inputs:
+- type: azure-blob-storage
+  id: my-azureblobstorage-id
+  enabled: true
+  account_name: some_account
+  auth.shared_credentials.account_key: some_key
+  retry:
+    max_retries: 20
+    initial_retry_delay: 1s
+    max_retry_delay: 30s
+  containers:
+  - name: container_1
+----
+
+This example lets the input ride out longer bursts of Azure throttling: it retries a failed request up to `20` times, starting with a `1s` delay and backing off exponentially up to a `30s` ceiling between attempts.
+
 [id="container-overrides"]
 *The sample configs below will explain the container level overriding of attributes a bit further :-*