fix(otelconsumer): do not retry 401 errors from Elasticsearch

[andrzej-stencel]

Proposed commit message

Prevents retrying of 401 Unauthorized errors from Elasticsearch in otelconsumer.

This is a (hopefully temporary) workaround for the fact that the Elasticsearch exporter does not mark 401 errors from Elasticsearch as permanent, which it probably should.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [ ] I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works. Where relevant, I have used the stresstest.sh script to run them under stress conditions and race detector to verify their stability.
• I have added an entry in ./changelog/fragments using the changelog tool.
  • I think this Beat receivers change does not need a changelog entry?

Disruptive User Impact

This change the behavior of Beat receivers to not retry 401 errors from Elasticsearch. I cannot imagine a scenario where someone would depend on this behavior, so this change should be non-consequential.

How to test this PR locally

• Build Elastic Agent with this Beats change.
• Run the Agent with config similar to below, with Filebeat receiver sending to Elasticsearch exporter that has invalid credentials to ES.
• Expect just one error for each failed batch instead of a flood of errors as described in the linked issue.

service:
  pipelines:
    logs:
      receivers:
        - filebeatreceiver
      exporters:
        - elasticsearch/no-retries
receivers:
  filebeatreceiver:
    filebeat:
      inputs:
        - type: filestream
          id: logs1
          paths:
            - /tmp/logs-1k.log
    path.data: /tmp/0422/data
    path.logs: /tmp/0422/logs
    queue.mem:
      flush.timeout: 0
exporters:
  elasticsearch/no-retries:
    endpoint: http://localhost:9200
    user: elastic
    password: wrong
    flush:
      interval: 1s
    retry:
      enabled: false
    sending_queue:
      enabled: false

Related issues

• Resolves otelconsumer tight-loops retries on 401 with zero backoff (9.3.2) #50217 by making otelconsumer not retry on 401 errors

[github-actions]

🤖 GitHub comments

Just comment with:

• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)
• /test : Run the Buildkite pipeline.

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @andrzej-stencel? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[botelastic]

This pull request doesn't have a Team:<team> label.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: c82cd0f4-66a5-4b8b-b08b-56e5ce20f6fc

📥 Commits

Reviewing files that changed from the base of the PR and between a008b37 and 02c0b0f.

📒 Files selected for processing (1)

• libbeat/otel/otelconsumer/otelconsumer_test.go

---

📝 Walkthrough

Walkthrough

The change updates otelconsumer.logsPublish to recognize errors exposing StatusCode() int (imports net/http) and treats StatusCode() == http.StatusUnauthorized (401) as a permanent error, causing the batch to be dropped. Other errors keep the existing retry vs permanent branching. Tests were added to assert 401 results in BatchDrop and non-401 results in BatchRetry.

🚥 Pre-merge checks | ✅ 2

✅ Passed checks (2 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	The PR directly addresses issue #50217 by treating 401 Unauthorized errors as permanent/non-retryable in otelconsumer, preventing zero-backoff retry storms.
Out of Scope Changes check	✅ Passed	All changes are scoped to the stated objective: adding HTTP status code error handling to otelconsumer.go and corresponding test coverage in otelconsumer_test.go.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

• 🛠️ Update Documentation: Commit on current branch
• 🛠️ Update Documentation: Create PR

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[github-actions]

TL;DR

The failures are not caused by the otelconsumer logic change itself; CI is failing in setup/tooling and in one pre-existing Windows leak test. Immediate actions: fix Mage bootstrap for FIPS/libbeat jobs and harden TestLeak cleanup on Windows.

Remediation

• Ensure Mage 1.15.0 is actually installed in the failing FIPS/libbeat jobs before invoking mage (or invoke Mage via go run github.com/magefile/mage@v1.15.0 ... in the failing steps to bypass asdf drift).
• For x-pack/auditbeat/abreceiver TestLeak, add Windows-safe shutdown/cleanup synchronization before tempdir cleanup (e.g., wait for file handle release on beat.db after rec.Shutdown).
• Re-run the failed jobs after the above; the current failures happen before/around test harness startup rather than in the PR’s otelconsumer behavior change.

Investigation details

Root Cause

1. Configuration/tooling failure (confirmed):

   • Libbeat: Ubuntu x86_64 Go Unit Tests with fips provider and requirefips build tag fails with:
     • No preset version installed for command mage
     • suggestion asdf install mage 1.15.0
     • then exit 126
   • Evidence in log: /tmp/gh-aw/buildkite-logs/beats-libbeat-ubuntu-libbeat-ubuntu-x86_64-go-unit-tests-with-fips-provide.txt lines containing those exact messages.
   • Relevant pipeline config pins Mage 1.15.0: .buildkite/libbeat/pipeline.libbeat.yml:24 (ASDF_MAGE_VERSION: 1.15.0).

2. FIPS ECH jobs (filebeat/metricbeat):

   • Both fail with exit status 126 and no produced test artifacts:
     • /tmp/gh-aw/buildkite-logs/beats-xpack-filebeat-ubuntu-x-packfilebeat-fips-ech-integration-tests.txt
     • /tmp/gh-aw/buildkite-logs/beats-xpack-metricbeat-ubuntu-x-packmetricbeat-fips-ech-integration-tests.txt
   • The invoked step is .buildkite/scripts/custom_fips_ech_test.sh, whose test execution command is mage build fipsECHTest at .buildkite/scripts/custom_fips_ech_test.sh:19.
   • Given the confirmed Mage bootstrap failure in libbeat and matching 126 symptom, these are consistent with the same CI toolchain bootstrap issue.

3. Windows test failure (test flake/resource leak):

   • x-pack/auditbeat: Win 2025 Unit Tests fails in TestLeak with:
     • TempDir RemoveAll cleanup: ...\data\beat.db: The process cannot access the file because it is being used by another process.
   • Evidence in log: /tmp/gh-aw/buildkite-logs/beats-xpack-auditbeat-windows-x-packauditbeat-win-2025-unit-tests.txt around the === FAIL: ... TestLeak section.
   • Test implementation: x-pack/auditbeat/abreceiver/receiver_leak_test.go (TestLeak at line 27, rec.Shutdown(...) at line 96). This indicates a Windows handle-release timing issue after shutdown.

Evidence

• Build: https://buildkite.com/elastic/beats/builds/44657
• Failed jobs:
  • https://buildkite.com/elastic/beats-libbeat/builds/28632
  • https://buildkite.com/elastic/beats-xpack-filebeat/builds/30520
  • https://buildkite.com/elastic/beats-xpack-metricbeat/builds/30970
  • https://buildkite.com/elastic/beats-xpack-auditbeat/builds/27474
• PR merge commit context from logs shows only:
  • libbeat/otel/otelconsumer/otelconsumer.go
  • libbeat/otel/otelconsumer/otelconsumer_test.go

Verification

• I did not run Windows integration locally in this environment; conclusions are based on Buildkite logs and repository source inspection.

Follow-up

• If Mage bootstrap is fixed and FIPS jobs still fail with 126, the next required data is the full pre-teardown output from custom_fips_ech_test.sh (current captured log is mostly teardown and omits the failing command output).

Note

🔒 Integrity filter blocked 2 items

The following items were blocked because they don't meet the GitHub integrity level.

• fix(otelconsumer): do not retry 401 errors from Elasticsearch #50261 pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #50261 pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

---

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

[cmacknz]

Force merging, test failures are unrelated.

[github-actions]

@Mergifyio backport 8.19 9.3 9.4

[mergify]

backport 8.19 9.3 9.4

✅ Backports have been created

Details

• #50271 [8.19](backport #50261) fix(otelconsumer): do not retry 401 errors from Elasticsearch has been created for branch 8.19 but encountered conflicts

Cherry-pick of fdea8cd has failed:

On branch mergify/bp/8.19/pr-50261
Your branch is up to date with 'origin/8.19'.

You are currently cherry-picking commit fdea8cd23.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   x-pack/libbeat/outputs/otelconsumer/otelconsumer_test.go

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   x-pack/libbeat/outputs/otelconsumer/otelconsumer.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

• #50272 [9.3](backport #50261) fix(otelconsumer): do not retry 401 errors from Elasticsearch has been created for branch 9.3 but encountered conflicts

Cherry-pick of fdea8cd has failed:

On branch mergify/bp/9.3/pr-50261
Your branch is up to date with 'origin/9.3'.

You are currently cherry-picking commit fdea8cd23.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   x-pack/libbeat/outputs/otelconsumer/otelconsumer_test.go

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   x-pack/libbeat/outputs/otelconsumer/otelconsumer.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

• #50273 [9.4](backport #50261) fix(otelconsumer): do not retry 401 errors from Elasticsearch has been created for branch 9.4 but encountered conflicts

Cherry-pick of fdea8cd has failed:

On branch mergify/bp/9.4/pr-50261
Your branch is up to date with 'origin/9.4'.

You are currently cherry-picking commit fdea8cd23.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   libbeat/otel/otelconsumer/otelconsumer_test.go

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   libbeat/otel/otelconsumer/otelconsumer.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

[pkoutsovasilis]

Hey @cmacknz @andrzej-stencel 👋

We've been investigating e2e test failures in ECK (cloud-on-k8s) that started occurring after this PR was backported to 9.4. I wanted to share some context about a scenario where treating 401 as not retryable from the start causes data loss.

Use-Case: Elasticsearch File-based Realm in Kubernetes

In Kubernetes deployments managed by ECK, Elasticsearch uses a file-based realm for authentication. When ECK creates credentials for an Elastic Agent, at a high level:

1. ECK creates a Kubernetes Secret containing the user credentials
2. This Secret is mounted into the ES pod and updates the file realm
3. Elasticsearch needs to detect and reload the updated file realm

There's an inherent delay between when the credentials are created and when Elasticsearch actually accepts them. During this window, any authentication attempt returns 401 - but this is a transient condition, not a permanent authentication failure.

Why This Matters

With this change, when Agent components start producing data before ES has reloaded the file realm:

• The 401 error is treated as permanent
• Data is immediately dropped via batch.Drop()
• Initial logs/metrics are permanently lost

This is particularly problematic because some components (like filebeat monitoring its own logs) only generate certain events at startup. Once those events are dropped, they're never regenerated.

IMO Expected Behavior

In general, workloads should be resilient to transient errors during startup. A 401 while credentials are still propagating through the system is fundamentally different from a 401 caused by genuinely invalid credentials.

Proposal

I understand the original issue (beats#50217) was about tight-loop retries with zero backoff causing CPU saturation. That's a real problem that needed fixing. However, completely removing retry capability for 401 swings too far in the other direction.

Is this commit part of the 9.4.0 upcoming release? More importantly is there any chance we can revert this PR in time? This change will cause a real issue for ECK users.

[cmacknz]

Is this commit part of the 9.4.0 upcoming release? More importantly is there any chance we can revert this PR in time? This change will cause a real issue for ECK users.

It is part of 9.4.0 and there is still time to revert it. This breaking something that used to work suggest this is an accidental breaking change.

Looking at the beat ES output logic here, it looks like 401s would not get retried at the exporter either. The collector in general wants 401s to be treated as permanent errors (though not yet the case for the ES exporter open-telemetry/opentelemetry-collector-contrib#47870).

beats/libbeat/outputs/elasticsearch/client.go

Lines 501 to 553 in bd70fc6

	encodedEvent := event.EncodedEvent.(*encodedEvent) //nolint:errcheck //safe to ignore type check
	if itemStatus < 300 {
	if encodedEvent.deadLetter {
	// This was ingested into the dead letter index, not the original target
	stats.deadLetter++
	} else {
	stats.acked++
	}
	return false // no retry needed
	}
	if itemStatus == 409 {
	// 409 is used to indicate there is already an event with the same ID, or
	// with identical Time Series Data Stream dimensions when TSDS is active.
	stats.duplicates++
	return false // no retry needed
	}
	if itemStatus == http.StatusTooManyRequests {
	stats.fails++
	stats.tooMany++
	return true
	}
	if itemStatus < 500 {
	// hard failure, apply policy action
	if encodedEvent.deadLetter {
	// Fatal error while sending an already-failed event to the dead letter
	// index, drop.
	client.pLogDeadLetter.Add()
	client.log.Errorw(fmt.Sprintf("Can't deliver to dead letter index event '%s' (status=%v): %s", encodedEvent, itemStatus, itemMessage), logp.TypeKey, logp.EventType)
	stats.nonIndexable++
	return false
	}
	if client.deadLetterIndex == "" {
	// Fatal error and no dead letter index, drop.
	client.pLogIndex.Add()
	client.log.Warnw(fmt.Sprintf("Cannot index event '%s' (status=%v): %s, dropping event!", encodedEvent, itemStatus, itemMessage), logp.TypeKey, logp.EventType)
	stats.nonIndexable++
	return false
	}
	// Send this failure to the dead letter index and "retry".
	// We count this as a "retryable failure", and then if the dead letter
	// ingestion succeeds it is counted in the "deadLetter" counter
	// rather than the "acked" counter.
	client.pLogIndexTryDeadLetter.Add()
	client.log.Warnw(fmt.Sprintf("Cannot index event '%s' (status=%v): %s, trying dead letter index", encodedEvent, itemStatus, itemMessage), logp.TypeKey, logp.EventType)
	encodedEvent.setDeadLetter(client.deadLetterIndex, itemStatus, string(itemMessage))
	}
	// Everything else gets retried.
	stats.fails++
	return true

I wonder if the Filebeat guaranteed send feature for logs is doing the retry here, and this is bypassing that? #47892. This would not apply to metrics though.

I would be in favour of reverting this to be safe while we work out how to handle this in a nicer way. We do not need to drop to avoid retrying rapidly, those options are seperate.

CC @andrzej-stencel

[pkoutsovasilis]

Thanks for the prompt reply @cmacknz! I totally get the logic behind marking a 401 as a permanent error - it just feels like a more subtle approach to treat it as permanent only after some retry period with backoff, rather than immediately on the first occurrence.

[cmacknz]

Revert PR #50341