Skip to content

handle PreInit error case for runReady.Close#51911

Merged
leehinman merged 2 commits into
elastic:mainfrom
leehinman:51893_exit_delay
Jul 14, 2026
Merged

handle PreInit error case for runReady.Close#51911
leehinman merged 2 commits into
elastic:mainfrom
leehinman:51893_exit_delay

Conversation

@leehinman

@leehinman leehinman commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Proposed commit message

WHAT

filebeat/beater: close runReady on Manager.PreInit failure

PR #51864 (commit 00068f7) reordered the defers in Filebeat.Run so that defer fb.runReady.Close() runs before the managerEarlyStop defer (correct LIFO ordering to avoid a 5-second wait when the manager's stop callback calls StopWithContext). However, both defers were placed after the b.Manager.PreInit() call. If PreInit returns an error, Run exits before either defer is registered, leaving runReady.ch permanently unclosed.

An unclosed runReady.ch means any subsequent call to StopWithContext (e.g. via a SIGINT-triggered Manager.Stop that invokes the beat stop callback) will wait the full 5-second timeout before proceeding, rather than returning immediately.

WHY

Restores the invariant from before #51864: runReady is closed regardless of how Run exits. The fix adds an explicit fb.runReady.Close() on the PreInit error return path, which is the one path the deferred close cannot reach. A new unit test TestRunClosesRunReadyOnPreInitFailure directly asserts the channel is closed after Run returns due to a PreInit error; confirmed to fail against the buggy code and pass with the fix.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works. Where relevant, I have used the stresstest.sh script to run them under stress conditions and race detector to verify their stability.
  • I have added an entry in ./changelog/fragments using the changelog tool.

Disruptive User Impact

None. This is an internal shutdown-sequencing fix. The only observable change is that Elastic Agent-managed Filebeat instances where PreInit fails will no longer incur a spurious 5-second delay during shutdown.

How to test this PR locally

go test ./filebeat/beater/ -run "TestRunClosesRunReadyOnPreInitFailure|TestStopWaitsForRunReady" -v -count=1 -race

Related issues

Use cases

Screenshots

Logs

@leehinman leehinman added Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team bugfix backport-active-all Automated backport with mergify to all the active branches skip-changelog labels Jul 13, 2026
@botelastic botelastic Bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jul 13, 2026
@github-actions

github-actions Bot commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

🤖 GitHub comments

Just comment with:

  • run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)
  • /test : Run the Buildkite pipeline.

@github-actions

github-actions Bot commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

TL;DR

The Buildkite failure is a formatting check failure, not a Filebeat runtime or test failure. filebeat/beater/stop_test.go was committed with imports that goimports -local github.com/elastic rewrites, so make -C filebeat check update left the file modified and check-no-changes exited 2.

Remediation

  • Run make -C filebeat check (or the repository formatter) and commit the resulting filebeat/beater/stop_test.go import-order change. In particular, keep the Elastic imports in the local-import group and sort github.com/elastic/elastic-agent-libs/logp before the github.com/elastic/beats/v7/... imports.
  • Re-run make -C filebeat check update followed by make check-no-changes.
Investigation details

Root Cause

The PR adds imports in filebeat/beater/stop_test.go:20-29, but the import block is not in the format enforced by the Beats Mage formatter. The repository config sets GoImportsLocalPrefix = "github.com/elastic" in dev-tools/mage/fmt.go:36-39; the PR places github.com/elastic/elastic-agent-libs/logp after the Beats imports, so the formatter changes the file.

Evidence

  • Build: https://buildkite.com/elastic/beats/builds/49338
  • Job/step: Filebeat: Run check/update
  • Key log excerpt: Error: some files are not up-to-date. Run makeupdate then review and commit the changes. Modified: [filebeat/beater/stop_test.go]
  • The failing command was make -C filebeat check update, followed by make check-no-changes; the log contains no compiler, test, or dependency error.

Verification

  • Not run locally; the supplied Buildkite log stopped at the source-format/check stage before tests.

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

@leehinman leehinman marked this pull request as ready for review July 13, 2026 22:24
@leehinman leehinman requested a review from a team as a code owner July 13, 2026 22:24
@infra-vault-gh-plugin-prod

infra-vault-gh-plugin-prod Bot commented Jul 13, 2026

Copy link
Copy Markdown

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

@mergify

mergify Bot commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

  • Queue this pull request

@leehinman leehinman merged commit 397822f into elastic:main Jul 14, 2026
48 checks passed
@github-actions

github-actions Bot commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

@Mergifyio backport 9.5 9.4 9.3 8.19

@mergify

mergify Bot commented Jul 14, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

backport 9.5 9.4 9.3 8.19

✅ Backports have been created

Details

Cherry-pick of 397822f has failed:

On branch mergify/bp/9.5/pr-51911
Your branch is up to date with 'origin/9.5'.

You are currently cherry-picking commit 397822f0b.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   filebeat/beater/filebeat.go

Unmerged paths:
  (use "git add/rm <file>..." as appropriate to mark resolution)
	deleted by us:   filebeat/beater/stop_test.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

Cherry-pick of 397822f has failed:

On branch mergify/bp/9.4/pr-51911
Your branch is up to date with 'origin/9.4'.

You are currently cherry-picking commit 397822f0b.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   filebeat/beater/filebeat.go

Unmerged paths:
  (use "git add/rm <file>..." as appropriate to mark resolution)
	deleted by us:   filebeat/beater/stop_test.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

Cherry-pick of 397822f has failed:

On branch mergify/bp/9.3/pr-51911
Your branch is up to date with 'origin/9.3'.

You are currently cherry-picking commit 397822f0b.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   filebeat/beater/filebeat.go

Unmerged paths:
  (use "git add/rm <file>..." as appropriate to mark resolution)
	deleted by us:   filebeat/beater/stop_test.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

Cherry-pick of 397822f has failed:

On branch mergify/bp/8.19/pr-51911
Your branch is up to date with 'origin/8.19'.

You are currently cherry-picking commit 397822f0b.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   filebeat/beater/filebeat.go

Unmerged paths:
  (use "git add/rm <file>..." as appropriate to mark resolution)
	deleted by us:   filebeat/beater/stop_test.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

This was referenced Jul 14, 2026
Merged
Open
Open
Open
leehinman added a commit that referenced this pull request Jul 15, 2026
@mergify @leehinman
[9.5](backport #51911) handle PreInit error case for runReady.Close (#…
5f2c89c 
…51929)

* handle PreInit error case for runReady.Close (#51911)

* handle PreInit error case for runReady.Close

* fix fmt issue

(cherry picked from commit 397822f)

# Conflicts:
#	filebeat/beater/stop_test.go

* fix the merge conflict

---------

Co-authored-by: Lee E Hinman <57081003+leehinman@users.noreply.github.com>
Co-authored-by: Lee E. Hinman <lee.e.hinman@elastic.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@khushijain21 khushijain21 khushijain21 approved these changes
@andrzej-stencel andrzej-stencel andrzej-stencel approved these changes

Assignees

@leehinman leehinman

Labels

backport-active-all Automated backport with mergify to all the active branches bugfix skip-changelog Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[breaking-change] Undocumented breaking change: Filebeat PreInit errors now incur a 5-second shutdown timeout

3 participants

@leehinman @khushijain21 @andrzej-stencel