[9.4](backport #51897) make statestore creation of meta.json atomic

[mergify]

Proposed commit message

WHAT: writeMetaFile in libbeat/statestore/backend/memlog previously opened the destination meta.json with O_TRUNC, wrote the JSON body, synced, and closed — all in-place on the final path. The new implementation follows the same temp-file-then-rename pattern introduced for Filebeat's registrar in #51791: os.CreateTemp creates a sibling .tmp-* file, the JSON is written and synced to it, and agentfile.SafeFileRotate atomically renames it into the final position. A deferred cleanup always removes the temp file on any error path.

WHY: With the old write pattern, if the process was killed between the O_TRUNC (which immediately empties the file) and the final Close, meta.json would be left empty or partially written. On the next start, readMetaFile would fail to parse it and the store would refuse to open. Because the rename is atomic at the OS level, the new approach guarantees that meta.json is either the previous valid version or the newly completed one — never an empty or partial file.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works. Where relevant, I have used the stresstest.sh script to run them under stress conditions and race detector to verify their stability.
• I have added an entry in ./changelog/fragments using the changelog tool.

Disruptive User Impact

None. The change is internal to the memlog backend store and the observable behavior (a valid meta.json on disk) is identical.

How to test this PR locally

go test ./libbeat/statestore/backend/memlog/... -run TestWriteMetaFileAtomic -v

Related issues

• Relastes make creation of meta.json file atomic #51791

Use cases

Screenshots

Logs

---

This is an automatic backport of pull request #51897 done by [Mergify](https://mergify.com).

[github-actions]

🤖 GitHub comments

Just comment with:

• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)
• /test : Run the Buildkite pipeline.

[infra-vault-gh-plugin-prod]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[mergify]

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

• Queue this pull request