[8.19](backport #51897) make statestore creation of meta.json atomic

[mergify]

Proposed commit message

WHAT: writeMetaFile in libbeat/statestore/backend/memlog previously opened the destination meta.json with O_TRUNC, wrote the JSON body, synced, and closed — all in-place on the final path. The new implementation follows the same temp-file-then-rename pattern introduced for Filebeat's registrar in #51791: os.CreateTemp creates a sibling .tmp-* file, the JSON is written and synced to it, and agentfile.SafeFileRotate atomically renames it into the final position. A deferred cleanup always removes the temp file on any error path.

WHY: With the old write pattern, if the process was killed between the O_TRUNC (which immediately empties the file) and the final Close, meta.json would be left empty or partially written. On the next start, readMetaFile would fail to parse it and the store would refuse to open. Because the rename is atomic at the OS level, the new approach guarantees that meta.json is either the previous valid version or the newly completed one — never an empty or partial file.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works. Where relevant, I have used the stresstest.sh script to run them under stress conditions and race detector to verify their stability.
• I have added an entry in ./changelog/fragments using the changelog tool.

Disruptive User Impact

None. The change is internal to the memlog backend store and the observable behavior (a valid meta.json on disk) is identical.

How to test this PR locally

go test ./libbeat/statestore/backend/memlog/... -run TestWriteMetaFileAtomic -v

Related issues

• Relastes make creation of meta.json file atomic #51791

Use cases

Screenshots

Logs

---

This is an automatic backport of pull request #51897 done by [Mergify](https://mergify.com).

[github-actions]

🤖 GitHub comments

Just comment with:

• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)
• /test : Run the Buildkite pipeline.

[infra-vault-gh-plugin-prod]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[github-actions]

TL;DR

The Buildkite failure is isolated to the x-pack/metricbeat CockroachDB integration module, but the supplied log does not contain the underlying test or service error. The PR changes only libbeat/statestore/backend/memlog, so the available evidence does not support a regression caused by this PR; retry the CockroachDB job and inspect its test artifact if it reproduces.

Remediation

• Retry Go fips140=only Integration Tests (Module) to determine whether this is a transient CockroachDB/container startup failure.
• If it reproduces, retrieve TEST-go-integration-cockroachdb.out.json (and the corresponding XML/log output) and fix the reported service or assertion failure.

Investigation details

Root Cause

The failure is classified as inconclusive infrastructure/test-specific: the job reports Error: failed modules: cockroachdb, while the provided log contains no CockroachDB startup, connection, assertion, or panic details. The test itself calls compose.EnsureUp(t, "cockroachdb") at x-pack/metricbeat/module/cockroachdb/status/status_integration_test.go:32, so the missing detail is required to distinguish service startup from fetch/assertion failure.

The PR diff is limited to changelog/fragments/1783957443-statestore-atomic-create-meta.json.yaml, libbeat/statestore/backend/memlog/diskstore.go, libbeat/statestore/backend/memlog/store.go, and libbeat/statestore/backend/memlog/store_test.go; none touches the CockroachDB module or its integration harness.

Evidence

• Build: https://buildkite.com/elastic/beats/builds/49390
• Job/step: x-pack/metricbeat: Go fips140=only Integration Tests (Module)
• Key log excerpt: Error: failed modules: cockroachdb
• The same log records the Tomcat integration test as passed immediately before the aggregate failure.

Verification

• Not run locally: this is an integration test requiring the CockroachDB service, and the supplied failure data does not include the failing test details.

Follow-up

If the retry fails, attach the CockroachDB JSON/XML artifacts or the complete module output so the specific remediation can be identified.

---

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.