Skip to content

[9.4](backport #50772) Mount Git alternates in crossbuild containers#50954

Open
mergify[bot] wants to merge 1 commit into
9.4from
mergify/bp/9.4/pr-50772
Open

[9.4](backport #50772) Mount Git alternates in crossbuild containers#50954
mergify[bot] wants to merge 1 commit into
9.4from
mergify/bp/9.4/pr-50772

Conversation

@mergify
Copy link
Copy Markdown
Contributor

@mergify mergify Bot commented May 27, 2026

Proposed commit message

Mount Git alternate object directories into golang-crossbuild Docker containers.

Buildkite checks out Beats using reference clones:

Skipping update and using existing mirror for repository git@github.com:elastic/beats.git at /opt/git-mirrors/git-github-com-elastic-beats-git.
git clone -v --reference /opt/git-mirrors/git-github-com-elastic-beats-git -- git@github.com:elastic/beats.git .

That checkout shape leaves .git/objects/info/alternates pointing at object directories under /opt/git-mirrors. The checkout itself is valid on the Buildkite host, but the crossbuild path runs go build inside a golang-crossbuild Docker container. Before this change, the container only had the Beats checkout mounted, not the host-side mirror object directory.

Go VCS stamping (-buildvcs, enabled by default for module builds) shells out to Git while building. In a reference clone, those Git commands may need borrowed objects from the alternate object directory to resolve repository metadata. When the alternate directory exists on the host but is not visible in the container, the repository appears incomplete from inside Docker and the build can fail even though the checkout is healthy on the agent.

This also explains why the failures showed up across unrelated Filebeat Go integration test PRs. The affected builds were tied to the CI checkout/build environment, not to the PR code under test:

Build Step PR
32590 Filebeat: Go Integration Tests #50552
32486 Filebeat: Go Integration Tests #50680
32484 Filebeat: Go Integration Tests #50257
32473 Filebeat: Go Integration Tests #49762

The fix teaches GolangCrossBuilder to detect .git/objects/info/alternates and mount each existing alternate object directory read-only into the crossbuild container. Absolute alternates, such as Buildkite's /opt/git-mirrors/.../objects, are mounted at the same absolute path in the container, which matches the path recorded by Git. Relative alternates are resolved relative to the checkout's object directory and mapped into the corresponding container checkout path.

The behavior is intentionally narrow:

  • No alternates file means no extra Docker mounts.
  • Missing alternate directories are ignored, matching Git's tolerance for stale alternates while avoiding broken Docker volume arguments.
  • Alternate mounts are read-only.
  • Duplicate container paths are de-duplicated.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works. Where relevant, I have used the stresstest.sh script to run them under stress conditions and race detector to verify their stability.
  • I have added an entry in ./changelog/fragments using the changelog tool.

Disruptive User Impact

None expected. This only affects crossbuild Docker invocations from repositories that use Git alternates. The additional mounts are read-only and are only added when .git/objects/info/alternates exists.

How to test this PR locally

Run the focused unit tests for the new helper behavior:

go test ./dev-tools/mage -run 'Test(GitAlternateObjectDirMounts|ContainerPathForHostPath)$'

The end-to-end validation is to rerun a Buildkite Filebeat: Go Integration Tests job on an agent that checks out Beats with git clone --reference /opt/git-mirrors/.... In that environment, the crossbuild docker run command should include read-only volume mounts for the alternate object directories.

## Related issues

Use cases

Buildkite agents use reference clones to avoid re-fetching the full Beats repository for every job. This PR keeps that optimization while making crossbuild containers see the same Git object graph that the host checkout sees.

The same behavior also helps any local or CI environment using Git alternates with the Beats crossbuild path.

Screenshots

Not applicable.

Logs

The relevant checkout evidence from the failing Buildkite log:

Skipping update and using existing mirror for repository git@github.com:elastic/beats.git at /opt/git-mirrors/git-github-com-elastic-beats-git.
git clone -v --reference /opt/git-mirrors/git-github-com-elastic-beats-git -- git@github.com:elastic/beats.git .

The error:

>> Building using: cmd='build/mage-linux-amd64 golangCrossBuild', env=[CC=gcc, CXX=g++, GOARCH=amd64, GOARM=, GOOS=linux, GOTOOLCHAIN=local, PLATFORM_ID=linux-amd64]
error obtaining VCS status: exit status 128
	Use -buildvcs=false to disable VCS stamping.
Error: running "go build -o build/golang-crossbuild/filebeat-linux-amd64 -buildmode pie -trimpath -ldflags -s -X github.com/elastic/beats/v7/libbeat/version.buildTime=2026-05-18T10:04:57Z -X github.com/elastic/beats/v7/libbeat/version.commit=a35628e6e0a2946c25b976c8cee5dec2db7d3cd9" failed with exit code 1
Error: failed building for linux/amd64: exit status 1
failed building for linux/amd64: exit status 1
```<hr>This is an automatic backport of pull request #50772 done by [Mergify](https://mergify.com).

Buildkite reference clones can borrow objects from host-side mirrors that are not visible inside Docker. Mount those alternate object directories read-only so Go VCS stamping can resolve repository metadata during crossbuilds.

GenAI-Assisted: Yes
Human-Reviewed: Yes
Tool: Cursor, Model: GPT-5.5 Agent Mode
Assisted-By: Cursor
Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
(cherry picked from commit 20a6af1)
@mergify mergify Bot requested a review from a team as a code owner May 27, 2026 15:17
@mergify mergify Bot added the backport label May 27, 2026
@mergify mergify Bot requested review from andrzej-stencel and mauri870 and removed request for a team May 27, 2026 15:17
@botelastic botelastic Bot added the needs_team Indicates that the issue/PR needs a Team:* label label May 27, 2026
@github-actions
Copy link
Copy Markdown
Contributor

🤖 GitHub comments

Just comment with:

  • run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)
  • /test : Run the Buildkite pipeline.

@github-actions github-actions Bot added Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team skip-changelog labels May 27, 2026
@botelastic botelastic Bot removed the needs_team Indicates that the issue/PR needs a Team:* label label May 27, 2026
@infra-vault-gh-plugin-prod
Copy link
Copy Markdown

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

@github-actions
Copy link
Copy Markdown
Contributor

TL;DR

Both failed Buildkite jobs are failing before pre-commit run --all-files starts, and the available logs stop during git clone setup with no hook/test/lint output. This points to an infrastructure/log-capture failure, not a code regression in this PR.

Remediation

  • Re-run the two failed jobs (x-pack/osquerybeat: Run pre-commit and Packetbeat: Run pre-commit) to confirm whether this was transient infra interruption.
  • If they fail again, capture full logs from after clone/checkout (or Buildkite agent termination reason) so we can identify a concrete failing hook.
  • No code change is indicated from current evidence.
Investigation details

Root Cause

Current evidence is inconclusive for a code-level issue. Both job logs end at repository clone initialization and never show execution of pre-commit run --all-files or any pre-commit hook output.

Given both independent pipelines failed in the same early phase with Exit status: -1, this is most consistent with infra/job interruption (agent termination/log truncation/transient CI issue).

Evidence

  • Build: https://buildkite.com/elastic/beats/builds/46851
  • Jobs:
  • Logged excerpts (both logs):
    • /tmp/gh-aw/buildkite-logs/beats-xpack-osquerybeat-x-packosquerybeat-run-pre-commit.txt:46:
      $ git clone -v --reference /usr/local/git-references/git-github-com-elastic-beats-git -- git@github.com:elastic/beats.git .
      Cloning into '.'.
      POST git-upload-pack (175 bytes)
      
    • /tmp/gh-aw/buildkite-logs/beats-packetbeat-packetbeat-run-pre-commit.txt:46:
      $ git clone -v --reference /usr/local/git-references/git-github-com-elastic-beats-git -- git@github.com:elastic/beats.git .
      Cloning into '.'.
      POST git-upload-pack (175 bytes)
      
  • Additional observation: each provided log file contains only 49 lines, with no downstream step output.

Verification

  • Not run locally (failure is in CI job bootstrap phase; no actionable failing hook/test command is present in the available logs).

Follow-up

If a rerun fails with complete logs, we can classify precisely (specific hook/config/dependency) and propose a targeted patch.


What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant