[9.3](backport #51345) Use native Go for Linux FIPS builds (#51953)

* Use native Go for Linux FIPS builds (#51345)

* feat: use native Go for Linux FIPS builds

Switch Linux FIPS builds from Microsoft Go (GOEXPERIMENT=systemcrypto)
to the upstream Go native FIPS module (GOFIPS140=v1.0.0, CMVP cert #5247).
Drops CGO requirement and the Microsoft Go cross-build Docker image.
Updates package tests to verify GOFIPS140 and DefaultGODEBUG buildinfo
instead of GOEXPERIMENT and OpenSSL ELF symbols.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* chore: clean up MS Go FIPS artifacts

- Remove GOEXPERIMENT=systemcrypto from CI pipelines and ECH test script
- Switch FIPS unit/ECH test steps from AWS FIPS image to standard GCP image; unskip
- Update VerifyFIPSBinary to check GOFIPS140+DefaultGODEBUG instead of GOEXPERIMENT+OpenSSL symbols
- Strengthen GOFIPS140 assertion to require v1.0.0 version; improve error messages
- Remove dead CGO field and OR from FIPSConfig/DefaultBuildArgs

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* chore: golangci-lint fixes

Signed-off-by: Dominik Rosiek <dominik.rosiek@elastic.co>

* fix: propagate FIPS env to unit tests and pre-populate module cache for crossbuild

- Inject FIPS compile env vars (GOFIPS140=v1.0.0) into go test args so
  that test binaries actually enforce FIPS crypto at runtime, not just
  compile with the requirefips tag.
- Pre-populate golang.org/fips140 in the host module cache before FIPS
  crossbuild containers mount it read-only, avoiding a write failure
  when the toolchain tries to unpack the module inside the container.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: update TestConnectionTLS for native Go FIPS

Add verification_mode:strict so Go's native TLS stack performs chain
verification (InsecureSkipVerify=false) and the FIPS key-size check
triggers. Update the expected error string to match native Go's message.
Simplify the test comment to remove implementation details.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* mod: pin elastic-agent-libs to main for native Go FIPS validation

Picks up the unreleased FIPS 140-3 peer-cert key-type enforcement
(elastic/elastic-agent-libs#426), which TestConnectionTLS depends on.
Update the test's expected error string to match the actual message
that check produces, rather than Go's own (unreachable, since a
custom TLS callback is always installed) internal FIPS error.

This is a temporary pin for CI validation ahead of a tagged
elastic-agent-libs release containing the fix.

* notice: regenerate for elastic-agent-libs pin bump

* mod: bump elastic-agent-libs to released v0.45.0

Swaps the temporary pseudo-version pin (unreleased main commit) for
the now-tagged release containing the FIPS 140-3 peer-cert key-type
enforcement fix (elastic/elastic-agent-libs#426). Same commit
content (d72ff2012311), now backed by a real release.

* notice: regenerate for elastic-agent-libs v0.45.0

* fix: address reviewer findings on FIPS migration

- gotest.go: wire fipsTestEnv() into makeGoTestArgsForPackage too, so
  per-module integration test builds (FIPS=true mage goIntegTest on a
  single module) get GOFIPS140 set, not just the requirefips tag.
- crossbuild.go: drop the bogus "G122" gosec rule code from the nolint
  comment on os.Chown (gosec has no G122; nolint:gosec suppresses by
  linter name, so the code was misleading, not functional).
- connection_test.go: cover both elastic-agent-libs FIPS callback paths
  (verification_mode strict and none use different callbacks) and
  assert the RSA-1024-specific error message instead of the more
  generic FIPS-rejection substring.

* test: require GOFIPS140 value to start with v1.0.0, not just contain it

Contains would tolerate v1.0.0 appearing anywhere in the buildinfo
value; HasPrefix asserts it's actually the version identifier itself.

* test: describe TestConnectionTLS modes by behavior, not libs internals

The comment named elastic-agent-libs' unexported checkAllChainsFIPS
and fipsVerifyNoneCallback, leaking implementation details of a
dependency with no compatibility guarantee. Describe the observable
strict-vs-none distinction instead.

---------

Signed-off-by: Dominik Rosiek <dominik.rosiek@elastic.co>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
(cherry picked from commit a829d56)

# Conflicts:
#	.buildkite/libbeat/pipeline.libbeat.yml
#	.buildkite/x-pack/pipeline.xpack.filebeat.yml
#	.buildkite/x-pack/pipeline.xpack.metricbeat.yml

* fix: resolve merge conflicts in backport of #51345

Mergify left unresolved conflict markers in three Buildkite pipeline
files. Resolve by keeping 9.3's image versions (1782878510) and
dropping IMAGE_UBUNTU_X86_64_FIPS, which no step references after the
switch to native Go FIPS (GOFIPS140=v1.0.0).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: suppress gosec G706 on dev-tool log line

DEV_OS/DEV_ARCH are trusted build-time env vars set by CI, not user
input. Add nolint:gosec to match the fix already present on main.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Dominik Rosiek <6997361+macdewee@users.noreply.github.com>
Co-authored-by: Dominik Rosiek <dominik.rosiek@elastic.co>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

Author: 3 people

.buildkite/libbeat/pipeline.libbeat.yml

@@ -16,7 +16,6 @@ env:
   IMAGE_WIN_2016: "platform-ingest-beats-windows-2016-1782878510"
   IMAGE_WIN_2019: "platform-ingest-beats-windows-2019-1782878510"
   IMAGE_WIN_2022: "platform-ingest-beats-windows-2022-1782878510"
-  IMAGE_UBUNTU_X86_64_FIPS: "platform-ingest-beats-ubuntu-2204-fips-1782878510"
 
   IMAGE_BEATS_WITH_HOOKS_LATEST: "docker.elastic.co/ci-agent-images/platform-ingest/buildkite-agent-beats-ci-with-hooks:latest"
 
@@ -97,7 +96,6 @@ steps:
               context: "libbeat: Ubuntu x86_64 Unit Tests"
 
       - label: ":ubuntu: Libbeat: Ubuntu x86_64 Go Unit Tests with fips provider and requirefips build tag"
-        skip: "https://github.com/elastic/beats/issues/50269: ssh keys not FIPS"
         key: "mandatory-linux-unit-test-fips-tag"
         command: |
           cd libbeat
@@ -106,13 +104,11 @@ steps:
           automatic:
             - limit: 1
         agents:
-          provider: "aws"
-          image: "${IMAGE_UBUNTU_X86_64_FIPS}"
-          instanceType: "m5.xlarge"
+          provider: "gcp"
+          image: "${IMAGE_UBUNTU_X86_64}"
+          machineType: "${GCP_DEFAULT_MACHINE_TYPE}"
         env:
           FIPS: "true"
-          GOEXPERIMENT: "systemcrypto"
-          ASDF_PYTHON_VERSION: "3.9.13"
         artifact_paths:
           - "libbeat/build/*.xml"
           - "libbeat/build/*.json"

.buildkite/scripts/custom_fips_ech_test.sh

@@ -16,5 +16,5 @@ ech_up $STACK_VERSION
 echo "~~~ Running custom FIPS ECH tests"
 
 pushd $BEAT_PATH
-GOEXPERIMENT=systemcrypto SNAPSHOT=true FIPS=true mage build fipsECHTest
+SNAPSHOT=true FIPS=true mage build fipsECHTest
 popd

.buildkite/x-pack/pipeline.xpack.filebeat.yml

@@ -10,7 +10,6 @@ env:
   GCP_WIN_MACHINE_TYPE: "n2-standard-8"
 
   IMAGE_UBUNTU_X86_64: "platform-ingest-beats-ubuntu-2204-1782878510"
-  IMAGE_UBUNTU_X86_64_FIPS: "platform-ingest-beats-ubuntu-2204-fips-1782878510"
   IMAGE_WIN_10: "platform-ingest-beats-windows-10-pro"
   IMAGE_WIN_11: "platform-ingest-beats-windows-11-pro"
   IMAGE_WIN_10_VERSION: "1.0.1782878510"
@@ -221,10 +220,8 @@ steps:
               context: "x-pack/filebeat: Go fips140=only Integration Tests"
 
       - label: ":ubuntu: x-pack/filebeat: FIPS ECH Integration Tests"
-        skip: "https://github.com/elastic/beats/issues/50269: ssh keys not FIPS"
         env:
           ASDF_TERRAFORM_VERSION: "1.9.3"
-          ASDF_PYTHON_VERSION: "3.9.13" # Not needed by ECH tests, but needed by VM
 # We are temporarily using the Production CFT environment instead of the Staging GovCloud
 # one.  This is being done until issues with creating deployments in Staging GovCloud are
 # fixed. Once this happens, uncomment the lines below and delete the gcp-us-west2 line.
@@ -238,9 +235,9 @@ steps:
           automatic:
             - limit: 1
         agents:
-          provider: "aws"
-          image: "${IMAGE_UBUNTU_X86_64_FIPS}"
-          instanceType: "m5.xlarge"
+          provider: "gcp"
+          image: "${IMAGE_UBUNTU_X86_64}"
+          machineType: "${GCP_DEFAULT_MACHINE_TYPE}"
         artifact_paths:
           - "x-pack/filebeat/build/*.xml"
           - "x-pack/filebeat/build/*.json"

.buildkite/x-pack/pipeline.xpack.metricbeat.yml

@@ -10,7 +10,6 @@ env:
   GCP_WIN_MACHINE_TYPE: "n2-standard-8"
 
   IMAGE_UBUNTU_X86_64: "platform-ingest-beats-ubuntu-2204-1782878510"
-  IMAGE_UBUNTU_X86_64_FIPS: "platform-ingest-beats-ubuntu-2204-fips-1782878510"
   IMAGE_WIN_10: "platform-ingest-beats-windows-10-pro"
   IMAGE_WIN_11: "platform-ingest-beats-windows-11-pro"
   IMAGE_WIN_10_VERSION: "1.0.1782878510"
@@ -260,10 +259,8 @@ steps:
               context: "x-pack/metricbeat: Go fips140=only Integration Tests (Module)"
 
       - label: ":ubuntu: x-pack/metricbeat: FIPS ECH Integration Tests"
-        skip: "https://github.com/elastic/beats/issues/50269: ssh keys not FIPS"
         env:
           ASDF_TERRAFORM_VERSION: "1.9.3"
-          ASDF_PYTHON_VERSION: "3.9.13" # Not needed by ECH tests, but needed by VM
 # We are temporarily using the Production CFT environment instead of the Staging GovCloud
 # one.  This is being done until issues with creating deployments in Staging GovCloud are
 # fixed. Once this happens, uncomment the lines below and delete the gcp-us-west2 line.
@@ -277,9 +274,9 @@ steps:
           automatic:
             - limit: 1
         agents:
-          provider: "aws"
-          image: "${IMAGE_UBUNTU_X86_64_FIPS}"
-          instanceType: "m5.xlarge"
+          provider: "gcp"
+          image: "${IMAGE_UBUNTU_X86_64}"
+          machineType: "${GCP_DEFAULT_MACHINE_TYPE}"
         artifact_paths:
           - "x-pack/metricbeat/build/*.xml"
           - "x-pack/metricbeat/build/*.json"

NOTICE.txt

@@ -10732,11 +10732,11 @@ SOFTWARE
 
 --------------------------------------------------------------------------------
 Dependency : github.com/elastic/elastic-agent-libs
-Version: v0.44.0
+Version: v0.45.0
 Licence type (autodetected): Apache-2.0
 --------------------------------------------------------------------------------
 
-Contents of probable licence file $GOMODCACHE/github.com/elastic/elastic-agent-libs@v0.44.0/LICENSE:
+Contents of probable licence file $GOMODCACHE/github.com/elastic/elastic-agent-libs@v0.45.0/LICENSE:
 
                                  Apache License
                            Version 2.0, January 2004

changelog/fragments/1781782084-use-native-go-for-linux-fips-builds.yaml

@@ -0,0 +1,45 @@
+# REQUIRED
+# Kind can be one of:
+# - breaking-change: a change to previously-documented behavior
+# - deprecation: functionality that is being removed in a later release
+# - bug-fix: fixes a problem in a previous version
+# - enhancement: extends functionality but does not break or fix existing behavior
+# - feature: new functionality
+# - known-issue: problems that we are aware of in a given version
+# - security: impacts on the security of a product or a user’s deployment.
+# - upgrade: important information for someone upgrading from a prior version
+# - other: does not fit into any of the other categories
+kind: enhancement
+
+# REQUIRED for all kinds
+# Change summary; a 80ish characters long description of the change.
+summary: Use native Go for Linux FIPS builds
+
+# REQUIRED for breaking-change, deprecation, known-issue
+# Long description; in case the summary is not enough to describe the change
+# this field accommodate a description without length limits.
+# description:
+
+# REQUIRED for breaking-change, deprecation, known-issue
+# impact:
+
+# REQUIRED for breaking-change, deprecation, known-issue
+# action:
+
+# REQUIRED for all kinds
+# Affected component; usually one of "elastic-agent", "fleet-server", "filebeat", "metricbeat", "auditbeat", "all", etc.
+component: all
+
+# AUTOMATED
+# OPTIONAL to manually add other PR URLs
+# PR URL: A link the PR that added the changeset.
+# If not present is automatically filled by the tooling finding the PR where this changelog fragment has been added.
+# NOTE: the tooling supports backports, so it's able to fill the original PR number instead of the backport PR number.
+# Please provide it if you are adding a fragment for a different PR.
+# pr: https://github.com/owner/repo/1234
+
+# AUTOMATED
+# OPTIONAL to manually add other issue URLs
+# Issue URL; optional; the GitHub issue related to this changeset (either closes or is part of).
+# If not present is automatically filled by the tooling with the issue linked to the PR number.
+# issue: https://github.com/owner/repo/1234

dev-tools/mage/build.go

@@ -29,6 +29,8 @@ import (
 
 	"github.com/josephspurrier/goversioninfo"
 	"github.com/magefile/mage/sh"
+	"golang.org/x/text/cases"
+	"golang.org/x/text/language"
 )
 
 // BuildArgs are the arguments used for the "build" target and they define how
@@ -113,7 +115,6 @@ func DefaultBuildArgs() BuildArgs {
 		for _, tag := range FIPSConfig.Compile.Tags {
 			args.ExtraFlags = append(args.ExtraFlags, "-tags="+tag)
 		}
-		args.CGO = args.CGO || FIPSConfig.Compile.CGO
 		for varName, value := range FIPSConfig.Compile.Env {
 			args.Env[varName] = value
 		}
@@ -168,8 +169,8 @@ func DefaultGolangCrossBuildArgs() BuildArgs {
 // environment.
 func GolangCrossBuild(params BuildArgs) error {
 	if os.Getenv("GOLANG_CROSSBUILD") != "1" {
-		return errors.New("Use the crossBuild target. golangCrossBuild can " +
-			"only be executed within the golang-crossbuild docker environment.")
+		return errors.New("use the crossBuild target; golangCrossBuild can " +
+			"only be executed within the golang-crossbuild docker environment")
 	}
 
 	defer DockerChown(filepath.Join(params.OutputDir, params.Name+binaryExtension(GOOS)))
@@ -292,7 +293,7 @@ func MakeWindowsSysoFile() (string, error) {
 		},
 		StringFileInfo: goversioninfo.StringFileInfo{
 			CompanyName:      BeatVendor,
-			ProductName:      strings.Title(BeatName),
+			ProductName:      cases.Title(language.English).String(BeatName),
 			ProductVersion:   version,
 			FileVersion:      version,
 			FileDescription:  BeatDescription,

dev-tools/mage/crossbuild.go

@@ -193,6 +193,15 @@ func CrossBuild(options ...CrossBuildOption) error {
 		// Make sure the module dependencies are downloaded on the host,
 		// as they will be mounted into the container read-only.
 		mg.Deps(func() error { return gotool.Mod.Download() })
+		if FIPSBuild {
+			// GOFIPS140=v1.0.0 unpacks golang.org/fips140 from GOROOT/lib/fips140
+			// into the module cache on first use. Pre-populate it on the host (as
+			// the host user) before the container mounts the cache read-only.
+			// Any go command triggers fips140.Init(), so list -m is sufficient.
+			mg.Deps(func() error {
+				return sh.RunWith(FIPSConfig.Compile.Env, "go", "list", "-m")
+			})
+		}
 	}
 
 	// Build the magefile for Linux, so we can run it inside the container.
@@ -278,9 +287,6 @@ func CrossBuildImage(platform string) (string, error) {
 	if err != nil {
 		return "", err
 	}
-	if FIPSBuild {
-		tagSuffix += "-fips"
-	}
 
 	return BeatsCrossBuildImage + ":" + goVersion + "-" + tagSuffix, nil
 }
@@ -584,7 +590,7 @@ func chownPaths(uid, gid int, path string) error {
 			return nil
 		}
 
-		if err := os.Chown(name, uid, gid); err != nil {
+		if err := os.Chown(name, uid, gid); err != nil { //nolint:gosec // paths are controlled build artifacts inside a Docker container, not user input
 			return fmt.Errorf("failed to chown path=%v: %w", name, err)
 		}
 		numFixed++

dev-tools/mage/fips-settings.yaml

@@ -4,10 +4,11 @@ beats:
   - filebeat
   - metricbeat
 compile:
-  cgo: true
   env:
-    GOEXPERIMENT: systemcrypto
-    MS_GOTOOLCHAIN_TELEMETRY_ENABLED: "0"
+    # See https://go.dev/doc/security/fips140#go-cryptographic-module-v100
+    # CMVP Certificate #5247: https://csrc.nist.gov/projects/cryptographic-module-validation-program/certificate/5247
+    # CAVP Certificate A6650: https://csrc.nist.gov/projects/cryptographic-algorithm-validation-program/details?product=19371
+    GOFIPS140: v1.0.0
   tags:
     - requirefips
   platforms:

dev-tools/mage/gotest.go

@@ -69,7 +69,7 @@ func makeGoTestArgs(name string) GoTestArgs {
 		TestName:        name,
 		Race:            RaceDetector,
 		Packages:        []string{"./..."},
-		Env:             make(map[string]string),
+		Env:             fipsTestEnv(),
 		OutputFile:      fileName + ".out",
 		JUnitReportFile: fileName + ".xml",
 		Tags:            testTagsFromEnv(),
@@ -89,6 +89,7 @@ func makeGoTestArgsForPackage(name, pkg string) GoTestArgs {
 		TestName:        fmt.Sprintf("%s-%s", name, pkg),
 		Race:            RaceDetector,
 		Packages:        []string{fmt.Sprintf("./module/%s", pkg)},
+		Env:             fipsTestEnv(),
 		OutputFile:      fileName + ".out",
 		JUnitReportFile: fileName + ".xml",
 		Tags:            testTagsFromEnv(),
@@ -141,6 +142,18 @@ func testTagsFromEnv() []string {
 	return tags
 }
 
+// fipsTestEnv returns the environment variables required to compile and run FIPS tests.
+func fipsTestEnv() map[string]string {
+	env := make(map[string]string, len(FIPSConfig.Compile.Env))
+	if !FIPSBuild {
+		return env
+	}
+	for k, v := range FIPSConfig.Compile.Env {
+		env[k] = v
+	}
+	return env
+}
+
 // DefaultGoTestUnitArgs returns a default set of arguments for running
 // all unit tests. We tag unit test files with '!integration'.
 func DefaultGoTestUnitArgs() GoTestArgs { return makeGoTestArgs("Unit") }
@@ -395,6 +408,7 @@ func GoTest(ctx context.Context, params GoTestArgs) error {
 		if raceAmd64 || raceArm64 {
 			testArgs = append(testArgs, "-race")
 		} else {
+			//nolint:gosec // G706: DEV_OS/DEV_ARCH are trusted build-time env vars, not untrusted input
 			log.Printf("Warning: skipping -race flag for unsupported platform %s/%s\n", devOS, devArch)
 		}
 	}