[Integration Test] Ensure that upgrading a FIPS-capable Agent results in a FIPS-capable Agent #7804

ycombinator · 2025-04-09T21:04:41Z

⚠️ Currently blocked on https://github.com/elastic/ingest-dev/issues/5264 and #8035 for CI being able to actually run these tests on a FIPS-configured VM. ⚠️

What does this PR do?

This PR allows a FIPS-capable Agent to upgrade to another FIPS-capable Agent.

It also adds three integration tests:

TestStandaloneUpgradeFIPStoFIPS: to check that a standalone FIPS-capable Agent will upgrade only to another FIPS-capable Agent. This test tests upgrades from and to various FIPS-capable Agent versions.
TestFleetManagedUpgradeUnprivilegedFIPS: to check that a Fleet-managed FIPS-capable unprivileged Agent will upgrade only to another FIPS-capable Agent.
TestFleetManagedUpgradePrivilegedFIPS: to check that a Fleet-managed FIPS-capable privileged Agent will upgrade only to another FIPS-capable Agent.

Why is it important?

To preserve FIPS-compliance across upgrades.

Checklist

I have read and understood the pull request guidelines of this project.
My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
~~I have made corresponding changes to the documentation~~
~~I have made corresponding change to the default configuration files~~
~~I have added tests that prove my fix is effective or that my feature works~~
~~I have added an entry in ./changelog/fragments using the changelog tool~~
I have added an integration test or an E2E test

Disruptive User Impact

None; this PR adds an integration test.

mergify · 2025-04-09T21:05:17Z

This pull request does not have a backport label. Could you fix it @ycombinator? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-./d./d is the label that automatically backports to the 8./d branch. /d is the digit
backport-active-all is the label that automatically backports to all active branches.
backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

elasticmachine · 2025-04-09T21:05:20Z

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

pkg/testing/fetcher_artifact.go

michel-laterman

testing looks good; should we also test a scenario where a failure is expected?

testing/integration/upgrade_standalone_fips_test.go

ycombinator · 2025-04-25T14:38:44Z

It turns out we currently have no way of upgrade a FIPS-capable Agent to another version of a FIPS-capable Agent. I just tried this manually and it doesn't work:

$ sudo elastic-agent version
Binary: 9.0.0-SNAPSHOT (build: 4da779d1fac682d7189ab14f7e058e84d82c653a fips-distribution: true at 2025-04-25 12:38:25 +0000 UTC)
Daemon: 9.0.0-SNAPSHOT (build: 4da779d1fac682d7189ab14f7e058e84d82c653a at 2025-04-25 12:38:25 +0000 UTC)

$ sudo elastic-agent status
┌─ fleet
│  └─ status: (STOPPED) Not enrolled into Fleet
└─ elastic-agent
   └─ status: (HEALTHY) Running

$ pwd
/home/shaunak

$ ls -al elastic-agent-fips-9.1.0-SNAPSHOT-linux-arm64.tar.gz*
-rw-r--r-- 1 shaunak shaunak 207491829 Apr 25 07:35 elastic-agent-fips-9.1.0-SNAPSHOT-linux-arm64.tar.gz
-rw-r--r-- 1 shaunak shaunak       182 Apr 25 07:35 elastic-agent-fips-9.1.0-SNAPSHOT-linux-arm64.tar.gz.sha512

$ sudo elastic-agent upgrade --source-uri file:///home/shaunak/ --skip-verify 9.1.0-SNAPSHOT
Error: Failed trigger upgrade of daemon: failed download of agent binary: unable to download package: package '/home/shaunak/elastic-agent-9.1.0-SNAPSHOT-linux-arm64.tar.gz' not found: open /home/shaunak/elastic-agent-9.1.0-SNAPSHOT-linux-arm64.tar.gz: no such file or directory
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/9.0/fleet-troubleshooting.html

Moving this PR into draft while I implement the missing capability.

ycombinator · 2025-04-29T02:18:55Z

The TestStandaloneUpgradeFIPStoNonFIPS/Upgrade_8.19.0-SNAPSHOT_to_9.0.0-SNAPSHOT_(privileged) and TestStandaloneUpgradeFIPStoNonFIPS/Upgrade_8.19.0-SNAPSHOT_to_9.0.0-SNAPSHOT_(unprivileged) FIPS upgrade integration tests are failing CI. The failures look like this:

upgrade_standalone_test.go:78:
--
  | Error Trace:	/opt/buildkite-agent/builds/bk-agent-prod-gcp-1745886251557149484/elastic/elastic-agent/testing/integration/upgrade_standalone_test.go:78
  | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1745886251557149484/elastic/elastic-agent/testing/integration/upgrade_standalone_test.go:58
  | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1745886251557149484/elastic/elastic-agent/testing/integration/upgrade_standalone_fips_test.go:146
  | Error:      	Received unexpected error:
  | could not unmarshal agent version output: error: exit status 2, output: panic: opensslcrypto: FIPS mode requested (requirefips tag set) but not available in OpenSSL 3.0.13 30 Jan 2024
  |  
  | goroutine 1 [running]:
  | crypto/internal/backend.init.1()
  | crypto/internal/backend/openssl_linux.go:40 +0x9f
  |  
  | yaml: mapping values are not allowed in this context
  | Test:       	TestStandaloneUpgradeFIPStoNonFIPS/Upgrade_8.19.0-SNAPSHOT_to_9.0.0-SNAPSHOT_(privileged)

@michel-laterman @simitt I think we need https://github.com/elastic/ingest-dev/issues/5264 for these tests?

simitt · 2025-04-29T10:00:17Z

@michel-laterman @simitt I think we need https://github.com/elastic/ingest-dev/issues/5264 for these tests?

Agree

elastic-sonarqube · 2025-05-08T22:13:20Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
77.8% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

elasticmachine · 2025-05-08T22:21:24Z

💔 Build Failed

Buildkite Build
Commit: 142ec35

Failed CI Steps

History

💔 Build #21064 failed 0f2c12a
💔 Build #21061 failed 2a0c837
💔 Build #21047 failed 7b43bbd
💔 Build #21030 failed 8bc5730

cc @ycombinator

ycombinator · 2025-05-09T00:25:18Z

The TestFleetManagedUpgradeUnprivilegedFIPS integration test is failing in CI like so:

=== FAIL: testing/integration TestFleetManagedUpgradeUnprivilegedFIPS (0.17s)
--
  | upgrade_fleet_test.go:157:
  | Error Trace:	/opt/buildkite-agent/builds/bk-agent-prod-gcp-1746743140257113053/elastic/elastic-agent/testing/integration/upgrade_fleet_test.go:157
  | /opt/buildkite-agent/builds/bk-agent-prod-gcp-1746743140257113053/elastic/elastic-agent/testing/integration/upgrade_fleet_test.go:103
  | Error:      	Received unexpected error:
  | failed to find build at /opt/buildkite-agent/builds/bk-agent-prod-gcp-1746743140257113053/elastic/elastic-agent/build/distributions: stat /opt/buildkite-agent/builds/bk-agent-prod-gcp-1746743140257113053/elastic/elastic-agent/build/distributions/elastic-agent-fips-9.1.0-SNAPSHOT-linux-x86_64.tar.gz: no such file or directory
  | Test:       	TestFleetManagedUpgradeUnprivilegedFIPS

The TestFleetManagedUpgradePrivilegedFIPS integration test also fails with the same error.

If I look at the build artifacts from the "Packaging: Ubuntu x86_64 FIPS" CI step, I see that build/distributions/elastic-agent-fips-9.1.0-SNAPSHOT-linux-x86_64.tar.gz artifact is being built.

But I think this build artifact is on a different Buildkite Agent from the ones that are running the TestFleetManagedUpgradeUnprivilegedFIPS and TestFleetManagedUpgradePrivilegedFIPS tests, as seen in the log output of the packaging step:

--
~/builds/bk-agent-prod-gcp-1746738804551933315/elastic/elastic-agent

@michel-laterman is this gap something the BK pipeline changes in #8035 will address?

cmacknz · 2025-05-23T19:07:55Z

The tests are still failing trying to find the FIPS artifacts, but just going off the description I think we are over testing this between all three of these cases:

TestStandaloneUpgradeFIPStoFIPS: to check that a standalone FIPS-capable Agent will upgrade only to another FIPS-capable Agent. This test tests upgrades from and to various FIPS-capable Agent versions.

TestFleetManagedUpgradeUnprivilegedFIPS: to check that a Fleet-managed FIPS-capable unprivileged Agent will upgrade only to another FIPS-capable Agent.

TestFleetManagedUpgradePrivilegedFIPS: to check that a Fleet-managed FIPS-capable privileged Agent will upgrade only to another FIPS-capable Agent.

What coverage are we gaining by testing this in privileged and unprivileged mode? Is there even any difference between privileged and unprivileged with respect to FIPS?

What are we gaining by testing Fleet and Standalone? I would expect if we do a single Fleet managed upgrade test we would have effectively smoke tested all of this.

All FIPS does is substitute out the implementation of crypto package and change crypto related code. Why do we need to test three variants of an upgrade because of this?

cmacknz · 2025-05-23T19:19:10Z

pkg/version/version_parser_test.go

@@ -583,3 +583,164 @@ func TestLess(t *testing.T) {
 		})
 	}
 }
+
+func TestEqual(t *testing.T) {


Instead of duplicating all the test cases from TestLess, maybe easier to just test both things in the same test? Maybe with a different sub-test?

I think maintaining the test cases is most of the work in here so we are better off duplicating them less.

ycombinator requested a review from a team as a code owner April 9, 2025 21:04

ycombinator requested review from michalpristas and michel-laterman April 9, 2025 21:04

mergify bot assigned ycombinator Apr 9, 2025

ycombinator added Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team skip-changelog Testing backport-8.x Automated backport to the 8.x branch with mergify backport-9.0 Automated backport to the 9.0 branch labels Apr 9, 2025

ycombinator commented Apr 9, 2025

View reviewed changes

pkg/testing/fetcher_artifact.go Outdated Show resolved Hide resolved

ycombinator removed the backport-9.0 Automated backport to the 9.0 branch label Apr 10, 2025

ycombinator force-pushed the it-upgrade-fips-to-fips branch from d71c0a5 to 1bf616f Compare April 11, 2025 19:54

michel-laterman previously approved these changes Apr 17, 2025

View reviewed changes

testing/integration/upgrade_standalone_fips_test.go Outdated Show resolved Hide resolved

ycombinator dismissed michel-laterman’s stale review via 917b25c April 17, 2025 23:58

ycombinator force-pushed the it-upgrade-fips-to-fips branch from 917b25c to 2ced379 Compare April 18, 2025 00:03

ycombinator added backport-8.19 Automated backport to the 8.19 branch and removed backport-8.x Automated backport to the 8.x branch with mergify labels Apr 22, 2025

ycombinator enabled auto-merge (squash) April 22, 2025 23:26

ycombinator requested a review from michel-laterman April 22, 2025 23:28

ycombinator changed the title ~~[Integration Test] Ensure that upgrading a FIPS-enabled Agent results in a FIPS-enabled Agent~~ [Integration Test] Ensure that upgrading a FIPS-capable Agent results in a FIPS-capable Agent Apr 23, 2025

ycombinator force-pushed the it-upgrade-fips-to-fips branch from 2ced379 to da48fd6 Compare April 24, 2025 00:01

ycombinator marked this pull request as draft April 25, 2025 14:39

auto-merge was automatically disabled April 25, 2025 14:39
Pull request was converted to draft

ycombinator marked this pull request as ready for review April 28, 2025 16:14

ycombinator requested a review from kaanyalti April 28, 2025 16:14

ycombinator added 20 commits May 8, 2025 14:10

Simplify upgradeOpts initialization

fc0200e

Add version equality comparison method

e973530

Fix version checks in tests

857e425

Refactor version check into own helper function

5f43cbc

Fixing args

c6e9138

No need to pass testing.T

a668cc9

Remove redundant test case

772f025

Restrict FIPS integration tests to Linux

8262637

Add Fleet-managed Agent FIPS upgrade test

28be696

Remove integration test trying to upgrade FIPS to non-FIPS

cea583a

Fix type of argument

4dc8306

Refactoring: extract common logic into helper function

af39588

Remove old code

b78720f

Require no error

eb8da3a

Fixing syntax errors

7d2161b

Define tests as needing a FIPS environment

f1162f9

FIPS upgrade tests should only run on Linux

34091e8

FIPS upgrade tests should start with FIPS-capable version

0ec4dfb

Fixing comment + skip message

5a3fb65

Fix syntax errors

142ec35

ycombinator force-pushed the it-upgrade-fips-to-fips branch from 0f2c12a to 142ec35 Compare May 8, 2025 21:10

ycombinator marked this pull request as ready for review May 20, 2025 23:56

ycombinator requested review from cmacknz, michel-laterman and kaanyalti May 20, 2025 23:56

cmacknz reviewed May 23, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Integration Test] Ensure that upgrading a FIPS-capable Agent results in a FIPS-capable Agent #7804

[Integration Test] Ensure that upgrading a FIPS-capable Agent results in a FIPS-capable Agent #7804

Uh oh!

ycombinator commented Apr 9, 2025 •

edited

Loading

Uh oh!

mergify bot commented Apr 9, 2025

Uh oh!

elasticmachine commented Apr 9, 2025

Uh oh!

Uh oh!

michel-laterman left a comment

Uh oh!

Uh oh!

ycombinator commented Apr 25, 2025

Uh oh!

ycombinator commented Apr 29, 2025

Uh oh!

simitt commented Apr 29, 2025

Uh oh!

elastic-sonarqube bot commented May 8, 2025

Uh oh!

elasticmachine commented May 8, 2025 •

edited

Loading

Uh oh!

ycombinator commented May 9, 2025 •

edited

Loading

Uh oh!

cmacknz commented May 23, 2025 •

edited

Loading

Uh oh!

cmacknz May 23, 2025

Uh oh!

Uh oh!

[Integration Test] Ensure that upgrading a FIPS-capable Agent results in a FIPS-capable Agent #7804

Are you sure you want to change the base?

[Integration Test] Ensure that upgrading a FIPS-capable Agent results in a FIPS-capable Agent #7804

Uh oh!

Conversation

ycombinator commented Apr 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Why is it important?

Checklist

Disruptive User Impact

Uh oh!

mergify bot commented Apr 9, 2025

Uh oh!

elasticmachine commented Apr 9, 2025

Uh oh!

Uh oh!

michel-laterman left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ycombinator commented Apr 25, 2025

Uh oh!

ycombinator commented Apr 29, 2025

Uh oh!

simitt commented Apr 29, 2025

Uh oh!

elastic-sonarqube bot commented May 8, 2025

Quality Gate passed

Uh oh!

elasticmachine commented May 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💔 Build Failed

Failed CI Steps

History

Uh oh!

ycombinator commented May 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cmacknz commented May 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cmacknz May 23, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ycombinator commented Apr 9, 2025 •

edited

Loading

elasticmachine commented May 8, 2025 •

edited

Loading

ycombinator commented May 9, 2025 •

edited

Loading

cmacknz commented May 23, 2025 •

edited

Loading