Manual rollback after grace period

[pchila]

What does this PR do?

This PR builds upon #8767 by allowing to manually rollback an elastic agent even after the grace period ends.
This PR uses a registry of elastic-agent available rollbacks introduced with PR #10344.

Having a registry of agent versions available for rollback allows for detection of possible manual rollback targets. The same available rollbacks are written in .update-marker file so that the watcher will preserve them at the end of the grace period, still allowing for a rollback after the watcher exits.

Installs available for rollbacks are assigned a TTL, so they can be cleaned up after a given time (governed by the agent.upgrade.rollback.window setting).

With this PR, cleanup and normalization of agent installs happens only at startup.
In a follow-up PR:

• the agent will be able to schedule a cleanup to run without needing a restart
• old agent installs available for rollback will be cleaned up when initiating a new upgrade operation (in order to save disk space)

Why is it important?

This PR allows to manually rollback an elastic-agent upgrade within a configurable window that extends beyond the grace period (the period during which an automatic rollback may be triggered by the upgraded agent misbehaving).

Checklist

• I have read and understood the pull request guidelines of this project.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [ ] I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• [ ] I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

Disruptive User Impact

No impact for users as the feature is deactivated when using the default value of agent.upgrade.rollback.window: 0

How to test this PR locally

1. Package elastic agent twice from this PR:

SNAPSHOT=true EXTERNAL=true PACKAGES=tar.gz  PLATFORMS="linux/amd64" mage -v package

AGENT_PACKAGE_VERSION="9.3.0+build20251022000000" BEAT_VERSION="9.3.0-SNAPSHOT" EXTERNAL=true PACKAGES=tar.gz  PLATFORMS="linux/amd64" mage -v package

2. Install the version 9.3.0-SNAPSHOT as usual
3. Set a rollback window duration > 0, for example 10 minutes and a shorter grace period and check_interval for watcher:

agent.upgrade:
  watcher:
      grace_period: 1m
      error_check.interval: 5s
  rollback:
      window: 10m

4. Trigger an update to the other package (saved on disk):

elastic-agent upgrade --skip-verify --source-uri=file:///vagrant/build/distributions 9.3.0+build20251022000000

5. Wait for the new agent to come online and check the upgrade details for UPG_WATCHING state. After the grace period of 1 minute, upgrade details should disappear and the watcher should exit.

6. Verify that the data/elastic-agent-9.3.0-SNAPSHOT-<hash> directory is still present after watcher exited.

7. Manually rollback to the previous version:

elastic-agent upgrade --rollback 9.3.0-SNAPSHOT

8. Check the output of elastic-agent status and verify that the agent restarted with version 9.2.0-SNAPSHOT and that the directory data/elastic-agent-9.3.0+build20251022000000-<hash> contains only logs.

Related issues

• Requires Introduce install TTL markers #10344

Questions to ask yourself

• How are we going to support this in production?
• How are we going to measure its adoption?
• How are we going to debug this?
• What are the metrics I should take care of?
• ...

[mergify]

⚠️ The sha of the head commit of this PR conflicts with #8767. Mergify cannot evaluate rules on this PR. ⚠️

[mergify]

This pull request does not have a backport label. Could you fix it @pchila? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-./d./d is the label that automatically backports to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b rollback-watcher-cleanup upstream/rollback-watcher-cleanup
git merge upstream/main
git push upstream rollback-watcher-cleanup

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b rollback-watcher-cleanup upstream/rollback-watcher-cleanup
git merge upstream/main
git push upstream rollback-watcher-cleanup

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b rollback-watcher-cleanup upstream/rollback-watcher-cleanup
git merge upstream/main
git push upstream rollback-watcher-cleanup

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b rollback-watcher-cleanup upstream/rollback-watcher-cleanup
git merge upstream/main
git push upstream rollback-watcher-cleanup

[elastic-sonarqube]

Quality Gate failed

Failed conditions
0.0% Coverage on New Code (required ≥ 40%)

See analysis details on SonarQube

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b rollback-watcher-cleanup upstream/rollback-watcher-cleanup
git merge upstream/main
git push upstream rollback-watcher-cleanup

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[pchila]

buildkite test this

[cmacknz]

Code looks good to me, I went through the manual test steps and it worked with the following observations:

• I don't see rollbacks available reported from the status command, just the watching state:

❯ sudo elastic-development-agent status
┌─ fleet
│  └─ status: (STOPPED) Not enrolled into Fleet
├─ elastic-agent
│  └─ status: (HEALTHY) Running
└─ upgrade_details
   ├─ target_version: 9.3.0+build20251022000000
   ├─ state: UPG_WATCHING
   └─ metadata

• The agent status appears that it will indefinitely contain the rolled back status, this is fine since that's what happened. Someone may ask us to change this later I think it will permanently highlight the agent in the Fleet UI with the rolled back status which is less useful if it was manual and not automatic.

❯ sudo elastic-development-agent status
┌─ fleet
│  └─ status: (STOPPED) Not enrolled into Fleet
├─ elastic-agent
│  └─ status: (HEALTHY) Running
└─ upgrade_details
   ├─ target_version: 9.3.0+build20251022000000
   ├─ state: UPG_ROLLBACK
   └─ metadata
      └─ reason: manual rollback requested to version 9.3.0-SNAPSHOT

[cmacknz]

@ycombinator still has open comments to resolve, @ycombinator can you confirm these are addressed and approve? This LGTM but I want to avoid approving when there are open comments from someone else remaining.

[ycombinator]

@pchila Thanks for addressing my comments. I've resolved most of them, just a couple where I'd like to see a bit more in the inline comments explaining the "why" + one other question about some code. After that, this LGTM!

[elasticmachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: f1568f6

Failed CI Steps

• fips:x86_64:sudo-false:fleet

History

• 💔 Build #29651 failed 52ab103
• 💚 Build #29535 succeeded dcf0ede
• 💛 Build #29101 was flaky db96ca9
• 💛 Build #28919 was flaky 6f4272b
• 💛 Build #28851 was flaky 53cdd8a

cc @pchila

[ycombinator]

LGTM!