The elastic-agent upgrade CLI command needs to require an uninstall token when Elastic Defend is installed

[cmacknz]

• Relates Add integration tests for upgrades that include endpoint security #4720
• Relates Allow DEB and RPM packages to supply an uninstall token when upgrading with Elastic Defend installed #6394
• Relates Elastic-Agents unexpectedly unenrolled after update to 8.16.x #6213
  • Elastic-Agents unexpectedly unenrolled after update to 8.16.x #6213 (comment)

For Fleet managed agents that are running Elastic Defend, upgrading from the CLI does not send a signed upgrade action to Defend which is what allows Defend to upgrade without believing it is being tampered with. The tamper protection feature of Defend is based on all operations requiring a digital signature signed by a private key in Kibana. There is no way to provide a signed token for an upgrade on the CLI, so Defend legitimately believed it was being tampered with and orphaned from agent here.

Specifically using the CLI skips this block

elastic-agent/internal/pkg/agent/application/actions/handlers/handler_action_upgrade.go

Lines 58 to 74 in 54932dc

	if h.tamperProtectionFn() {
	// Find inputs that want to receive UPGRADE action
	// Endpoint needs to receive a signed UPGRADE action in order to be able to uncontain itself
	state := h.coord.State()
	ucs := findMatchingUnitsByActionType(state, a.Type())
	if len(ucs) > 0 {
	h.log.Debugf("handlerUpgrade: proxy/dispatch action '%+v'", a)
	err := notifyUnitsOfProxiedAction(ctx, h.log, action, ucs, h.coord.PerformAction)
	h.log.Debugf("handlerUpgrade: after action dispatched '%+v', err: %v", a, err)
	if err != nil {
	return err
	}
	} else {
	// Log and continue
	h.log.Debugf("No components running for %v action type", a.Type())
	}
	}

To make the upgrade command work with tamper protection properly we'd have to allow it to accept an uninstall token to allow endpoint to unprotect itself for a CLI upgrade.

We could simply block CLI upgrades with endpoint, but we continue to preserve use of the upgrade CLI command as an escape hatch for when fleet upgrades are unavailable or broken for an unforeseen reason, and without making this change this escape hatch does not work properly for agents that run defend.

What is the definition of done?

• A test exists proving that the elastic-agent upgrade command can be used to upgrade Elastic Defend is installed without disabling tamper protection.
• The upgrade CLI command refuses to upgrade when defend is install and a tamper protection uninstall token is not provided.
• The documentation for the CLI is updated to indicate that an uninstall token is required in this situation.

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[cmacknz]

We likely also need to support the uninstall token for the RPM and DEB packages when endpoint is installed.

[cmacknz]

Created #6394 to track the RPM and DEB change, which is more critical as Fleet cannot be used to upgrade those package types.