UnitChangeModified triggered on units that are not changed creating wierd statuses on Kibana

[bhapas]

Scenario

Scenario where there is an agent policy with an existing package policies with some input [ cel for ex., ]. The status of integrations looks like

But when a package is added / removed from the agent policy then the input status are in wierd state.

What's happening

When 1 out of 3 package policies is removed from the agent policy syncUnits seems to be called and a UnitChangeModified is triggered on existing units [ Although nothing has changed ].

When it reaches beats listener, an updateUnit on the unit concerned is triggered leading to a status of Configuring and Agent health shown Healthy.

Although there are no changes on the units, since a UnitChanged is triggered beat input recalculates the status of all the input units and streams associated and triggers another UpdateState on the clientUnit.

Not sure if this is a bug or intended with the current design between agent-client and beats.

cc: @cmacknz @andrewkroh

[bhapas]

@elastic/elastic-agent-control-plane May be you can take a look at this?

[bhapas]

Logged this in this repo since the origin of UnitChangeModified is seen in agent-client

[cmacknz]

Taking a quick look my first guess is that in this block:

elastic-agent-client/pkg/client/unit.go

Lines 306 to 309 in a59bedc

	if u.expectedState != exp {
	u.expectedState = exp
	triggers |= TriggeredStateChange
	}

The exp is actually the current or observed state of the unit, which would be degraded or failed on error. The checkin expected only ever gets set to healthy or stopped, so they'd be a difference incorrectly detected here whenever the current unit state wasn't one of those two.

I think this happens because we pull out the current state of the unit we are maintaining when passing it to this function

elastic-agent-client/pkg/client/client_v2.go

Line 675 in a59bedc

unit := c.findUnit(agentUnit.Id, UnitType(agentUnit.Type))