[beats receivers] Monitoring receiver status persists after an unenroll action when they are not actually running

[cmacknz]

• Relates Switch to the otel runtime manager by default for self-monitoring #10204 (comment)

The reproduction is what the testInstallAndUnenrollWithEndpointSecurity does in the failing tests in the PR above:

1. Enroll agent in a policy with defend that uses beats receivers
2. Unenroll the agent via the Fleet UI
3. Wait for Defend to be removed
4. Observe the state of the beats receivers for monitoring continue to be reported
5. Also observe the collector sub-process is not running. This seems to be a bug where we are not clearing the state on unenroll. Unenroll through fleet sends agent and empty agent policy to execute to remove all inputs. We seem to stop the collector but not clear its status.

ubuntu@ubuntu24:~/elastic-agent-9.3.0-SNAPSHOT-linux-arm64$ systemctl status elastic-agent
● elastic-agent.service - Elastic Agent is a unified agent to observe, monitor and protect your system.
     Loaded: loaded (/etc/systemd/system/elastic-agent.service; enabled; preset: enabled)
     Active: active (running) since Wed 2025-10-15 16:15:04 EDT; 5min ago
   Main PID: 7497 (elastic-agent)
      Tasks: 12 (limit: 1059)
     Memory: 135.7M (peak: 637.1M)
        CPU: 5.586s
     CGroup: /system.slice/elastic-agent.service
             └─7497 elastic-agent

ubuntu@ubuntu24:~/elastic-agent-9.3.0-SNAPSHOT-linux-arm64$ sudo elastic-agent status --output=full
┌─ fleet
│  └─ status: (HEALTHY) Connected
└─ elastic-agent
   ├─ status: (HEALTHY) Running
   ├─ info
   │  ├─ id: 5bcd2335-398b-4a39-aa67-9185ccb0538f
   │  ├─ version: 9.3.0
   │  └─ commit: c26dee602037f4808433a7a40d9fcb7097555267
   ├─ beat/metrics-monitoring
   │  ├─ status: (HEALTHY) HEALTHY
   │  ├─ beat/metrics-monitoring
   │  │  ├─ status: (HEALTHY) Healthy
   │  │  └─ type: OUTPUT
   │  └─ beat/metrics-monitoring-metrics-monitoring-beats
   │     ├─ status: (HEALTHY) Healthy
   │     └─ type: INPUT
   ├─ filestream-monitoring
   │  ├─ status: (HEALTHY) HEALTHY
   │  ├─ filestream-monitoring
   │  │  ├─ status: (HEALTHY) Healthy
   │  │  └─ type: OUTPUT
   │  └─ filestream-monitoring-filestream-monitoring-agent
   │     ├─ status: (HEALTHY) Healthy
   │     └─ type: INPUT
   ├─ http/metrics-monitoring
   │  ├─ status: (HEALTHY) HEALTHY
   │  ├─ http/metrics-monitoring
   │  │  ├─ status: (HEALTHY) Healthy
   │  │  └─ type: OUTPUT
   │  └─ http/metrics-monitoring-metrics-monitoring-agent
   │     ├─ status: (HEALTHY) Healthy
   │     └─ type: INPUT
   └─ prometheus/metrics-monitoring
      ├─ status: (HEALTHY) HEALTHY
      ├─ prometheus/metrics-monitoring
      │  ├─ status: (HEALTHY) Healthy
      │  └─ type: OUTPUT
      └─ prometheus/metrics-monitoring-metrics-monitoring-collector
         ├─ status: (HEALTHY) Healthy
         └─ type: INPUT

Debug logs from when this happens are attached.

elastic-agent.log

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[swiatekm]

The following commit appears to fix the problem: ba45fad, but I'm not sure why exactly. I've been a bit suspicious of our status update handling when stopping the collector for a while, and I think this cleanup makes things simpler regardless, but I still can't think of a way for the current code to lead to a race condition. WDYT @pkoutsovasilis ?

The way this currently works is the following:

1. Manager stops the otel subprocess and waits until it actually stops. This involves, in order:
   1. Waiting for the process to actually exit.
   2. Cancelling the process context so the healthcheck loop emits its final status and exists.
   3. Waiting until the healthcheck loop exits.
2. After the healtcheck loop exits, the execution reports a final error from the process exit.
3. Manager waits until it gets the error above.
4. Manager drains the internal status channel.
5. Manager reports a final nil status, indicating nothing is running.

I have a suspicion that there's something funky in steps 4 and 5, where it's possible for an old status to be left in the internal channel and reported after the nil status, but I don't see how that can happen.

[swiatekm]

I think I understand how this happens. The coordinator expects the otel manager to report component statuses incrementally, whereas the otel manager reports them as point-in-time snapshots. More specifically, if the coordinator knows about a component A, it will only ever remove this component after it gets an explicit STOPPED status for it. The otel manager will usually emit that, but the actual guarantee it gives is that the statuses it emits are true at a given point in time. So instead of an array of STOPPED, it may just emit an empty array.

My previous change makes this less likely by making sure the otel manager doesn't emit its own status updates, and only forwards updates from the otel collector itself. But as long as the external status update channel is buffered and we drain it before publishing a new update, the bug can still happen.

I'm inclined to make the otel manager behave the same as the process manager and wait until the update is received by the coordinator synchronously.