Encountered channel not found error on adding Windows integration to the Windows agent.

[amolnater-qasource]

Kibana Build details:

VERSION: 8.16.0 SNAPSHOT
BUILD: 78938
COMMIT: 7b832691e8b07c67b411da95b0398a04711da864

Artifact: https://snapshots.elastic.co/8.16.0-39df64b4/downloads/beats/elastic-agent/elastic-agent-8.16.0-SNAPSHOT-windows-x86_64.zip

Host: Windows Server 2022- Test Signing ON

Preconditions:

1. 8.16.0 SNAPSHOT Kibana cloud environment should be available.
2. Agent should be installed with policy having System and Windowsintegrations.

Steps to reproduce:

1. Navigate to Agents tab.
2. Observe the Agent gets unhealthy and navigate to policy details page.
3. Observe error for Windows integration: Encountered channel not found error

Expected Result:
No error should be displayed on adding Windows integration to the Windows agent.

Logs:
elastic-agent-diagnostics-2024-10-09T06-48-15Z-00.zip

Screenshots:

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[amolnater-qasource]

@muskangulati-qasource Please review.

[muskangulati-qasource]

Secondary review is done for this ticket!

[cmacknz]

I see this is privileged/admin agent looking in agent-info.yaml:

agent_id: 881c5687-32af-4bf9-b62f-4b74f2f688ec
headers: {}
log_level: info
snapshot: true
unprivileged: false
version: 8.16.0

Also that this is coming from the winlog input. Tagging @nfritts and @elastic/sec-windows-platform.

            input-winlog-default-winlog-windows-4ea5f67a-48fc-41ea-b586-2a29eac6423a:
                message: 'Encountered channel not found error when opening Windows Event Log: The specified channel could not be found.'
                payload:
                    streams:
                        winlog-windows.forwarded-4ea5f67a-48fc-41ea-b586-2a29eac6423a:
                            error: ""
                            status: HEALTHY
                        winlog-windows.powershell-4ea5f67a-48fc-41ea-b586-2a29eac6423a:
                            error: ""
                            status: HEALTHY
                        winlog-windows.powershell_operational-4ea5f67a-48fc-41ea-b586-2a29eac6423a:
                            error: ""
                            status: HEALTHY
                        winlog-windows.sysmon_operational-4ea5f67a-48fc-41ea-b586-2a29eac6423a:
                            error: 'Encountered channel not found error when opening Windows Event Log: The specified channel could not be found.'
                            status: DEGRADED

[bjmcnic]

This issue has arisen following changes in this PR (elastic/beats#40163).

The default configuration for for the Windows Integration has historically included Sysmon Operational channel. Sysmon is not a core component of Windows, it's a SysInternals tool (https://learn.microsoft.com/en-us/sysinternals/downloads/sysmon) that users can download and use at their discretion. As such, the Windows Integration has historically failed to open the Sysmon Operational channel, but that didn't propagate a DEGRADED status until the recent PR.

Users can remedy the degraded status by either installing Sysmon, causing the channel to exist; or by deselecting the Sysmon Operational channel in the Windows Integration configuration...

Possible solutions:

• Revert the code change that causes a DEGRADED status when a configured channel is not found.
  • Obviously that could risk missing a helpful DEGRADED status that could indicate an unexpectedly missing non-default channel.
• Document and explain how this is now more correct.
• Change the Windows Integration default to not include Sysmon.

Thoughts @cmacknz @nfritts @andrewkroh ?

[cmacknz]

Change the Windows Integration default to not include Sysmon.

This sounds like the most correct path if sysmon is not expected to be present the majority of the time. The counter argument is that this is a breaking change.

Something we did for some of the system metricsets that were in a similar situation is keep the error message but report the status as healthy, since the input was working as well as it could with the configuration of the host system it was running on.

[jamiehynds]

In an ideal world, we'd move Sysmon out of Windows and have it as a standalone integration but that'd be very disruptive for existing users and would likely impact rules, dashboards, etc.

As a quick fix, could we exclude Sysmon from our DEGRADED logic? So keep Sysmon within the Windows integration, but if a user doesn't have Sysmon installed, we don't trigger a DEGRADED status?

[intxgo]

Can't Agent get it from https://live.sysinternals.com/Sysmon64.exe and install if it's missing? When adding policy with sysmon data collection.

[bjmcnic]

@jamiehynds

As a quick fix, could we exclude Sysmon from our DEGRADED logic? So keep Sysmon within the Windows integration, but if a user doesn't have Sysmon installed, we don't trigger a DEGRADED status?

I don't see a technical reason we couldn't just insert a check for whether we're trying to grab that particular channel near the code that changed. But that's filebeat code and not the Windows integration code. That'd be kind of an awkward place for the check in the long term and wouldn't scale well if we want to handle other things differently.

I noticed the PR that changed this was addressing: elastic/beats#39735. Which related to wanting to see failures for channels when permission is denied, typically when Agent is installed unprivileged. I recreated that and now DO see the desired Access is denied error:

c:\>"c:\Program Files\Elastic\Agent\elastic-agent.exe" status
┌─ fleet
│  └─ status: (HEALTHY) Connected
└─ elastic-agent
   ├─ status: (DEGRADED) 1 or more components/units in a failed state
   ├─ system/metrics-default
   │  ├─ status: (HEALTHY) Healthy: communicating with pid '8300'
   │  └─ system/metrics-default-system/metrics-system-68951c20-696b-4518-b64c-63de7317ef29
   │     └─ status: (DEGRADED) Error fetching data for metricset system.diskio: disk io counters: cannot open new key in the registry in order to enable the performance counters: Access is denied.
   ├─ windows/metrics-default
   │  ├─ status: (HEALTHY) Healthy: communicating with pid '5976'
   │  └─ windows/metrics-default-windows/metrics-windows-e02b045d-e899-4ef1-b6ce-400be3d94119
   │     └─ status: (FAILED) 1 error: initialization of reader failed: failed to expand counter (query='\Process(*)\% Processor Time'): Unable to connect to the specified computer or the computer is offline.
   └─ winlog-default
      ├─ status: (HEALTHY) Healthy: communicating with pid '5416'
      ├─ winlog-default-winlog-system-68951c20-696b-4518-b64c-63de7317ef29
      │  └─ status: (DEGRADED) failed to open Windows Event Log channel "Security": Access is denied.
      └─ winlog-default-winlog-windows-e02b045d-e899-4ef1-b6ce-400be3d94119
         └─ status: (DEGRADED) Encountered channel not found error when opening Windows Event Log: The specified channel could not be found.

The thing that strikes me there is we have multiple types of failures for winlog-default. One we newly want reported that hadn't been, and one we don't want reported that previously hadn't been.

I wonder if a good long term solution would be for the integration to feed in some type of filter data along with each channel it wants. Some enum or struct that tells it to actually degrade on access denied errors for this channel, but ignore not found errors for this channel, or warn/log (but not DEGRADE) for some other type of error for some other channel. Such that the integration can communicate how specific channel subscription failures should impact the integration's state. Trying to place that logic in code in filebeat seems awkward.

I'm not sure that change could be ready for 8.16.0. Perhaps we should rollback the change that caused this issue to arise and continue to tolerate the missing Access is denied as unprivileged appears to still be technically beta. And then we could incorporate the channel specific failure handling from the Windows integration in the next release. Thoughts? @cmacknz @jamiehynds @nfritts