Windows agent getting unhealthy with System integration and no data under kafka output.

[amolnater-qasource]

Kibana Build details:

VERSION: 8.17.0 SNAPSHOT
BUILD: 80188
COMMIT: fdb16ae8cbdf4236db3696aa00d0bb98c943d864

Artifact Link: https://snapshots.elastic.co/8.17.0-7a041bf5/downloads/beats/elastic-agent/elastic-agent-8.17.0-SNAPSHOT-windows-x86_64.zip

Preconditions:

1. 8.17.0-SNAPSHOT Kibana cloud environment should be available.

Steps to reproduce:

1. Install Windows agent.
2. Update output to kafka output.
3. Observe agent gets unhealthy and no output under kafka topic.

Expected Result:
Windows agent should remain healthy with System integration and data should be generated under kafka output.

Screenshot:

Agent Logs:
elastic-agent-diagnostics-2024-11-18T10-38-10Z-00.zip

What's working fine:

• Data for same kafka output is available for 8.16.0 agent.

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[amolnater-qasource]

@muskangulati-qasource Please review.

[muskangulati-qasource]

Secondary review is Done for this ticket

[cmacknz]

            input-beat/metrics-monitoring-metrics-monitoring-beats:
                message: 'Error fetching data for metricset beat.stats: error making http request: Get "http://npipe/stats": open \\.\pipe\Q540iWXFlriVpKtDsr_d4ccTYz2N3a_K.sock: The system cannot find the file specified.'
                payload:
                    streams:
                        metrics-monitoring-filebeat:
                            error: ""
                            status: HEALTHY
                        metrics-monitoring-metricbeat:
                            error: 'Error fetching data for metricset beat.stats: error making http request: Get "http://npipe/stats": open \\.\pipe\Q540iWXFlriVpKtDsr_d4ccTYz2N3a_K.sock: The system cannot find the file specified.'
                            status: DEGRADED
                state: 3

• Agent gets unhealthy temporarily because Beat monitoring sockets are not available #5332

This is #5332 again

[ycombinator]

#5332 was resolved last week. @amolnater-qasource would you mind re-testing this issue with the latest 8.17.0-SNAPSHOT? Thanks.

[amolnater-qasource]

@ycombinator Thank you for the update.

We have revalidated this issue on latest 8.17.0 snapshot available and found this issue still reproducible.

• No data under Kafka output is generated.

Artifact: https://snapshots.elastic.co/8.17.0-1c58bcd8/downloads/beats/elastic-agent/elastic-agent-8.17.0-SNAPSHOT-windows-x86_64.zip

Logs:
elastic-agent-diagnostics-2024-11-26T04-52-15Z-00.zip

Please let us know if we are missing anything here.

Thanks!!

[cmacknz]

Same problem as before Get \"http://npipe/stats\": open \\\\.\\pipe\\K5P2Tc74wcqesMl7DzGLkORtOcGrPO1a.sock: The system cannot find the file specified.:

{"log.level":"warn","@timestamp":"2024-11-26T04:43:43.726Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":663},"message":"Unit state changed beat/metrics-monitoring-metrics-monitoring-beats (HEALTHY->DEGRADED): Error fetching data for metricset beat.stats: error making http request: Get \"http://npipe/stats\": open \\\\.\\pipe\\K5P2Tc74wcqesMl7DzGLkORtOcGrPO1a.sock: The system cannot find the file specified.","log":{"source":"elastic-agent"},"component":{"id":"beat/metrics-monitoring","state":"HEALTHY"},"unit":{"id":"beat/metrics-monitoring-metrics-monitoring-beats","type":"input","state":"DEGRADED","old_state":"HEALTHY"},"ecs.version":"1.6.0"}

The intended fix for this is in 8.17 but it looks like it isn't helping here for some reason, https://github.com/elastic/elastic-agent/commits/8.17

Assigning to @pchila to investigate.

[pchila]

@cmacknz from the logs we can see that we get the error about opening the named pipe more than once (with the fix of #5332 we trigger by default at the second consecutive error, with the assumption that the fetch will be retried in 60s)

What we can see from the logs is that the unit beat/metrics-monitoring-metrics-monitoring-beats is retrying to fetch much more quickly than expected...

{"log.level":"info","@timestamp":"2024-11-26T04:25:42.656Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":663},"message":"Unit state changed beat/metrics-monitoring-metrics-monitoring-beats (CONFIGURING->HEALTHY): Healthy","log":{"source":"elastic-agent"},"component":{"id":"beat/metrics-monitoring","state":"HEALTHY"},"unit":{"id":"beat/metrics-monitoring-metrics-monitoring-beats","type":"input","state":"HEALTHY","old_state":"CONFIGURING"},"ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-11-26T04:25:42.658Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":663},"message":"Unit state changed beat/metrics-monitoring-metrics-monitoring-beats (HEALTHY->DEGRADED): Error fetching data for metricset beat.stats: error making http request: Get \"http://npipe/stats\": open \\\\.\\pipe\\K5P2Tc74wcqesMl7DzGLkORtOcGrPO1a.sock: The system cannot find the file specified.","log":{"source":"elastic-agent"},"component":{"id":"beat/metrics-monitoring","state":"HEALTHY"},"unit":{"id":"beat/metrics-monitoring-metrics-monitoring-beats","type":"input","state":"DEGRADED","old_state":"HEALTHY"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-26T04:25:42.657Z","message":"Beat ID: 779a746f-aa35-4946-8b9e-06ab08036161","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"winlog-222f7bb5-918e-4247-87cc-d93c0f1d496b","type":"winlog"},"log":{"source":"winlog-222f7bb5-918e-4247-87cc-d93c0f1d496b"},"service.name":"filebeat","ecs.version":"1.6.0","log.origin":{"file.line":1070,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).configure"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-11-26T04:25:42.669Z","message":"Error fetching data for metricset http.json: error making http request: Get \"http://npipe/stats\": open \\\\.\\pipe\\K5P2Tc74wcqesMl7DzGLkORtOcGrPO1a.sock: The system cannot find the file specified.","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"log.origin":{"file.line":333,"file.name":"module/wrapper.go","function":"github.com/elastic/beats/v7/metricbeat/mb/module.(*metricSetWrapper).handleFetchError"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-11-26T04:25:42.671Z","message":"Error fetching data for metricset http.json: error making http request: Get \"http://npipe/inputs\": open \\\\.\\pipe\\K5P2Tc74wcqesMl7DzGLkORtOcGrPO1a.sock: The system cannot find the file specified.","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"service.name":"metricbeat","ecs.version":"1.6.0","log.origin":{"file.line":333,"file.name":"module/wrapper.go","function":"github.com/elastic/beats/v7/metricbeat/mb/module.(*metricSetWrapper).handleFetchError"},"ecs.version":"1.6.0"}","type":"beat/metrics"},"log":{"source":"beat/metrics-monitoring"},"service.name":"metricbeat","ecs.version":"1.6.0","log.origin":{"file.line":333,"file.name":"module/wrapper.go","function":"github.com/elastic/beats/v7/metricbeat/mb/module.(*metricSetWrapper).handleFetchError"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-26T04:25:42.656Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":663},"message":"Unit state changed beat/metrics-monitoring-metrics-monitoring-beats (CONFIGURING->HEALTHY): Healthy","log":{"source":"elastic-agent"},"component":{"id":"beat/metrics-monitoring","state":"HEALTHY"},"unit":{"id":"beat/metrics-monitoring-metrics-monitoring-beats","type":"input","state":"HEALTHY","old_state":"CONFIGURING"},"ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-11-26T04:25:42.658Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":663},"message":"Unit state changed beat/metrics-monitoring-metrics-monitoring-beats (HEALTHY->DEGRADED): Error fetching data for metricset beat.stats: error making http request: Get \"http://npipe/stats\": open \\\\.\\pipe\\K5P2Tc74wcqesMl7DzGLkORtOcGrPO1a.sock: The system cannot find the file specified.","log":{"source":"elastic-agent"},"component":{"id":"beat/metrics-monitoring","state":"HEALTHY"},"unit":{"id":"beat/metrics-monitoring-metrics-monitoring-beats","type":"input","state":"DEGRADED","old_state":"HEALTHY"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-26T04:25:42.657Z","message":"Beat ID: 779a746f-aa35-4946-8b9e-06ab08036161","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"winlog-222f7bb5-918e-4247-87cc-d93c0f1d496b","type":"winlog"},"log":{"source":"winlog-222f7bb5-918e-4247-87cc-d93c0f1d496b"},"service.name":"filebeat","ecs.version":"1.6.0","log.origin":{"file.line":1070,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).configure"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-11-26T04:25:42.669Z","message":"Error fetching data for metricset http.json: error making http request: Get \"http://npipe/stats\": open \\\\.\\pipe\\K5P2Tc74wcqesMl7DzGLkORtOcGrPO1a.sock: The system cannot find the file specified.","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"log.origin":{"file.line":333,"file.name":"module/wrapper.go","function":"github.com/elastic/beats/v7/metricbeat/mb/module.(*metricSetWrapper).handleFetchError"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-11-26T04:25:42.671Z","message":"Error fetching data for metricset http.json: error making http request: Get \"http://npipe/inputs\": open \\\\.\\pipe\\K5P2Tc74wcqesMl7DzGLkORtOcGrPO1a.sock: The system cannot find the file specified.","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"service.name":"metricbeat","ecs.version":"1.6.0","log.origin":{"file.line":333,"file.name":"module/wrapper.go","function":"github.com/elastic/beats/v7/metricbeat/mb/module.(*metricSetWrapper).handleFetchError"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-11-26T04:25:42.671Z","message":"Error fetching data for metricset http.json: error making http request: Get \"http://npipe/stats\": open \\\\.\\pipe\\hC6H1faJ6uJdcqwMEc7XDxNvsCB7nGo1.sock: The system cannot find the file specified.","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"log.origin":{"file.line":333,"file.name":"module/wrapper.go","function":"github.com/elastic/beats/v7/metricbeat/mb/module.(*metricSetWrapper).handleFetchError"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-26T04:25:42.677Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":663},"message":"Unit state changed http/metrics-monitoring-metrics-monitoring-agent (CONFIGURING->HEALTHY): Healthy","log":{"source":"elastic-agent"},"component":{"id":"http/metrics-monitoring","state":"HEALTHY"},"unit":{"id":"http/metrics-monitoring-metrics-monitoring-agent","type":"input","state":"HEALTHY","old_state":"CONFIGURING"},"ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-11-26T04:25:42.677Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":663},"message":"Unit state changed http/metrics-monitoring-metrics-monitoring-agent (HEALTHY->DEGRADED): Error fetching data for metricset http.json: error making http request: Get \"http://npipe/inputs\": open \\\\.\\pipe\\K5P2Tc74wcqesMl7DzGLkORtOcGrPO1a.sock: The system cannot find the file specified.","log":{"source":"elastic-agent"},"component":{"id":"http/metrics-monitoring","state":"HEALTHY"},"unit":{"id":"http/metrics-monitoring-metrics-monitoring-agent","type":"input","state":"DEGRADED","old_state":"HEALTHY"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-26T04:25:42.681Z","message":"Output reload is enabled, the beat will restart as needed on change of output config","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"winlog-222f7bb5-918e-4247-87cc-d93c0f1d496b","type":"winlog"},"log":{"source":"winlog-222f7bb5-918e-4247-87cc-d93c0f1d496b"},"log.logger":"centralmgmt","log.origin":{"file.line":204,"file.name":"management/managerV2.go","function":"github.com/elastic/beats/v7/x-pack/libbeat/management.NewV2AgentManagerWithClient"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-26T04:25:42.681Z","message":"Set gc percentage to: 100","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"winlog-222f7bb5-918e-4247-87cc-d93c0f1d496b","type":"winlog"},"log":{"source":"winlog-222f7bb5-918e-4247-87cc-d93c0f1d496b"},"log.origin":{"file.line":1124,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).configure"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-11-26T04:25:42.684Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":663},"message":"Unit state changed beat/metrics-monitoring-metrics-monitoring-beats (DEGRADED->HEALTHY): Healthy","log":{"source":"elastic-agent"},"component":{"id":"beat/metrics-monitoring","state":"HEALTHY"},"unit":{"id":"beat/metrics-monitoring-metrics-monitoring-beats","type":"input","state":"HEALTHY","old_state":"DEGRADED"},"ecs.version":"1.6.0"}

Probably for these cases the default of triggering the DEGRADED state on the second consecutive error is not enough... I will test to see if there's a more appropriate value for this input tomorrow.

[cmacknz]

It may be because the Beats restarted because of an output change that caused the second instance of the problem. Beats decide to do this on their own and don't communicate their intent to restart to agent.