Elastic-Agents unexpectedly unenrolled after update to 8.16.x

[syk-99]

• Version:
  8.16.0 and 8.16.1

• Operating System:
  Windows Server 2019 (64-bit), Microsoft Windows Server 2022 (64-bit)

• Discuss Forum URL:
  https://discuss.elastic.co/t/elastic-agent-unexpectedly-gets-unenrolled-version-8-16-x/371463

• related Links:
  [Fleet] Agents erroneously showing "Agent not upgradeable: agent has been unenrolled" kibana#202440
  [Fleet]: Unable to force unenroll multiple offline agents from bulk actions. kibana#197180

• Steps to Reproduce:

  1. Upgrade Elasticsearch, Kibana, Fleet, Agents from 8.15.4 -> 8.16.0 -> 8.16.1
  2. Stop Elastic-Agent Service in Windows for 1-2 days (we didn't stop it manually - it just didn't start automatically after patchday reboot)
  3. Start Elastic-Agent Service again

Agent-Logs (.fleet-agent) shows timestamp of 2) -> unenrolled_at and timestamp of 3) -> upgraded_at

• Example-Logs and Screenshots:
  Upgraded agent from 8.16.0 to 8.16.1 on Nov. 25th
  Rebooted Host on Nov. 30th
  Started Agent-Service on Dec. 2nd
  

• Agent-Log from day of upgrade (I think, the error happens here):
  too long to fit here -> could someone provide a hint how to export the relevant log entries from a specific agent?

• Agent-Log from day of reboot & day of service-start (copy/paste from Kibana->Fleet->Agent->specific agent->Logs):
  Nov 30, 2024
  20:09:04.513
  elastic_agent
  [elastic_agent][info] signal "terminated" received
  20:09:04.513
  elastic_agent
  [elastic_agent][info] Shutting down Elastic Agent and sending last events...
  20:09:04.520
  elastic_agent
  [elastic_agent][warn] Possible transient error during checkin with fleet-server, retrying
  20:09:04.552
  elastic_agent
  [elastic_agent][error] failed accept conn info connection: use of closed network connection
  20:09:04.552
  elastic_agent
  [elastic_agent][info] stopping endpoint service runtime
  20:09:04.720
  elastic_agent
  [elastic_agent][info] Shutting down completed.
  20:09:04.728
  elastic_agent
  [elastic_agent][info] Stopping monitoring server
  20:09:04.728
  elastic_agent
  [elastic_agent][info] Stats endpoint (127.0.0.1:6791) finished: accept tcp 127.0.0.1:6791: use of closed network connection

Dec 2, 2024
09:25:50.042
elastic_agent
[elastic_agent][info] Elastic Agent started
09:25:50.331
elastic_agent
[elastic_agent][info] Starting upgrade watcher
09:25:50.365
elastic_agent
[elastic_agent][info] Upgrade Watcher invoked
09:25:50.692
elastic_agent
[elastic_agent][info] Upgrade Watcher started
09:25:50.708
elastic_agent
[elastic_agent][info] Loaded update marker &{Version:8.16.1 Hash:b6da7f VersionedHome:data\elastic-agent-8.16.1-b6da7f UpdatedOn:2024-11-25 12:20:00.3690588 +0100 CET PrevVersion:8.16.0 PrevHash:3f07f2 PrevVersionedHome:data\elastic-agent-8.16.0-3f07f2 Acked:false Action:id: f0d5d0c4-b283-419e-b826-a8e830f755cc, type: UPGRADE Details:}
09:25:50.714
elastic_agent
[elastic_agent][info] not within grace [updatedOn 2024-11-25 12:20:00.3690588 +0100 CET] 165h5m50.3458541s
09:25:50.714
elastic_agent
[elastic_agent][info] Cleaning up upgrade
09:25:50.828
elastic_agent
[elastic_agent][info] APM instrumentation disabled
09:25:50.838
elastic_agent
[elastic_agent][info] Gathered system information
09:25:50.870
elastic_agent
[elastic_agent][info] Detected available inputs and outputs
09:25:50.870
elastic_agent
[elastic_agent][info] Capabilities file not found in C:\Program Files\Elastic\Agent\capabilities.yml
09:25:50.870
elastic_agent
[elastic_agent][info] Determined allowed capabilities
09:25:50.870
elastic_agent
[elastic_agent][info] Loading baseline config from C:\Program Files\Elastic\Agent\elastic-agent.yml
09:25:51.312
elastic_agent
[elastic_agent][info] GRPC comms socket listening at localhost:6789
09:25:51.439
elastic_agent
[elastic_agent][info] Parsed configuration and determined agent is managed by Fleet
09:25:51.439
elastic_agent
[elastic_agent][warn] SSL/TLS verifications disabled.
09:25:51.652
elastic_agent
[elastic_agent][info] GRPC control socket listening at npipe:///elastic-agent-system
09:25:51.656
elastic_agent
[elastic_agent][info] updated upgrade details
09:25:51.660
elastic_agent
[elastic_agent][info] Starting grpc control protocol listener on port 6789 with max_message_size 104857600
09:25:51.660
elastic_agent
[elastic_agent][info] Docker provider skipped, unable to connect: protocol not available
09:25:51.879
elastic_agent
[elastic_agent][warn] SSL/TLS verifications disabled.
09:25:52.986
elastic_agent
[elastic_agent][info] restoring current policy from disk
09:25:53.030
elastic_agent
[elastic_agent][info] Setting fallback log level from policy
09:25:53.067
elastic_agent
[elastic_agent][info] Fleet gateway started
09:25:53.080
elastic_agent
[elastic_agent][info] Source URI changed from "https://artifacts.elastic.co/downloads/" to "https://artifacts.elastic.co/downloads/"
09:25:53.080
elastic_agent
[elastic_agent][info] Starting monitoring server with cfg &config.MonitoringConfig{Enabled:true, MonitorLogs:true, MonitorMetrics:false, MetricsPeriod:"", LogMetrics:true, HTTP:(*config.MonitoringHTTPConfig)(0xc00067d5f0), Namespace:"default", Pprof:(*config.PprofConfig)(nil), MonitorTraces:false, APM:config.APMConfig{Environment:"", APIKey:"", SecretToken:"", Hosts:[]string(nil), GlobalLabels:map[string]string(nil), TLS:config.APMTLS{SkipVerify:false, ServerCertificate:"", ServerCA:""}, SamplingRate:(*float32)(nil)}, Diagnostics:config.Diagnostics{Uploader:config.Uploader{MaxRetries:10, InitDur:1000000000, MaxDur:600000000000}, Limit:config.Limit{Interval:60000000000, Burst:1}}}
09:25:53.083
elastic_agent
[elastic_agent][info] creating monitoring API with cfg api.Config{Enabled:true, Host:"http://localhost:6791", Port:6791, User:"", SecurityDescriptor:"", Timeout:5000000000}
09:25:53.084
elastic_agent
[elastic_agent][info] Starting stats endpoint
09:25:53.105
elastic_agent
[elastic_agent][info] Metrics endpoint listening on: 127.0.0.1:6791 (configured: http://localhost:6791)
09:25:53.107
elastic_agent
[elastic_agent][info] Updating running component model
09:25:54.355
elastic_agent
[elastic_agent][info] Creating connection info server for endpoint service, address: npipe:///.eaci.sock
09:25:54.356
elastic_agent
[elastic_agent][info] check if endpoint service is installed
09:25:54.552
elastic_agent
endpoint-default
[elastic_agent][info] Spawned new component endpoint-default: Starting: endpoint service runtime
09:25:54.552
elastic_agent
endpoint-default
[elastic_agent][info] Spawned new unit endpoint-default-85821b10-0064-11ee-b676-af36e033a9ae: Starting: endpoint service runtime
09:25:54.553
elastic_agent
endpoint-default
[elastic_agent][info] Spawned new unit endpoint-default: Starting: endpoint service runtime
09:25:56.570
elastic_agent
[elastic_agent][error] 2024-12-02 08:25:56: info: Main.cpp:569 Verifying existing installation
09:25:56.574
elastic_agent
[elastic_agent][error] 2024-12-02 08:25:56: info: InstallLib.cpp:611 Running [C:\Program Files\Elastic\Endpoint\elastic-endpoint.exe] [version --log stdout]
09:25:56.574
elastic_agent
[elastic_agent][error] 2024-12-02 08:25:56: debug: Service.cpp:804 PPL is supported. This process is unprotected. (TrustLevelSid: absent)
09:25:57.282
elastic_agent
[elastic_agent][error] 2024-12-02 08:25:57: info: InstallLib.cpp:650 Installed endpoint is expected version (version: 8.16.1, compiled: Tue Nov 19 12:00:00 2024, branch: HEAD, commit: 7d50b182b0f0ddc7170095904dc1e341224bb1f4)
09:25:57.282
elastic_agent
[elastic_agent][error] 2024-12-02 08:25:57: info: Util.cpp:2146 Endpoint Service is running.
09:25:57.286
elastic_agent
[elastic_agent][info] after check if endpoint service is installed, err:
09:26:01.240
elastic_agent
winlog-default
[elastic_agent][info] Spawned new component winlog-default: Starting: spawned pid '20108'
09:26:01.241
elastic_agent
winlog-default
[elastic_agent][info] Spawned new unit winlog-default-winlog-system-85821b11-0064-11ee-b676-af36e033a9ae: Starting: spawned pid '20108'
09:26:01.241
elastic_agent
winlog-default
[elastic_agent][info] Spawned new unit winlog-default-winlog-windows-85821b12-0064-11ee-b676-af36e033a9ae: Starting: spawned pid '20108'
09:26:01.241
elastic_agent
winlog-default
[elastic_agent][info] Spawned new unit winlog-default: Starting: spawned pid '20108'
09:26:03.099
elastic_agent
[elastic_agent][info] control checkin v2 protocol has chunking enabled
09:26:03.100
elastic_agent
winlog-default
[elastic_agent][info] Component state changed winlog-default (STARTING->HEALTHY): Healthy: communicating with pid '20108'
09:26:03.175
elastic_agent
[elastic_agent][info] control checkin v2 protocol has chunking enabled
09:26:04.111
elastic_agent
winlog-default
[elastic_agent][info] Unit state changed winlog-default (STARTING->HEALTHY): Healthy
09:26:04.114
elastic_agent
winlog-default
[elastic_agent][info] Unit state changed winlog-default-winlog-system-85821b11-0064-11ee-b676-af36e033a9ae (STARTING->HEALTHY): Healthy
09:26:04.114
elastic_agent
winlog-default
[elastic_agent][info] Unit state changed winlog-default-winlog-windows-85821b12-0064-11ee-b676-af36e033a9ae (STARTING->HEALTHY): Healthy
09:26:10.278
elastic_agent
endpoint-default
[elastic_agent][info] Component state changed endpoint-default (STARTING->HEALTHY): Healthy: communicating with endpoint service
09:26:10.818
elastic_agent
[elastic_agent][info] Removing marker file
09:26:10.822
elastic_agent
[elastic_agent][info] Removing previous symlink path
09:26:10.822
elastic_agent
[elastic_agent][error] clean up of prior watcher run failedextracting elastic-agent path relative to data directory from C:\Program Files\Elastic\Agent\data\elastic-agent-8.16.1-b6da7f: Rel: can't make C:\Program Files\Elastic\Agent\data\elastic-agent-8.16.1-b6da7f relative to data
09:26:15.763
elastic_agent
endpoint-default
[elastic_agent][info] Unit state changed endpoint-default-85821b10-0064-11ee-b676-af36e033a9ae (STARTING->HEALTHY): Applied policy {85821b10-0064-11ee-b676-af36e033a9ae}
09:26:15.764
elastic_agent
endpoint-default
[elastic_agent][info] Unit state changed endpoint-default (STARTING->HEALTHY): Applied policy {85821b10-0064-11ee-b676-af36e033a9ae}
09:26:21.526
elastic_agent
[elastic_agent][info] component model updated
09:26:21.527
elastic_agent
[elastic_agent][info] Updating running component model
11:16:50.539
elastic_agent
endpoint-default
[elastic_agent][info] Unit state changed endpoint-default (HEALTHY->CONFIGURING): Applied policy {85821b10-0064-11ee-b676-af36e033a9ae}
11:16:50.539
elastic_agent
endpoint-default
[elastic_agent][info] Unit state changed endpoint-default-85821b10-0064-11ee-b676-af36e033a9ae (HEALTHY->CONFIGURING): Applied policy {85821b10-0064-11ee-b676-af36e033a9ae}
11:17:10.534
elastic_agent
endpoint-default
[elastic_agent][info] Unit state changed endpoint-default-85821b10-0064-11ee-b676-af36e033a9ae (CONFIGURING->HEALTHY): Applied policy {85821b10-0064-11ee-b676-af36e033a9ae}
11:17:10.534
elastic_agent
endpoint-default
[elastic_agent][info] Unit state changed endpoint-default (CONFIGURING->HEALTHY): Applied policy {85821b10-0064-11ee-b676-af36e033a9ae}
13:28:44.169
elastic_agent
[elastic_agent][warn] SSL/TLS verifications disabled.
13:28:44.170
elastic_agent
[elastic_agent][warn] SSL/TLS verifications disabled.
13:28:44.185
elastic_agent
[elastic_agent][info] Setting fallback log level from policy
13:28:44.218
elastic_agent
[elastic_agent][warn] SSL/TLS verifications disabled.
13:28:44.236
elastic_agent
[elastic_agent][info] Source URI changed from "https://artifacts.elastic.co/downloads/" to "https://artifacts.elastic.co/downloads/"
13:28:44.237
elastic_agent
[elastic_agent][info] Stopping monitoring server
13:28:44.237
elastic_agent
[elastic_agent][info] Stats endpoint (127.0.0.1:6791) finished: accept tcp 127.0.0.1:6791: use of closed network connection
13:28:44.238
elastic_agent
[elastic_agent][info] Starting monitoring server with cfg &config.MonitoringConfig{Enabled:true, MonitorLogs:true, MonitorMetrics:false, MetricsPeriod:"", LogMetrics:true, HTTP:(*config.MonitoringHTTPConfig)(0xc00078af90), Namespace:"default", Pprof:(*config.PprofConfig)(nil), MonitorTraces:false, APM:config.APMConfig{Environment:"", APIKey:"", SecretToken:"", Hosts:[]string(nil), GlobalLabels:map[string]string(nil), TLS:config.APMTLS{SkipVerify:false, ServerCertificate:"", ServerCA:""}, SamplingRate:(*float32)(nil)}, Diagnostics:config.Diagnostics{Uploader:config.Uploader{MaxRetries:10, InitDur:1000000000, MaxDur:600000000000}, Limit:config.Limit{Interval:60000000000, Burst:1}}}
13:28:44.238
elastic_agent
[elastic_agent][info] creating monitoring API with cfg api.Config{Enabled:true, Host:"http://localhost:6791", Port:6791, User:"", SecurityDescriptor:"", Timeout:5000000000}
13:28:44.240
elastic_agent
[elastic_agent][info] Starting stats endpoint
13:28:44.242
elastic_agent
[elastic_agent][info] Metrics endpoint listening on: 127.0.0.1:6791 (configured: http://localhost:6791)
13:28:44.267
elastic_agent
[elastic_agent][info] component model updated
13:28:44.267
elastic_agent
[elastic_agent][info] Updating running component model
13:28:44.274
elastic_agent
winlog-default
[elastic_agent][info] Unit state changed winlog-default-winlog-windows-85821b12-0064-11ee-b676-af36e033a9ae (HEALTHY->CONFIGURING): Configuring
13:28:44.274
elastic_agent
winlog-default
[elastic_agent][info] Unit state changed winlog-default-winlog-system-85821b11-0064-11ee-b676-af36e033a9ae (HEALTHY->CONFIGURING): Configuring
13:28:44.300
elastic_agent
[elastic_agent][warn] SSL/TLS verifications disabled.
13:28:45.275
elastic_agent
winlog-default
[elastic_agent][info] Unit state changed winlog-default-winlog-windows-85821b12-0064-11ee-b676-af36e033a9ae (CONFIGURING->HEALTHY): Healthy
13:28:45.275
elastic_agent
winlog-default
[elastic_agent][info] Unit state changed winlog-default-winlog-system-85821b11-0064-11ee-b676-af36e033a9ae (CONFIGURING->HEALTHY): Healthy
19:17:13.888
elastic_agent
endpoint-default
[elastic_agent][info] Unit state changed endpoint-default-85821b10-0064-11ee-b676-af36e033a9ae (HEALTHY->CONFIGURING): Applied policy {85821b10-0064-11ee-b676-af36e033a9ae}
19:17:13.889
elastic_agent
endpoint-default
[elastic_agent][info] Unit state changed endpoint-default (HEALTHY->CONFIGURING): Applied policy {85821b10-0064-11ee-b676-af36e033a9ae}
19:17:33.888
elastic_agent
endpoint-default
[elastic_agent][info] Unit state changed endpoint-default-85821b10-0064-11ee-b676-af36e033a9ae (CONFIGURING->HEALTHY): Applied policy {85821b10-0064-11ee-b676-af36e033a9ae}
19:17:33.888
elastic_agent
endpoint-default
[elastic_agent][info] Unit state changed endpoint-default (CONFIGURING->HEALTHY): Applied policy {85821b10-0064-11ee-b676-af36e033a9ae}

[ycombinator]

Agent-Log from day of upgrade (I think, the error happens here):
too long to fit here -> could someone provide a hint how to export the relevant log entries from a specific agent?

You could collect a diagnostics bundle for the specific Agent. If you extract the bundle, you'll see a folder in it for logs. You could try to attach those files here or post the logs in a gist and link it from here.

[syk-99]

Here the agent-log from the day of the upgrade:

Showing entries from Nov 25, 11:21:41
11:21:41.266	elastic_agent	endpoint-default	[elastic_agent][warn] Unit state changed endpoint-default-85821b10-0064-11ee-b676-af36e033a9ae (HEALTHY->DEGRADED): Applied policy {85821b10-0064-11ee-b676-af36e033a9ae}
11:21:41.266	elastic_agent	endpoint-default	[elastic_agent][warn] Unit state changed endpoint-default (HEALTHY->DEGRADED): Applied policy {85821b10-0064-11ee-b676-af36e033a9ae}
11:23:38.425	elastic_agent	[elastic_agent][warn] Possible transient error during checkin with fleet-server, retrying
11:25:08.415	elastic_agent	[elastic_agent][warn] Possible transient error during checkin with fleet-server, retrying
11:28:04.264	elastic_agent	[elastic_agent][error] Cannot checkin in with fleet-server, retrying
11:34:21.351	elastic_agent	endpoint-default	[elastic_agent][info] Unit state changed endpoint-default-85821b10-0064-11ee-b676-af36e033a9ae (DEGRADED->HEALTHY): Applied policy {85821b10-0064-11ee-b676-af36e033a9ae}
11:34:21.351	elastic_agent	endpoint-default	[elastic_agent][info] Unit state changed endpoint-default (DEGRADED->HEALTHY): Applied policy {85821b10-0064-11ee-b676-af36e033a9ae}
11:40:10.135	elastic_agent	[elastic_agent][warn] Checkin request to fleet-server succeeded after 3 failures
12:14:32.285	elastic_agent	[elastic_agent][warn] Possible transient error during checkin with fleet-server, retrying
12:19:26.770	elastic_agent	[elastic_agent][warn] Checkin request to fleet-server succeeded after 1 failures
12:19:27.872	elastic_agent	[elastic_agent][info] starting upgrade to version 8.16.1 in background
12:19:27.874	elastic_agent	[elastic_agent][info] Upgrading agent
12:19:27.874	elastic_agent	[elastic_agent][info] updated upgrade details
12:19:27.874	elastic_agent	[elastic_agent][info] Cleaning up non-matching downloaded versions
12:19:27.876	elastic_agent	[elastic_agent][info] updated upgrade details
12:19:27.876	elastic_agent	[elastic_agent][info] Downloading upgrade artifact
12:19:27.879	elastic_agent	[elastic_agent][info] download attempt 1
12:19:27.879	elastic_agent	[elastic_agent][info] updated upgrade details
12:19:28.008	elastic_agent	[elastic_agent][info] updated upgrade details
12:19:42.730	elastic_agent	[elastic_agent][info] download from https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.16.1-windows-x86_64.zip completed in 14 seconds @ 13.39MBps
12:19:42.732	elastic_agent	[elastic_agent][info] updated upgrade details
12:19:42.746	elastic_agent	[elastic_agent][info] updated upgrade details
12:19:42.747	elastic_agent	[elastic_agent][info] download from https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.16.1-windows-x86_64.zip.sha512 completed in Less than a second @ +InfYBps
12:19:42.747	elastic_agent	[elastic_agent][info] updated upgrade details
12:19:42.749	elastic_agent	[elastic_agent][info] updated upgrade details
12:19:42.749	elastic_agent	[elastic_agent][info] updated upgrade details
12:19:43.417	elastic_agent	[elastic_agent][info] Default PGP appended
12:19:43.452	elastic_agent	[elastic_agent][info] Using 2 PGP keys
12:19:44.145	elastic_agent	[elastic_agent][info] Default PGP appended
12:19:44.158	elastic_agent	[elastic_agent][info] Using 2 PGP keys
12:19:44.765	elastic_agent	[elastic_agent][info] Verification with PGP[0] successful
12:19:44.766	elastic_agent	[elastic_agent][info] updated upgrade details
12:19:44.774	elastic_agent	[elastic_agent][info] Unpacking agent package
12:20:01.401	elastic_agent	[elastic_agent][info] Upgrade Watcher started
12:20:01.414	elastic_agent	[elastic_agent][info] Loaded update marker &{Version:8.16.1 Hash:b6da7f VersionedHome:data\elastic-agent-8.16.1-b6da7f UpdatedOn:2024-11-25 12:20:00.3690588 +0100 CET PrevVersion:8.16.0 PrevHash:3f07f2 PrevVersionedHome:data\elastic-agent-8.16.0-3f07f2 Acked:false Action:id: f0d5d0c4-b283-419e-b826-a8e830f755cc, type: UPGRADE Details:0xc00045b040}
12:20:01.442	elastic_agent	[elastic_agent][info] Agent watcher started
12:20:19.147	elastic_agent	[elastic_agent][info] Elastic Agent started
12:20:19.424	elastic_agent	[elastic_agent][info] Starting upgrade watcher
12:20:19.457	elastic_agent	[elastic_agent][info] Upgrade Watcher invoked
12:20:19.574	elastic_agent	[elastic_agent][info] APM instrumentation disabled
12:20:19.592	elastic_agent	[elastic_agent][info] Gathered system information
12:20:19.626	elastic_agent	[elastic_agent][info] Detected available inputs and outputs
12:20:19.626	elastic_agent	[elastic_agent][info] Capabilities file not found in C:\Program Files\Elastic\Agent\capabilities.yml
12:20:19.626	elastic_agent	[elastic_agent][info] Determined allowed capabilities
12:20:19.626	elastic_agent	[elastic_agent][info] Loading baseline config from C:\Program Files\Elastic\Agent\elastic-agent.yml
12:20:19.672	elastic_agent	[elastic_agent][info] Upgrade Watcher started
12:20:19.693	elastic_agent	[elastic_agent][info] Loaded update marker &{Version:8.16.1 Hash:b6da7f VersionedHome:data\elastic-agent-8.16.1-b6da7f UpdatedOn:2024-11-25 12:20:00.3690588 +0100 CET PrevVersion:8.16.0 PrevHash:3f07f2 PrevVersionedHome:data\elastic-agent-8.16.0-3f07f2 Acked:false Action:id: f0d5d0c4-b283-419e-b826-a8e830f755cc, type: UPGRADE Details:0xc000463400}
12:20:19.699	elastic_agent	[elastic_agent][info] exiting, lock already exists
12:20:19.854	elastic_agent	[elastic_agent][info] GRPC comms socket listening at localhost:6789
12:20:19.875	elastic_agent	[elastic_agent][info] Parsed configuration and determined agent is managed by Fleet
12:20:19.875	elastic_agent	[elastic_agent][warn] SSL/TLS verifications disabled.
12:20:20.010	elastic_agent	[elastic_agent][info] GRPC control socket listening at npipe:///elastic-agent-system
12:20:20.014	elastic_agent	[elastic_agent][info] updated upgrade details
12:20:20.014	elastic_agent	[elastic_agent][info] Docker provider skipped, unable to connect: protocol not available
12:20:20.016	elastic_agent	[elastic_agent][info] Starting grpc control protocol listener on port 6789 with max_message_size 104857600
12:20:20.236	elastic_agent	[elastic_agent][warn] SSL/TLS verifications disabled.
12:20:20.934	elastic_agent	[elastic_agent][info] restoring current policy from disk
12:20:20.982	elastic_agent	[elastic_agent][info] Setting fallback log level <nil> from policy
12:20:21.030	elastic_agent	[elastic_agent][info] Fleet gateway started
12:20:21.050	elastic_agent	[elastic_agent][info] Source URI changed from "https://artifacts.elastic.co/downloads/" to "https://artifacts.elastic.co/downloads/"
12:20:21.051	elastic_agent	[elastic_agent][info] Starting monitoring server with cfg &config.MonitoringConfig{Enabled:true, MonitorLogs:true, MonitorMetrics:false, MetricsPeriod:"", LogMetrics:true, HTTP:(*config.MonitoringHTTPConfig)(0xc0006fcdb0), Namespace:"default", Pprof:(*config.PprofConfig)(nil), MonitorTraces:false, APM:config.APMConfig{Environment:"", APIKey:"", SecretToken:"", Hosts:[]string(nil), GlobalLabels:map[string]string(nil), TLS:config.APMTLS{SkipVerify:false, ServerCertificate:"", ServerCA:""}, SamplingRate:(*float32)(nil)}, Diagnostics:config.Diagnostics{Uploader:config.Uploader{MaxRetries:10, InitDur:1000000000, MaxDur:600000000000}, Limit:config.Limit{Interval:60000000000, Burst:1}}}
12:20:21.051	elastic_agent	[elastic_agent][info] creating monitoring API with cfg api.Config{Enabled:true, Host:"http://localhost:6791", Port:6791, User:"", SecurityDescriptor:"", Timeout:5000000000}
12:20:21.053	elastic_agent	[elastic_agent][info] Starting stats endpoint
12:20:21.055	elastic_agent	[elastic_agent][info] Metrics endpoint listening on: 127.0.0.1:6791 (configured: http://localhost:6791)
12:20:21.070	elastic_agent	[elastic_agent][info] Updating running component model
12:20:21.755	elastic_agent	[elastic_agent][info] Creating connection info server for endpoint service, address: npipe:///.eaci.sock
12:20:21.756	elastic_agent	[elastic_agent][info] check if endpoint service is installed
12:20:21.760	elastic_agent	endpoint-default	[elastic_agent][info] Spawned new component endpoint-default: Starting: endpoint service runtime
12:20:21.760	elastic_agent	endpoint-default	[elastic_agent][info] Spawned new unit endpoint-default-85821b10-0064-11ee-b676-af36e033a9ae: Starting: endpoint service runtime
12:20:21.760	elastic_agent	endpoint-default	[elastic_agent][info] Spawned new unit endpoint-default: Starting: endpoint service runtime
12:20:22.524	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:22: info: Main.cpp:569 Verifying existing installation
12:20:22.524	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:22: info: InstallLib.cpp:611 Running [C:\Program Files\Elastic\Endpoint\elastic-endpoint.exe] [version --log stdout]
12:20:22.525	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:22: debug: Service.cpp:804 PPL is supported. This process is unprotected. (TrustLevelSid: absent)
12:20:23.192	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: notice: InstallLib.cpp:641 Installed endpoint is a different version; found: [version: 8.16.0, compiled: Thu Nov 7 17:00:00 2024, branch: HEAD, commit: 975e499526d083e8a8c18ac70a23a1cb0f676c20], expected: [version: 8.16.1, compiled: Tue Nov 19 12:00:00 2024, branch: HEAD, commit: 7d50b182b0f0ddc7170095904dc1e341224bb1f4]
12:20:23.196	elastic_agent	[elastic_agent][info] after check if endpoint service is installed, err: 2024-11-25 11:20:23: notice: InstallLib.cpp:641 Installed endpoint is a different version; found: [version: 8.16.0, compiled: Thu Nov 7 17:00:00 2024, branch: HEAD, commit: 975e499526d083e8a8c18ac70a23a1cb0f676c20], expected: [version: 8.16.1, compiled: Tue Nov 19 12:00:00 2024, branch: HEAD, commit: 7d50b182b0f0ddc7170095904dc1e341224bb1f4]: exit status 2
12:20:23.196	elastic_agent	[elastic_agent][info] failed check endpoint service: 2024-11-25 11:20:23: notice: InstallLib.cpp:641 Installed endpoint is a different version; found: [version: 8.16.0, compiled: Thu Nov 7 17:00:00 2024, branch: HEAD, commit: 975e499526d083e8a8c18ac70a23a1cb0f676c20], expected: [version: 8.16.1, compiled: Tue Nov 19 12:00:00 2024, branch: HEAD, commit: 7d50b182b0f0ddc7170095904dc1e341224bb1f4]: exit status 2, try install
12:20:23.390	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Main.cpp:533 Upgrading existing installation (protected)
12:20:23.390	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:1129 Attempting to process existing artifacts manifest
12:20:23.390	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:3360 Attempting to verify signature with global public key
12:20:23.406	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: CryptoLib.cpp:1465 RSA signature verified
12:20:23.406	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:3371 Successfully verified manifest signature
12:20:23.406	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:1551 Attempting to process artifact from manifest:
12:20:23.406	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:615     Identifier:            diagnostic-endpointpe-v4-exceptionlist
12:20:23.406	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:616     Relative URL:          /downloads/endpoint/diagnostic-endpointpe-v4-exceptionlist/0c81d5efede689c16ea30963f9b67534920ebee186bc2a5a2f0fa8b36d55a53a
12:20:23.406	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:617     Decoded SHA256:        b796d2373126a5176706ac6725e08e5e04d3dad14a3db90040bfd8863dc40022
12:20:23.406	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:618     Decoded size:          3924989
12:20:23.406	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:619     Encoded SHA256:        0c81d5efede689c16ea30963f9b67534920ebee186bc2a5a2f0fa8b36d55a53a
12:20:23.406	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:620     Encoded size:          2221390
12:20:23.406	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:622     compression_algorithm: zlib
12:20:23.406	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:624     encryption_algorithm:  RC4-c51f5065b860947796aeff228ebe409c
12:20:23.435	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:754 Artifact diagnostic-endpointpe-v4-exceptionlist successfully initialized
12:20:23.435	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:1551 Attempting to process artifact from manifest:
12:20:23.435	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:615     Identifier:            diagnostic-endpointpe-v4-blocklist
12:20:23.435	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:616     Relative URL:          /downloads/endpoint/diagnostic-endpointpe-v4-blocklist/02ffbee79a29413f8479aa0e3106acd2a5aa5cad366364da1ea30b59125d8667
12:20:23.435	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:617     Decoded SHA256:        338eb3e0d0c2b7efcd291ca270a0a6188d3632a25c7a8dc5d020fdb44293e12f
12:20:23.435	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:618     Decoded size:          464
12:20:23.435	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:619     Encoded SHA256:        02ffbee79a29413f8479aa0e3106acd2a5aa5cad366364da1ea30b59125d8667
12:20:23.435	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:620     Encoded size:          310
12:20:23.435	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:622     compression_algorithm: zlib
12:20:23.435	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:624     encryption_algorithm:  RC4-c51f5065b860947796aeff228ebe409c
12:20:23.435	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:754 Artifact diagnostic-endpointpe-v4-blocklist successfully initialized
12:20:23.435	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:1551 Attempting to process artifact from manifest:
12:20:23.436	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:615     Identifier:            endpointpe-v4-model
12:20:23.436	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:616     Relative URL:          /downloads/endpoint/endpointpe-v4-model/507764964ae8ebf0a478c1fd3358b3e1b5905fb106604255d20e1be91ac38a8f
12:20:23.436	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:617     Decoded SHA256:        8d7dbad963e64c4767596f349c5c70098af1eaf9037e336bc230f89ab591f795
12:20:23.436	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:618     Decoded size:          18134152
12:20:23.436	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:619     Encoded SHA256:        507764964ae8ebf0a478c1fd3358b3e1b5905fb106604255d20e1be91ac38a8f
12:20:23.436	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:620     Encoded size:          6889561
12:20:23.436	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:622     compression_algorithm: zlib
12:20:23.436	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:624     encryption_algorithm:  RC4-c51f5065b860947796aeff228ebe409c
12:20:23.506	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:618     Decoded size:          18124712
12:20:23.506	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:619     Encoded SHA256:        f0cc0b5fc48f9fa8256b96b323fb25a2b42edeb345a10fc79634eea131652516
12:20:23.506	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:620     Encoded size:          6886551
12:20:23.506	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:622     compression_algorithm: zlib
12:20:23.506	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:624     encryption_algorithm:  RC4-c51f5065b860947796aeff228ebe409c
12:20:23.570	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:754 Artifact diagnostic-endpointpe-v4-model successfully initialized
12:20:23.570	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:1551 Attempting to process artifact from manifest:
12:20:23.570	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:615     Identifier:            endpointpe-v4-blocklist
12:20:23.570	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:616     Relative URL:          /downloads/endpoint/endpointpe-v4-blocklist/634a2c993c09fc5a583e3d7f4207ae655d2cdcfb8a31b38e24037514e31f00b2
12:20:23.570	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:617     Decoded SHA256:        5916cc6752c216d55164f8c18b036390c2ceb2474095d738ca1ef8381beda04f
12:20:23.570	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:618     Decoded size:          3924989
12:20:23.570	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:619     Encoded SHA256:        634a2c993c09fc5a583e3d7f4207ae655d2cdcfb8a31b38e24037514e31f00b2
12:20:23.570	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:620     Encoded size:          2227228
12:20:23.570	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:622     compression_algorithm: zlib
12:20:23.570	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:624     encryption_algorithm:  RC4-c51f5065b860947796aeff228ebe409c
12:20:23.610	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:754 Artifact endpointpe-v4-blocklist successfully initialized
12:20:23.610	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:1551 Attempting to process artifact from manifest:
12:20:23.610	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:615     Identifier:            endpointpe-v4-exceptionlist
12:20:23.610	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:616     Relative URL:          /downloads/endpoint/endpointpe-v4-exceptionlist/00b0d058bb418f5a3add854b6cf80160a6da02e5cadc5ca26ccd25ac27e3c056
12:20:23.610	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:617     Decoded SHA256:        f28e867f2a448ee6fc8ca94e8869e9b30061ccd613f877dd2722cd5344700d53
12:20:23.610	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:618     Decoded size:          3968539
12:20:23.610	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:619     Encoded SHA256:        00b0d058bb418f5a3add854b6cf80160a6da02e5cadc5ca26ccd25ac27e3c056
12:20:23.610	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:620     Encoded size:          2206969
12:20:23.610	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:622     compression_algorithm: zlib
12:20:23.610	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:624     encryption_algorithm:  RC4-c51f5065b860947796aeff228ebe409c
12:20:23.618	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:754 Artifact endpointpe-v4-exceptionlist successfully initialized
12:20:23.618	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:1551 Attempting to process artifact from manifest:
12:20:23.618	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:615     Identifier:            diagnostic-configuration-v1
12:20:23.618	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:616     Relative URL:          /downloads/endpoint/diagnostic-configuration-v1/59bd0f6791b3da2a67765d8c899bbd2d546b420d9edb29c1a302c256bf878293
12:20:23.618	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:617     Decoded SHA256:        f70edb954f800cd878b041b7ea610a3ddecef18403c97ca08cef5121044bcc9e
12:20:23.618	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:618     Decoded size:          8468
12:20:23.618	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:619     Encoded SHA256:        59bd0f6791b3da2a67765d8c899bbd2d546b420d9edb29c1a302c256bf878293
12:20:23.618	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:620     Encoded size:          1257
12:20:23.618	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:622     compression_algorithm: zlib
12:20:23.618	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:624     encryption_algorithm:  RC4-c51f5065b860947796aeff228ebe409c
12:20:23.618	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:754 Artifact diagnostic-configuration-v1 successfully initialized
12:20:23.618	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:1551 Attempting to process artifact from manifest:
12:20:23.619	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:615     Identifier:            global-trustlist-windows-v1
12:20:23.619	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:616     Relative URL:          /downloads/endpoint/global-trustlist-windows-v1/ca99c72828da451526f7d3d624de5659c72a2f43434feff64bf28ba64a3d7c6d
12:20:23.619	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:617     Decoded SHA256:        b2eacc24c52b4fb86c129ddf5902750d8eb0db77ac90cfee2822baa72d1efba1
12:20:23.619	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:618     Decoded size:          50505
12:20:23.619	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:619     Encoded SHA256:        ca99c72828da451526f7d3d624de5659c72a2f43434feff64bf28ba64a3d7c6d
12:20:23.619	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:620     Encoded size:          4662
12:20:23.619	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:622     compression_algorithm: zlib
12:20:23.619	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:624     encryption_algorithm:  RC4-c51f5065b860947796aeff228ebe409c
12:20:23.619	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:754 Artifact global-trustlist-windows-v1 successfully initialized
12:20:23.619	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:1551 Attempting to process artifact from manifest:
12:20:23.619	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:615     Identifier:            global-exceptionlist-windows
12:20:23.619	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:616     Relative URL:          /downloads/endpoint/global-exceptionlist-windows/a60a1cd85845c93e971da55b4bb28c45351150c64d471989553f9813641b7c55
12:20:23.619	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:617     Decoded SHA256:        56e4c888d4bb77fb254ec797501a61c444679631cb94e00cf9f159f2798a6806
12:20:23.619	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:618     Decoded size:          755006
12:20:23.619	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:619     Encoded SHA256:        a60a1cd85845c93e971da55b4bb28c45351150c64d471989553f9813641b7c55
12:20:23.619	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:620     Encoded size:          57337
12:20:23.619	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:622     compression_algorithm: zlib
12:20:23.619	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:624     encryption_algorithm:  RC4-c51f5065b860947796aeff228ebe409c
12:20:23.619	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:754 Artifact global-exceptionlist-windows successfully initialized
12:20:23.619	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:1551 Attempting to process artifact from manifest:
12:20:23.619	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:615     Identifier:            global-configuration-v1
12:20:23.619	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:616     Relative URL:          /downloads/endpoint/global-configuration-v1/b5404b735406dd944266ffe778d77d1845d747c0f4413c2fc0983482f0307e27
12:20:23.619	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:617     Decoded SHA256:        a1a28e2940d8b926ddee5f59d0ea44fe1d4f0e46f98e4ef2861a3e288c5758cf
12:20:23.619	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:618     Decoded size:          58023
12:20:23.619	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:619     Encoded SHA256:        b5404b735406dd944266ffe778d77d1845d747c0f4413c2fc0983482f0307e27
12:20:23.619	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:620     Encoded size:          9177
12:20:23.619	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:622     compression_algorithm: zlib
12:20:23.619	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:624     encryption_algorithm:  RC4-c51f5065b860947796aeff228ebe409c
12:20:23.619	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:754 Artifact global-configuration-v1 successfully initialized
12:20:23.619	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:1551 Attempting to process artifact from manifest:
12:20:23.619	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:615     Identifier:            tamper-protection-config-v1
12:20:23.619	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:616     Relative URL:          /downloads/endpoint/tamper-protection-config-v1/bdc4f41615a708fbabaf29e7aca660f71321bf3ef9f2f263079bb372f0de1303
12:20:23.619	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:617     Decoded SHA256:        07f2a166efe84d3b52b6cd8b841f33ffe6eb8e2297cefd4eaa3e50e567b4d30e
12:20:23.619	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:618     Decoded size:          157
12:20:23.619	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:619     Encoded SHA256:        bdc4f41615a708fbabaf29e7aca660f71321bf3ef9f2f263079bb372f0de1303
12:20:23.619	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:620     Encoded size:          126
12:20:23.620	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:622     compression_algorithm: zlib
12:20:23.620	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:624     encryption_algorithm:  RC4-c51f5065b860947796aeff228ebe409c
12:20:23.699	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:754 Artifact tamper-protection-config-v1 successfully initialized
12:20:23.699	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:1551 Attempting to process artifact from manifest:
12:20:23.699	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:615     Identifier:            diagnostic-malware-signature-v1-windows
12:20:23.699	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:616     Relative URL:          /downloads/endpoint/diagnostic-malware-signature-v1-windows/707297d0b581ea32441dabd1fbd28a7fdf3a764b75a0896e85ad9f1558b8d5d7
12:20:23.699	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:617     Decoded SHA256:        b3cf061bffeb4d885c7a8db91bb81848990614d8fef94167a3c8a1464bb51bbc
12:20:23.699	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:618     Decoded size:          834292
12:20:23.699	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:619     Encoded SHA256:        707297d0b581ea32441dabd1fbd28a7fdf3a764b75a0896e85ad9f1558b8d5d7
12:20:23.699	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:620     Encoded size:          215028
12:20:23.699	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:622     compression_algorithm: zlib
12:20:23.699	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:624     encryption_algorithm:  RC4-c51f5065b860947796aeff228ebe409c
12:20:23.699	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:754 Artifact diagnostic-malware-signature-v1-windows successfully initialized
12:20:23.699	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:1551 Attempting to process artifact from manifest:
12:20:23.699	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:615     Identifier:            production-malware-signature-v1-windows
12:20:23.699	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:616     Relative URL:          /downloads/endpoint/production-malware-signature-v1-windows/679bec186a24c0207c670205b7e1f0ccb1ced1226c4bad1be7cb92d28be0927e
12:20:23.699	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:617     Decoded SHA256:        00f347c435a1eb258a8895302cfc3d2f0ec6a91c0b4888b74832b75456aaf857
12:20:23.699	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:618     Decoded size:          793544
12:20:23.699	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:619     Encoded SHA256:        679bec186a24c0207c670205b7e1f0ccb1ced1226c4bad1be7cb92d28be0927e
12:20:23.699	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:620     Encoded size:          205363
12:20:23.699	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:622     compression_algorithm: zlib
12:20:23.699	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:624     encryption_algorithm:  RC4-c51f5065b860947796aeff228ebe409c
12:20:23.699	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:754 Artifact production-malware-signature-v1-windows successfully initialized
12:20:23.699	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:1551 Attempting to process artifact from manifest:
12:20:23.699	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:615     Identifier:            global-eventfilterlist-windows-v1
12:20:23.699	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:616     Relative URL:          /downloads/endpoint/global-eventfilterlist-windows-v1/3976ff6da37f364209a0eedf377c0d88858c894a280922f8d4be59d4a61b498d
12:20:23.699	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:617     Decoded SHA256:        bf4785f2a76763571cd717a3a3f61daaef276546425c55c2cc1a163e47640ce4
12:20:23.699	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:618     Decoded size:          25460
12:20:23.699	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:619     Encoded SHA256:        3976ff6da37f364209a0eedf377c0d88858c894a280922f8d4be59d4a61b498d
12:20:23.699	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:620     Encoded size:          2657
12:20:23.699	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:622     compression_algorithm: zlib
12:20:23.699	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:624     encryption_algorithm:  RC4-c51f5065b860947796aeff228ebe409c
12:20:23.699	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:754 Artifact global-eventfilterlist-windows-v1 successfully initialized
12:20:23.699	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:1551 Attempting to process artifact from manifest:
12:20:23.699	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:615     Identifier:            production-ransomware-v1-windows
12:20:23.699	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:616     Relative URL:          /downloads/endpoint/production-ransomware-v1-windows/ce28867532c8e0c502bda95de6df69e424394f192c85c09f3fbca577293631a4
12:20:23.699	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:617     Decoded SHA256:        86fd40e32b62d190c4ec0aa2420456b6c0746a742c98d160a13540e9d5b6e172
12:20:23.699	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:618     Decoded size:          240153
12:20:23.699	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:619     Encoded SHA256:        ce28867532c8e0c502bda95de6df69e424394f192c85c09f3fbca577293631a4
12:20:23.700	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:620     Encoded size:          34807
12:20:23.700	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:622     compression_algorithm: zlib
12:20:23.700	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:624     encryption_algorithm:  RC4-c51f5065b860947796aeff228ebe409c
12:20:23.700	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:754 Artifact production-ransomware-v1-windows successfully initialized
12:20:23.700	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:1551 Attempting to process artifact from manifest:
12:20:23.700	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:615     Identifier:            diagnostic-ransomware-v1-windows
12:20:23.700	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:616     Relative URL:          /downloads/endpoint/diagnostic-ransomware-v1-windows/2bb3efd0d907b3575039b227d341f6cf132398a764d0adb0d3b94c45152f9574
12:20:23.700	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:617     Decoded SHA256:        2424aa43148dc94bb5a44e99429f218a6d65c8eb2a3f4b53ede7ff9af406d934
12:20:23.700	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:618     Decoded size:          251909
12:20:23.700	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:619     Encoded SHA256:        2bb3efd0d907b3575039b227d341f6cf132398a764d0adb0d3b94c45152f9574
12:20:23.700	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:620     Encoded size:          37262
12:20:23.700	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:622     compression_algorithm: zlib
12:20:23.700	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:624     encryption_algorithm:  RC4-c51f5065b860947796aeff228ebe409c
12:20:23.700	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:754 Artifact diagnostic-ransomware-v1-windows successfully initialized
12:20:23.700	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:1551 Attempting to process artifact from manifest:
12:20:23.700	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:615     Identifier:            diagnostic-rules-windows-v1
12:20:23.700	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:616     Relative URL:          /downloads/endpoint/diagnostic-rules-windows-v1/91213e89f7aeeccdde017e05dd660a81f2316c6f1ce07303e39290ea339a66ce
12:20:23.700	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:617     Decoded SHA256:        987513b0c48c44f09053565f583bcb8f8988b1f75583a3ea67b1eadf8c07494a
12:20:23.700	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:618     Decoded size:          3006760
12:20:23.700	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:619     Encoded SHA256:        91213e89f7aeeccdde017e05dd660a81f2316c6f1ce07303e39290ea339a66ce
12:20:23.700	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:620     Encoded size:          524560
12:20:23.700	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:622     compression_algorithm: zlib
12:20:23.700	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:624     encryption_algorithm:  RC4-c51f5065b860947796aeff228ebe409c
12:20:23.700	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:754 Artifact diagnostic-rules-windows-v1 successfully initialized
12:20:23.700	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:1551 Attempting to process artifact from manifest:
12:20:23.700	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:615     Identifier:            production-rules-windows-v1
12:20:23.700	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:616     Relative URL:          /downloads/endpoint/production-rules-windows-v1/d6c76a469c5137e06a59ed053f604402f2230054de3ba414b9617d499cd91b61
12:20:23.700	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:617     Decoded SHA256:        fcd33d440ff1882a4d5117650e1b8009fc94ea28a8cb0b929baab88c70a273cf
12:20:23.700	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:618     Decoded size:          3194306
12:20:23.700	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:619     Encoded SHA256:        d6c76a469c5137e06a59ed053f604402f2230054de3ba414b9617d499cd91b61
12:20:23.700	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:620     Encoded size:          491348
12:20:23.700	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:622     compression_algorithm: zlib
12:20:23.700	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: debug: Artifacts.cpp:624     encryption_algorithm:  RC4-c51f5065b860947796aeff228ebe409c
12:20:23.700	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:23: info: Artifacts.cpp:754 Artifact production-rules-windows-v1 successfully initialized
12:20:24.781	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:24: info: FilterLib.cpp:2929 Loaded 373 of 373 entries
12:20:24.781	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:24: info: FilterLib.cpp:2929 Loaded 35 of 35 entries
12:20:24.781	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:24: info: Artifacts.cpp:1591 Artifact manifest successfully processed
12:20:24.781	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:24: info: Artifacts.cpp:1129 Attempting to process existing artifacts manifest
12:20:24.781	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:24: debug: Artifacts.cpp:3352 Attempting to verify signature with configured public key
12:20:24.781	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:24: debug: CryptoLib.cpp:1465 RSA signature verified
12:20:24.781	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:24: debug: Artifacts.cpp:3371 Successfully verified manifest signature
12:20:24.974	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:24: info: InstallLib.cpp:509 Attempting uninstall with preserved state for upgrade
12:20:25.271	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:25: info: InstallLib.cpp:313 Running [C:\Program Files\Elastic\Agent\data\elastic-agent-8.16.1-b6da7f\components\previous\elastic-endpoint.exe] [uninstall --keepstate --log stdout]
12:20:25.271	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:25: debug: Service.cpp:804 PPL is supported. This process is unprotected. (TrustLevelSid: absent)
12:20:27.920	elastic_agent	winlog-default	[elastic_agent][info] Spawned new component winlog-default: Starting: spawned pid '5624'
12:20:27.920	elastic_agent	winlog-default[elastic_agent][info] Spawned new unit winlog-default: Starting: spawned pid '5624'
12:20:27.920	elastic_agent	winlog-default	[elastic_agent][info] Spawned new unit winlog-default-winlog-system-85821b11-0064-11ee-b676-af36e033a9ae: Starting: spawned pid '5624'
12:20:27.920	elastic_agent	winlog-default	[elastic_agent][info] Spawned new unit winlog-default-winlog-windows-85821b12-0064-11ee-b676-af36e033a9ae: Starting: spawned pid '5624'
12:20:29.278	elastic_agent	[elastic_agent][info] control checkin v2 protocol has chunking enabled
12:20:29.734	elastic_agent	[elastic_agent][info] control checkin v2 protocol has chunking enabled
12:20:29.734	elastic_agent	winlog-default	[elastic_agent][info] Component state changed winlog-default (STARTING->HEALTHY): Healthy: communicating with pid '5624'
12:20:30.744	elastic_agent	winlog-default	[elastic_agent][info] Unit state changed winlog-default (STARTING->HEALTHY): Healthy
12:20:30.751	elastic_agent	winlog-default	[elastic_agent][info] Unit state changed winlog-default-winlog-system-85821b11-0064-11ee-b676-af36e033a9ae (STARTING->HEALTHY): Healthy
12:20:30.751	elastic_agent	winlog-default	[elastic_agent][info] Unit state changed winlog-default-winlog-windows-85821b12-0064-11ee-b676-af36e033a9ae (STARTING->HEALTHY): Healthy
12:20:31.445	elastic_agent	[elastic_agent][info] Trying to connect to agent
12:20:31.448	elastic_agent	[elastic_agent][info] Connected to agent
12:20:31.450	elastic_agent	[elastic_agent][debug] received state: HEALTHY:Running
12:20:31.450	elastic_agent	[elastic_agent][info] Communicating with PID 20092
12:21:11.427	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:11: info: InstallLib.cpp:348 Upgrade helper succeeded with output 2024-11-25 11:20:27: info: Main.cpp:481 Executing uninstall with persisted state
12:21:11.427	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:27: debug: VaultLib.cpp:207 Vault initialized with existing seed file
12:21:11.427	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:27: debug: VaultLib.cpp:614 Successfully read vault key: config
12:21:11.427	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:27: error: VaultConfig.cpp:97 Failed to load Endpoint config
12:21:11.427	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:27: info: InstallLib.cpp:1155 Skipping uninstall token validation as tamper protection is not enabled.
12:21:11.427	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:27: debug: Service.cpp:804 PPL is supported. This process is unprotected. (TrustLevelSid: absent)
12:21:11.427	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:27: info: Util.cpp:787 Sending service command to facilitate uninstall
12:21:11.427	elastic_agent	[elastic_agent][error] 2024-11-25 11:20:27: info: Util.cpp:814 Service command to facilitate uninstall succeeded
12:21:11.427	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:08: debug: File.cpp:453 Removing [C:\WINDOWS\System32\Drivers\elastic-endpoint-driver.sys]
12:21:11.427	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:08: debug: File.cpp:453 Removing [C:\Program Files\Elastic\Endpoint\SecurityProductInformation.ini]
12:21:11.427	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:08: debug: File.cpp:453 Removing [C:\Program Files\Elastic\Endpoint\elastic-endpoint.exe]
12:21:11.427	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:08: debug: File.cpp:453 Removing [C:\Program Files\Elastic\Endpoint\state\artifacts\global-artifacts\endpointpe-v4-model]
12:21:11.427	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:08: debug: File.cpp:453 Removing [C:\Program Files\Elastic\Endpoint\state\artifacts\global-artifacts\endpointpe-v4-exceptionlist]
12:21:11.427	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:08: debug: File.cpp:453 Removing [C:\Program Files\Elastic\Endpoint\state\artifacts\global-artifacts\endpointpe-v4-blocklist]
12:21:11.427	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:08: debug: File.cpp:453 Removing [C:\Program Files\Elastic\Endpoint\state\artifacts\global-artifacts\production-ransomware-v1-windows]
12:21:11.427	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:08: debug: File.cpp:453 Removing [C:\Program Files\Elastic\Endpoint\state\artifacts\global-artifacts\diagnostic-ransomware-v1-windows]
12:21:11.427	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:08: debug: File.cpp:453 Removing [C:\Program Files\Elastic\Endpoint\state\artifacts\global-artifacts\global-exceptionlist-windows]
12:21:11.427	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:08: debug: File.cpp:453 Removing [C:\Program Files\Elastic\Endpoint\state\artifacts\global-artifacts\global-trustlist-windows-v1]
12:21:11.428	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:08: debug: File.cpp:453 Removing [C:\Program Files\Elastic\Endpoint\state\artifacts\global-artifacts\global-eventfilterlist-windows-v1]
12:21:11.428	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:08: debug: File.cpp:453 Removing [C:\Program Files\Elastic\Endpoint\state\artifacts\global-artifacts\global-configuration-v1]
12:21:11.428	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:08: debug: File.cpp:453 Removing [C:\Program Files\Elastic\Endpoint\state\artifacts\global-artifacts\tamper-protection-config-v1]
12:21:11.428	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:08: debug: File.cpp:453 Removing [C:\Program Files\Elastic\Endpoint\state\artifacts\global-artifacts\diagnostic-rules-windows-v1]
12:21:11.428	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:08: debug: File.cpp:453 Removing [C:\Program Files\Elastic\Endpoint\state\artifacts\global-artifacts\production-rules-windows-v1]
12:21:11.428	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:08: debug: File.cpp:453 Removing [C:\Program Files\Elastic\Endpoint\state\artifacts\global-artifacts\manifest.json]
12:21:11.428	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:08: debug: File.cpp:453 Removing [C:\Program Files\Elastic\Endpoint\state\artifacts\global-artifacts\manifest.sig]
12:21:11.428	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:08: debug: File.cpp:453 Removing [C:\Program Files\Elastic\Endpoint\state\artifacts\global-artifacts\production-malware-signature-v1-windows]
12:21:11.428	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:08: debug: File.cpp:453 Removing [C:\Program Files\Elastic\Endpoint\state\artifacts\global-artifacts\diagnostic-malware-signature-v1-windows]
12:21:11.428	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:08: debug: File.cpp:453 Removing [C:\Program Files\Elastic\Endpoint\state\artifacts\global-artifacts\diagnostic-endpointpe-v4-exceptionlist]
12:21:11.428	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:08: debug: File.cpp:453 Removing [C:\Program Files\Elastic\Endpoint\state\artifacts\global-artifacts\diagnostic-endpointpe-v4-blocklist]
12:21:11.428	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:08: debug: File.cpp:453 Removing [C:\Program Files\Elastic\Endpoint\state\artifacts\global-artifacts\diagnostic-endpointpe-v4-model]
12:21:11.428	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:08: debug: File.cpp:453 Removing [C:\Program Files\Elastic\Endpoint\cache\resources\elastic-endpoint-security.png]
12:21:11.428	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:08: debug: File.cpp:453 Removing [C:\Program Files\Elastic\Endpoint\LICENSE.txt]
12:21:11.428	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:08: debug: File.cpp:453 Removing [C:\Program Files\Elastic\Endpoint\NOTICE.txt]
12:21:11.428	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:09: debug: File.cpp:453 Removing [C:\Program Files\Elastic\Endpoint\elastic-endpoint.exe]
12:21:11.428	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:10: debug: File.cpp:453 Removing [C:\Program Files\Elastic\Endpoint\elastic-endpoint.exe]
12:21:11.428	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:11: debug: File.cpp:453 Removing [C:\Program Files\Elastic\Endpoint\elastic-endpoint.exe]
12:21:11.428	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:11: info: InstallLib.cpp:171 Installing from endpoint-security-resources.zip
12:21:11.428	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:11: info: Internal.cpp:215 Extracting installation artifacts
12:21:11.702	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:11: warning: Service.cpp:82 Service ElasticEndpoint does not exist
12:21:11.702	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:11: warning: Service.cpp:82 Service ElasticEndpointDriver does not exist
12:21:11.702	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:11: debug: File.cpp:453 Removing [C:\WINDOWS\System32\Drivers\ElasticElam.sys]
12:21:11.770	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:11: info: Internal.cpp:408 Writing installation file: C:\WINDOWS\System32\Drivers\elastic-endpoint-driver.sys
12:21:11.805	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:11: info: Internal.cpp:408 Writing installation file: C:\WINDOWS\System32\Drivers\ElasticElam.sys
12:21:11.813	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:11: info: Internal.cpp:408 Writing installation file: C:\Program Files\Elastic\Endpoint\SecurityProductInformation.ini
12:21:11.885	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:11: info: Internal.cpp:408 Writing installation file: C:\Program Files\Elastic\Endpoint\elastic-endpoint.exe
12:21:12.393	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:12: info: Internal.cpp:408 Writing installation file: C:\Program Files\Elastic\Endpoint\state\artifacts\global-artifacts\endpointpe-v4-model
12:21:12.403	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:12: info: Internal.cpp:408 Writing installation file: C:\Program Files\Elastic\Endpoint\state\artifacts\global-artifacts\endpointpe-v4-exceptionlist
12:21:12.412	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:12: info: Internal.cpp:408 Writing installation file: C:\Program Files\Elastic\Endpoint\state\artifacts\global-artifacts\endpointpe-v4-blocklist
12:21:12.420	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:12: info: Internal.cpp:408 Writing installation file: C:\Program Files\Elastic\Endpoint\state\artifacts\global-artifacts\production-ransomware-v1-windows
12:21:12.426	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:12: info: Internal.cpp:408 Writing installation file: C:\Program Files\Elastic\Endpoint\state\artifacts\global-artifacts\diagnostic-ransomware-v1-windows
12:21:12.432	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:12: info: Internal.cpp:408 Writing installation file: C:\Program Files\Elastic\Endpoint\state\artifacts\global-artifacts\global-exceptionlist-windows
12:21:12.437	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:12: info: Internal.cpp:408 Writing installation file: C:\Program Files\Elastic\Endpoint\state\artifacts\global-artifacts\global-trustlist-windows-v1
12:21:12.442	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:12: info: Internal.cpp:408 Writing installation file: C:\Program Files\Elastic\Endpoint\state\artifacts\global-artifacts\global-eventfilterlist-windows-v1
12:21:12.446	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:12: info: Internal.cpp:408 Writing installation file: C:\Program Files\Elastic\Endpoint\state\artifacts\global-artifacts\global-configuration-v1
12:21:12.449	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:12: info: Internal.cpp:408 Writing installation file: C:\Program Files\Elastic\Endpoint\state\artifacts\global-artifacts\tamper-protection-config-v1
12:21:12.458	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:12: info: Internal.cpp:408 Writing installation file: C:\Program Files\Elastic\Endpoint\state\artifacts\global-artifacts\diagnostic-rules-windows-v1
12:21:12.466	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:12: info: Internal.cpp:408 Writing installation file: C:\Program Files\Elastic\Endpoint\state\artifacts\global-artifacts\production-rules-windows-v1
12:21:12.469	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:12: info: Internal.cpp:408 Writing installation file: C:\Program Files\Elastic\Endpoint\state\artifacts\global-artifacts\manifest.json
12:21:12.474	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:12: info: Internal.cpp:408 Writing installation file: C:\Program Files\Elastic\Endpoint\state\artifacts\global-artifacts\manifest.sig
12:21:12.479	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:12: info: Internal.cpp:408 Writing installation file: C:\Program Files\Elastic\Endpoint\state\artifacts\global-artifacts\production-malware-signature-v1-windows
12:21:12.483	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:12: info: Internal.cpp:408 Writing installation file: C:\Program Files\Elastic\Endpoint\state\artifacts\global-artifacts\diagnostic-malware-signature-v1-windows
12:21:12.492	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:12: info: Internal.cpp:408 Writing installation file: C:\Program Files\Elastic\Endpoint\state\artifacts\global-artifacts\diagnostic-endpointpe-v4-exceptionlist
12:21:12.496	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:12: info: Internal.cpp:408 Writing installation file: C:\Program Files\Elastic\Endpoint\state\artifacts\global-artifacts\diagnostic-endpointpe-v4-blocklist
12:21:12.515	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:12: info: Internal.cpp:408 Writing installation file: C:\Program Files\Elastic\Endpoint\state\artifacts\global-artifacts\diagnostic-endpointpe-v4-model
12:21:12.520	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:12: info: Internal.cpp:408 Writing installation file: C:\Program Files\Elastic\Endpoint\cache\resources\elastic-endpoint-security.png
12:21:12.525	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:12: info: Internal.cpp:408 Writing installation file: C:\Program Files\Elastic\Endpoint\LICENSE.txt
12:21:12.531	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:12: info: Internal.cpp:408 Writing installation file: C:\Program Files\Elastic\Endpoint\NOTICE.txt
12:21:12.533	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:12: warning: Service.cpp:82 Service ElasticEndpoint does not exist
12:21:12.533	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:12: warning: Service.cpp:82 Service ElasticEndpointDriver does not exist
12:21:12.533	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:12: warning: Service.cpp:82 Service ElasticELAMDriver does not exist
12:21:12.533	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:12: debug: Util.cpp:1323 Creating service to start "C:\Program Files\Elastic\Endpoint\elastic-endpoint.exe" run
12:21:12.564	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:12: info: Util.cpp:566 Endpoint restart settings [ElasticEndpoint] count=15 delay=15 reset=600
12:21:12.644	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:12: info: Util.cpp:1053 Service Configuration has PPL: enabled
12:21:12.648	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:12: debug: Util.cpp:496 Setting up minifilter registry keys successful.
12:21:13.357	elastic_agent	[elastic_agent][error] 2024-11-25 11:21:13: debug: File.cpp:453 Removing [C:\Program Files\Elastic\Agent\data\elastic-agent-8.16.1-b6da7f\components\previous\elastic-endpoint.exe]
12:21:13.380	elastic_agent	endpoint-default	[elastic_agent][info] Component state changed endpoint-default (STARTING->HEALTHY): Healthy: communicating with endpoint service
12:21:13.388	elastic_agent	[elastic_agent][debug] received state: HEALTHY:Running
12:21:19.936	elastic_agent	[elastic_agent][info] component model updated
12:21:19.936	elastic_agent	[elastic_agent][info] Updating running component model
12:21:19.937	elastic_agent	[elastic_agent][debug] received state: HEALTHY:Running
12:21:19.937	elastic_agent	[elastic_agent][debug] received state: HEALTHY:Running
12:21:30.943	elastic_agent	[elastic_agent][debug] received state: HEALTHY:Running
12:21:47.426	elastic_agent	endpoint-default	[elastic_agent][info] Unit state changed endpoint-default (STARTING->HEALTHY): Applied policy {85821b10-0064-11ee-b676-af36e033a9ae}
12:21:47.426	elastic_agent	endpoint-default	[elastic_agent][info] Unit state changed endpoint-default-85821b10-0064-11ee-b676-af36e033a9ae (STARTING->HEALTHY): Applied policy {85821b10-0064-11ee-b676-af36e033a9ae}
12:21:47.438	elastic_agent	[elastic_agent][debug] received state: HEALTHY:Running
12:21:49.931	elastic_agent	[elastic_agent][info] component model updated
12:21:49.932	elastic_agent	[elastic_agent][debug] received state: HEALTHY:Running
12:21:49.932	elastic_agent	[elastic_agent][info] Updating running component model
12:21:49.935	elastic_agent	[elastic_agent][debug] received state: HEALTHY:Running
12:25:14.076	elastic_agent	[elastic_agent][debug] received state: HEALTHY:Running
12:30:00.433	elastic_agent	[elastic_agent][info] Grace period passed, not watching
12:30:00.434	elastic_agent	[elastic_agent][debug] received state: error: rpc error: code = Canceled desc = context canceled
12:30:00.434	elastic_agent	[elastic_agent][error] Lost connection: failed reading next state: rpc error: code = Canceled desc = context canceled
12:30:00.456	elastic_agent	[elastic_agent][info] Cleaning up upgrade
12:30:00.459	elastic_agent	[elastic_agent][info] updated upgrade details
12:30:20.461	elastic_agent	[elastic_agent][info] Removing previous symlink path
12:30:20.461	elastic_agent	[elastic_agent][info] Removing hashed data directory
12:34:52.670	elastic_agent	[elastic_agent][warn] SSL/TLS verifications disabled.
12:34:52.671	elastic_agent	[elastic_agent][warn] SSL/TLS verifications disabled.
12:34:52.698	elastic_agent	[elastic_agent][info] Setting fallback log level <nil> from policy
12:34:52.751	elastic_agent	[elastic_agent][warn] SSL/TLS verifications disabled.
12:34:52.795	elastic_agent	[elastic_agent][info] Source URI changed from "https://artifacts.elastic.co/downloads/" to "https://artifacts.elastic.co/downloads/"
12:34:52.795	elastic_agent	[elastic_agent][info] Stopping monitoring server
12:34:52.799	elastic_agent	[elastic_agent][info] Stats endpoint (127.0.0.1:6791) finished: accept tcp 127.0.0.1:6791: use of closed network connection
12:34:52.799	elastic_agent	[elastic_agent][info] Starting monitoring server with cfg &config.MonitoringConfig{Enabled:true, MonitorLogs:true, MonitorMetrics:false, MetricsPeriod:"", LogMetrics:true, HTTP:(*config.MonitoringHTTPConfig)(0xc000585710), Namespace:"default", Pprof:(*config.PprofConfig)(nil), MonitorTraces:false, APM:config.APMConfig{Environment:"", APIKey:"", SecretToken:"", Hosts:[]string(nil), GlobalLabels:map[string]string(nil), TLS:config.APMTLS{SkipVerify:false, ServerCertificate:"", ServerCA:""}, SamplingRate:(*float32)(nil)}, Diagnostics:config.Diagnostics{Uploader:config.Uploader{MaxRetries:10, InitDur:1000000000, MaxDur:600000000000}, Limit:config.Limit{Interval:60000000000, Burst:1}}}
12:34:52.799	elastic_agent	[elastic_agent][info] creating monitoring API with cfg api.Config{Enabled:true, Host:"http://localhost:6791", Port:6791, User:"", SecurityDescriptor:"", Timeout:5000000000}
12:34:52.805	elastic_agent	[elastic_agent][info] Starting stats endpoint
12:34:52.809	elastic_agent	[elastic_agent][info] Metrics endpoint listening on: 127.0.0.1:6791 (configured: http://localhost:6791)
12:34:52.840	elastic_agent	[elastic_agent][info] component model updated
12:34:52.840	elastic_agent	[elastic_agent][info] Updating running component model
12:34:52.862	elastic_agent	winlog-default	[elastic_agent][info] Unit state changed winlog-default-winlog-system-85821b11-0064-11ee-b676-af36e033a9ae (HEALTHY->CONFIGURING): Configuring
12:34:52.862	elastic_agent	winlog-default	[elastic_agent][info] Unit state changed winlog-default-winlog-windows-85821b12-0064-11ee-b676-af36e033a9ae (HEALTHY->CONFIGURING): Configuring
12:34:52.885	elastic_agent	[elastic_agent][warn] SSL/TLS verifications disabled.
12:34:53.865	elastic_agent	winlog-default	[elastic_agent][info] Unit state changed winlog-default-winlog-system-85821b11-0064-11ee-b676-af36e033a9ae (CONFIGURING->HEALTHY): Healthy
12:34:53.865	elastic_agent	winlog-default	[elastic_agent][info] Unit state changed winlog-default-winlog-windows-85821b12-0064-11ee-b676-af36e033a9ae (CONFIGURING->HEALTHY): Healthy

[ycombinator]

Hmmmm, not seeing anything about (un)enrollment in the upgrade day logs.

It looks like you have debug level logging turned on for Agent. Do you see a "handlerUnenroll: action '... UNENROLL ...' received" message anywhere in your logs from November 25 through December 2nd?

[jlind23]

@ycombinator @michel-laterman can we make this one of our P0 please? I'd like to make sure that this is not a bug on our end or if it is get it fixed.

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[syk-99]

@ycombinator : there's no occurence of any handlerUnenroll: action or '... UNENROLL ... in any agent-log.

On day of host-reboot (elastic-agent service shutdown) - Nov. 30th - we see this audit unenroll successful event in fleet-server log:

> {
>   "_index": ".ds-logs-elastic_agent.fleet_server-default-2024.11.29-000099",
>   "_id": "mHZ9fpMBDofmwoj9oymf",
>   "_version": 1,
>   "_score": 0,
>   "_source": {
>     "container": {
>       "id": "elastic-agent-8.16.1-b6da7f"
>     },
>     "agent": {
>       "name": "srv-elastic01",
>       "id": "a6cba17f-70ba-49a7-ba5a-68e25bbc2e38",
>       "type": "filebeat",
>       "ephemeral_id": "3af151e0-5443-41d4-9800-2aa4ef8889b7",
>       "version": "8.16.1"
>     },
>     "service.name": "fleet-server",
>     "log": {
>       "file": {
>         "inode": "3026003",
>         "path": "/opt/Elastic/Agent/data/elastic-agent-8.16.1-b6da7f/logs/elastic-agent-20241130-6.ndjson",
>         "device_id": "64768"
>       },
>       "offset": 9850948,
>       "source": "fleet-server-default"
>     },
>     "http.request.id": "2982e60d-6c13-4c66-b3bf-0dd5ae482db3",
>     "elastic_agent": {
>       "id": "a6cba17f-70ba-49a7-ba5a-68e25bbc2e38",
>       "version": "8.16.1",
>       "snapshot": false
>     },
>     "fleet.access.apikey.id": "pVC__ZAB8xxxxxxxxxx",
>     "message": "audit unenroll successful",
>     "service.type": "fleet-server",
>     "server.address": "",
>     "input": {
>       "type": "filestream"
>     },
>     "component": {
>       "binary": "fleet-server",
>       "id": "fleet-server-default",
>       "type": "fleet-server",
>       "dataset": "elastic_agent.fleet_server"
>     },
>     "@timestamp": "2024-11-30T19:12:59.992Z",
>     "ecs": {
>       "version": "8.0.0"
>     },
>     "data_stream": {
>       "namespace": "default",
>       "type": "logs",
>       "dataset": "elastic_agent.fleet_server"
>     },
>     "host": {
>       "hostname": "srv-elastic01",
>       "os": {
>         "kernel": "5.15.0-126-generic",
>         "codename": "jammy",
>         "name": "Ubuntu",
>         "type": "linux",
>         "family": "debian",
>         "version": "22.04.5 LTS (Jammy Jellyfish)",
>         "platform": "ubuntu"
>       },
>       "containerized": false,
>       "ip": [
>         "x.x.x.x",
>         "x.x.x.x",
>         "172.17.0.1",
>         "172.19.0.1",
>         "172.18.0.1",
>         "172.20.0.1"
>       ],
>       "name": "srv-elastic01",
>       "id": "5d58ef4f7e1e4e96a564a75d011eea07",
>       "mac": [
>         "00-50-56-88-28-77",
>         "02-42-1A-B8-42-BB",
>         "02-42-41-08-1F-D0",
>         "02-42-47-77-45-CD",
>         "02-42-84-38-AA-9A",
>         "66-EF-56-99-E7-FB",
>         "82-99-79-C1-F2-90",
>         "BE-5D-A3-75-46-78",
>         "E2-1E-FA-71-65-4C"
>       ],
>       "architecture": "x86_64"
>     },
>     "log.level": "info",
>     "event": {
>       "agent_id_status": "verified",
>       "ingested": "2024-11-30T19:13:03Z",
>       "dataset": "elastic_agent.fleet_server"
>     },
>     "fleet.agent.id": "0f9cfb74-8f2f-408c-918e-6d16301e1370"
>   },
>   "fields": {
>     "elastic_agent.version": [
>       "8.16.1"
>     ],
>     "component.binary": [
>       "fleet-server"
>     ],
>     "host.os.name.text": [
>       "Ubuntu"
>     ],
>     "http.request.id": [
>       "2982e60d-6c13-4c66-b3bf-0dd5ae482db3"
>     ],
>     "host.hostname": [
>       "srv-elastic01"
>     ],
>     "host.mac": [
>       "00-50-56-88-28-77",
>       "02-42-1A-B8-42-BB",
>       "02-42-41-08-1F-D0",
>       "02-42-47-77-45-CD",
>       "02-42-84-38-AA-9A",
>       "66-EF-56-99-E7-FB",
>       "82-99-79-C1-F2-90",
>       "BE-5D-A3-75-46-78",
>       "E2-1E-FA-71-65-4C"
>     ],
>     "container.id": [
>       "elastic-agent-8.16.1-b6da7f"
>     ],
>     "service.type": [
>       "fleet-server"
>     ],
>     "server.address": [
>       ""
>     ],
>     "component.id": [
>       "fleet-server-default"
>     ],
>     "host.os.version": [
>       "22.04.5 LTS (Jammy Jellyfish)"
>     ],
>     "host.os.name": [
>       "Ubuntu"
>     ],
>     "log.level": [
>       "info"
>     ],
>     "agent.name": [
>       "srv-elastic01"
>     ],
>     "host.name": [
>       "srv-elastic01"
>     ],
>     "event.agent_id_status": [
>       "verified"
>     ],
>     "host.os.type": [
>       "linux"
>     ],
>     "log.source": [
>       "fleet-server-default"
>     ],
>     "input.type": [
>       "filestream"
>     ],
>     "fleet.access.apikey.id": [
>       "pVC__ZAB8xxxxxxxxxx"
>     ],
>     "log.offset": [
>       9850948
>     ],
>     "data_stream.type": [
>       "logs"
>     ],
>     "host.architecture": [
>       "x86_64"
>     ],
>     "agent.id": [
>       "a6cba17f-70ba-49a7-ba5a-68e25bbc2e38"
>     ],
>     "ecs.version": [
>       "8.0.0"
>     ],
>     "host.containerized": [
>       false
>     ],
>     "agent.version": [
>       "8.16.1"
>     ],
>     "host.os.family": [
>       "debian"
>     ],
>     "host.ip": [
>       "x.x.x.x",
>       "x.x.x.x",
>       "172.17.0.1",
>       "172.19.0.1",
>       "172.18.0.1",
>       "172.20.0.1"
>     ],
>     "agent.type": [
>       "filebeat"
>     ],
>     "host.os.kernel": [
>       "5.15.0-126-generic"
>     ],
>     "log.file.device_id": [
>       "64768"
>     ],
>     "component.dataset": [
>       "elastic_agent.fleet_server"
>     ],
>     "elastic_agent.snapshot": [
>       false
>     ],
>     "fleet.agent.id": [
>       "0f9cfb74-8f2f-408c-918e-6d16301e1370"
>     ],
>     "host.id": [
>       "5d58ef4f7e1e4e96a564a75d011eea07"
>     ],
>     "service.name": [
>       "fleet-server"
>     ],
>     "elastic_agent.id": [
>       "a6cba17f-70ba-49a7-ba5a-68e25bbc2e38"
>     ],
>     "data_stream.namespace": [
>       "default"
>     ],
>     "host.os.codename": [
>       "jammy"
>     ],
>     "message": [
>       "audit unenroll successful"
>     ],
>     "component.type": [
>       "fleet-server"
>     ],
>     "event.ingested": [
>       "2024-11-30T19:13:03.000Z"
>     ],
>     "@timestamp": [
>       "2024-11-30T19:12:59.992Z"
>     ],
>     "host.os.platform": [
>       "ubuntu"
>     ],
>     "log.file.inode": [
>       "3026003"
>     ],
>     "data_stream.dataset": [
>       "elastic_agent.fleet_server"
>     ],
>     "log.file.path": [
>       "/opt/Elastic/Agent/data/elastic-agent-8.16.1-b6da7f/logs/elastic-agent-20241130-6.ndjson"
>     ],
>     "agent.ephemeral_id": [
>       "3af151e0-5443-41d4-9800-2aa4ef8889b7"
>     ],
>     "event.dataset": [
>       "elastic_agent.fleet_server"
>     ]
>   }
> }

Dec. 2nd - first start of agent after host reboot:
`

09:25:50.708 elastic_agent [elastic_agent][info] Loaded update marker &{Version:8.16.1 Hash:b6da7f VersionedHome:data\elastic-agent-8.16.1-b6da7f UpdatedOn:2024-11-25 12:20:00.3690588 +0100 CET PrevVersion:8.16.0 PrevHash:3f07f2 PrevVersionedHome:data\elastic-agent-8.16.0-3f07f2 Acked:false Action:id: f0d5d0c4-b283-419e-b826-a8e830f755cc, type: UPGRADE Details:}

`

[syk-99]

In case it's relevant: some windows-hosts with agents in that strange state (active and unenrolled) seem to be unable to start the agent-service at boot time anymore.
When starting the elastic-agent service manually, the error in the screenshot below is shown in kibana->fleet->agents:

I cannot find any log entry anywhere matching this error (maybe in kibana.log - I didn't check that one ...)

[michel-laterman]

I think that upgrading from 8.16.0 -> 8.16.1 calls uninstall on 8.16.0, which makes an API call to audit/unenroll. However this call should be informative, it does not revoke the API keys, and a subsequent checkin from an agent removes these attributes.

Can you provide a diagnostics bundle of a failing agent?

[michel-laterman]

Doing a quick search of the codebase, i don't see a call to Uninstall anywhere within the upgrade handler, so my theory may be incorrect

[syk-99]

elastic-agent-diagnostics-2024-12-11T15-11-45Z-00.zip

this should be from the same agent all the log-snippets above originated...