Windows: agent hangs at system shutdown when running as a Windows service (SIGTERM path not terminated) #2214

Describe the bug

When the CloudWatch Agent runs as a Windows service on hosts where the collector is spawned as a console-attached child of start-amazon-cloudwatch-agent.exe, an OS shutdown can hang for the duration of WaitToKillServiceTimeout (up to several minutes when the value is raised from the default). The system event log records Event ID 6008 (unexpected shutdown) and Kernel-Power 41. On EC2 the hang exceeds the platform's ~4-minute hard-stop and the instance is force-powered-off.

Root cause (short form): Windows shutdown delivers two independent notifications:
(1) csrss / CsrShutdownProcesses sends CTRL_SHUTDOWN_EVENT to console-attached processes and waits for them to exit (the Go runtime maps this to SIGTERM), then (2) the SCM's ScmAutoShutdownServices delivers SERVICE_CONTROL_SHUTDOWN to registered services.
The collector is spawned as a child of start-amazon-cloudwatch-agent.exe. csrss delivers CTRL_SHUTDOWN to the child, but the SCM routes SERVICE_CONTROL_SHUTDOWN to the SCM-registered launcher process, not the child. Inside kardianos/service v1.2.1's non-interactive windowsService.Run(), the code waits solely on the SCM control channel and does not watch SIGTERM.

Result:

• otelcol receives SIGTERM via signal.Notify, does its graceful shutdown, and logs Shutdown complete.
• cmd.Execute() returns, runAgent() returns, reloadLoop returns, (*program).run() returns.
• The outer svc.Run() in kardianos is still blocked waiting for SERVICE_CONTROL_SHUTDOWN that never arrives; main never returns; the Go runtime's ctrlHandler is parked in block(); the process does not exit.
• csrss is still waiting for the process to exit -> the whole shutdown deadlocks until the platform's hard-timeout expires.

sc stop AmazonCloudWatchAgent (SCM STOP via ControlService) is unaffected because it goes through the service control pipe and prg.Stop handles it normally.

Steps to reproduce

1. Install the CloudWatch Agent MSI on Windows Server. Reproduced against 1.300067.0b1404 and confirmed present on the current main (kardianos/service v1.2.1 is still pinned).
2. Configure any minimal JSON that starts the agent as a Windows service (attached below).
3. Set HKLM\SYSTEM\CurrentControlSet\Control\WaitToKillServiceTimeout to a value large enough to exceed the platform hard-stop, e.g. 300000 (5 minutes). The default 20000 masks the deadlock by force-killing the process; the collector is still hung, but Windows terminates it before the 4-minute EC2 hard-stop hits.
4. Reboot to apply WaitToKillServiceTimeout, then trigger an OS shutdown (shutdown /s /t 0 or, on EC2, StopInstances).
5. Observe:
   • Shutdown duration approximately equals WaitToKillServiceTimeout (or the platform hard-stop,
     whichever is smaller).
   • System event log: 1074 (shutdown initiated), no 6006, 41 (Kernel-Power), 6008
     (previous system shutdown was unexpected).
   • amazon-cloudwatch-agent.log ends with Shutdown complete. while the process is still alive.

What did you expect to see?

An OS shutdown that completes cleanly (System event 6006, no 6008, no 41) whether the agent is stopped via SCM STOP or via CTRL_SHUTDOWN.

What did you see instead?

OS shutdown hangs ~290 s (with WaitToKillServiceTimeout=300000) and results in 6008 + Kernel-Power 41. Agent log stops updating after Shutdown complete. while the process remains alive until Windows force-kills it.

What version did you use?

Reproduced against 1.300067.0b1404. The relevant code path exists on main (cmd/amazon-cloudwatch-agent/amazon-cloudwatch-agent.go + github.com/kardianos/service v1.2.1).

What config did you use?

Minimal Windows Event Log collector config (identical failure with any config that starts the agent as a service):

{
  "logs": {
    "logs_collected": {
      "windows_events": {
        "collect_list": [
          {
            "event_name": "System",
            "event_levels": ["INFORMATION","WARNING","ERROR","CRITICAL"],
            "log_group_name": "repro/System",
            "log_stream_name": "{instance_id}"
          }
        ]
      }
    }
  }
}

Environment

OS: Windows Server 2022 Datacenter (JP; EN reproduces identically), on EC2 m5.xlarge in ap-northeast-1. Reproduced on stock Windows Server 2025 as well when WaitToKillServiceTimeout is set high enough to expose the same deadlock; the OS version is not the differentiator.