Mitigate and instrument the utility-VM boot-connect timeout - #41165
Mitigate and instrument the utility-VM boot-connect timeout#41165yeelam-gordon wants to merge 1 commit into
Conversation
A healthy-but-slow guest boot (loaded host, cold first boot) trips the KernelBootTimeout watchdog and surfaces as CreateVm/0x800705b4. Raise the default KernelBootTimeout from 30s to 90s for headroom, and wrap the notify-channel accept in its own SlowOperationWatcher so a slow or hung boot is attributed to the correct phase. Diagnostic-only; no control-flow change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e76db014-5ee2-46f5-b078-42ad74b5197a
There was a problem hiding this comment.
Pull request overview
This PR adjusts the utility-VM boot/connect behavior in the Windows service to reduce intermittent boot-connect timeouts under host contention, and adds telemetry to better attribute where boot is stalling.
Changes:
- Increase the default
wsl2.kernelBootTimeoutfrom 30s to 90s. - Add a
SlowOperationWatcherscope around the notify-channel accept to attribute slow boots to the correct phase.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| src/windows/service/exe/WslCoreVm.cpp | Adds a SlowOperationWatcher around the guest notify-channel accept during VM initialization. |
| src/windows/common/WslCoreConfig.h | Raises the default KernelBootTimeout from 30s to 90s. |
| std::filesystem::path DebugConsoleLogFile; | ||
| std::wstring VmSwitch; | ||
| int KernelBootTimeout = (30 * 1000); | ||
| int KernelBootTimeout = (90 * 1000); | ||
| int DistributionStartTimeout = (60 * 1000); | ||
| int MountDeviceTimeout = (5 * 1000); |
There was a problem hiding this comment.
I'm surprised that we find machines that take more than 30 seconds to connect the hvsocket.
The bug mentioned in the description points to a test failure, but in the test we increase that timeout to 120 :
Lines 2096 to 2097 in 404453c
(Default is here)
So if we timed out in the tests, I'd expect that the issue is somewhere else. Was there anything in dmesg in that test failure ?
Why (the symptom)
On heavily loaded machines, utility-VM creation intermittently fails with
.../CreateVm/0x800705b4(ERROR_TIMEOUT), which blocks install. Seen in automated test runs. Tracked as 62543002.Root cause
During VM
Initialize, the service waits for the guest to connect back over hvsocket withinKernelBootTimeout(default 30s). This is a give-up watchdog, not a boot budget — a healthy boot connects in ~1-3s. Under host CPU contention a healthy-but-slow boot exceeds 30s and trips it. Two gaps:The fix (two small changes)
KernelBootTimeout30s -> 90s. Still bounds a genuinely stuck VM; just gives a slow-but-alive boot room to finish. Overridable viawsl2.kernelBootTimeout.SlowOperationWatcher{"WaitForNotifyChannelConnect"}(mirrors the adjacent mini_init one). Diagnostic only — no control-flow change.Note: this is not fixed by #40519. That protects processes inside the guest after it connects; this failure is the host starving the VM's vCPUs before it connects — different phase, different contention. The real cure (a host-side CPU reservation for the booting VM) is a follow-up, gated on the telemetry above confirming CPU is the starved resource.
Testing
SlowOperationWatcherunit tests: 6/6 pass (emits realelapsedMs; hang backstop emits once).wslservicebuild-verified with both changes.kernelBootTimeout=1yields the exact.../CreateVm/0x800705b4.@shuaiyuanxx — could you review? You own the SlowOperationWatcher / boot path. (Bug 62543002 assigned to you.)