You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Filed as issue #1205 and PR #1286, but since CONTRIBUTING.md points here for "is anyone looking at this" questions, trying this board too.
The bug: on affected systems the machine hard-hangs on a deterministic suspend/resume cycle (6th cycle every boot, 7/7 boots, both deep and s2idle) with a soft-lockup watchdog trace inside nvDIFRPrefetchSurfaces. Confirmed independently by five other users on #1205 across Blackwell mobile, Blackwell desktop, and Ada mobile — so it's not a one-machine quirk.
Root cause:PrefetchSingleSurface() in nvkms-difr.c kicks off a CE copy and on a semaphore-wait timeout returns FAIL_CE_HW_ERROR — but the GPFIFO entries it already queued stay stuck on a channel the CE never drained. Nothing reclaims them. The ring is tiny (16 entries, 2 consumed at alloc, 2 per kickoff), so a handful of faults exhausts it. The next kickoff then blocks in nvWriteGpEntry() (nvidia-push.c), whose wait loop has no deadline, no yield, and its only exit condition (nvPushCheckChannelError()) never fires for a channel that stalls without faulting — so the kernel thread spins forever and the watchdog escalates to a hang.
Fix in #1286 (two commits, both isolated to the affected files):
Bound the nvWriteGpEntry() wait with the same timeout idiom IdleChannel() already uses, plus a yield between polls.
Reset the prefetch channel on FAIL_CE_HW_ERROR so a fault doesn't leak ring space.
I've soaked both patches past the point where my machine used to hang every time (no failures since), and the PR includes a build/install script for anyone else who wants to test it independently.
Given the repo doesn't merge community PRs directly (understood — internal porting, not a GitHub review queue), I'm mainly hoping someone can either confirm the driver team has this on their radar for the internal tree, or point me at a better channel to get it in front of them. Happy to answer questions or provide more data — full per-boot logs and the cycle-count analysis are on #1205.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
Filed as issue #1205 and PR #1286, but since CONTRIBUTING.md points here for "is anyone looking at this" questions, trying this board too.
The bug: on affected systems the machine hard-hangs on a deterministic suspend/resume cycle (6th cycle every boot, 7/7 boots, both
deepands2idle) with a soft-lockup watchdog trace insidenvDIFRPrefetchSurfaces. Confirmed independently by five other users on #1205 across Blackwell mobile, Blackwell desktop, and Ada mobile — so it's not a one-machine quirk.Root cause:
PrefetchSingleSurface()innvkms-difr.ckicks off a CE copy and on a semaphore-wait timeout returnsFAIL_CE_HW_ERROR— but the GPFIFO entries it already queued stay stuck on a channel the CE never drained. Nothing reclaims them. The ring is tiny (16 entries, 2 consumed at alloc, 2 per kickoff), so a handful of faults exhausts it. The next kickoff then blocks innvWriteGpEntry()(nvidia-push.c), whose wait loop has no deadline, no yield, and its only exit condition (nvPushCheckChannelError()) never fires for a channel that stalls without faulting — so the kernel thread spins forever and the watchdog escalates to a hang.Fix in #1286 (two commits, both isolated to the affected files):
nvWriteGpEntry()wait with the same timeout idiomIdleChannel()already uses, plus a yield between polls.FAIL_CE_HW_ERRORso a fault doesn't leak ring space.I've soaked both patches past the point where my machine used to hang every time (no failures since), and the PR includes a build/install script for anyone else who wants to test it independently.
Given the repo doesn't merge community PRs directly (understood — internal porting, not a GitHub review queue), I'm mainly hoping someone can either confirm the driver team has this on their radar for the internal tree, or point me at a better channel to get it in front of them. Happy to answer questions or provide more data — full per-boot logs and the cycle-count analysis are on #1205.
All reactions