fix: prevent the MIG reconfigure from hanging on systemd-less hosts#431
Closed
lexfrei wants to merge 2 commits into
Closed
fix: prevent the MIG reconfigure from hanging on systemd-less hosts#431lexfrei wants to merge 2 commits into
lexfrei wants to merge 2 commits into
Conversation
Constructing the reconfigurer unconditionally opened a systemd D-Bus connection, even for reconfigurations that never touch host systemd (WITH_SHUTDOWN_HOST_GPU_CLIENTS unset and no host state file to persist). On hosts without systemd - such as Talos Linux - the system bus socket may exist (bind-mounted from the host) but nothing answers it, so the D-Bus auth handshake blocks forever and the MIG geometry change is never applied, with no error logged. Establish the connection lazily instead: build it the first time a code path genuinely needs systemd (persisting the host state file, or stopping/restarting host GPU client services) and cache it. When none of those run, the socket is never dialed, so the reconfigure - including the MIG geometry change, which is applied by the nvidia-mig-parted binary and not via systemd - completes on systemd-less hosts. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>
go-systemd's connection setup performs a blocking read during the D-Bus auth handshake. When the system bus socket exists but no daemon answers it - as on a systemd-less host where /run/dbus/system_bus_socket is bind-mounted from the host - that read never returns and the caller hangs indefinitely. Bound the connection setup with a timeout. The dial runs in a goroutine; if it does not complete in time, the connection context is canceled, go-systemd's watcher closes the socket to unblock the stuck read, and NewManager returns a clear error instead of blocking forever. This is a backstop for the paths that do require systemd, so a misconfigured or absent host daemon surfaces an actionable error rather than a silent hang. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>
Contributor
Author
|
Closing in favour of #382, which already carries both of my commits ( @rajathagasthya thanks for the pointer. I've left one piece of review feedback over on #382 rather than here. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
On hosts without systemd — notably Talos Linux —
k8s-mig-managerv0.14.0 hangs indefinitely when thenvidia.com/mig.configlabel changes, and never applies the new MIG geometry.The Go-native reconfigure path opens a systemd D-Bus connection unconditionally when it is constructed. On a systemd-less host the system bus socket (
/run/dbus/system_bus_socket) still exists — it is bind-mounted from the host — but nothing answers it, so the D-Bus auth handshake performs a blocking read that never returns. This happens even whenWITH_SHUTDOWN_HOST_GPU_CLIENTS=false, i.e. when no host-systemd work is actually required.Why it matters
Most of the reconfigure flow does not need systemd at all — the MIG geometry change is applied by the
nvidia-mig-partedbinary, not through systemd. systemd is only needed to persist the host mig-manager state file (a daemon-reload) and to stop/restart host GPU client services. A host that never touches those paths should never need a D-Bus connection.Changes
Testing
go build ./...,go vet,gofmt, andgolangci-lintpass on the changed packages.Addresses #356