Wait for MPS daemon readiness before advertising shared resources

[jonathan-meiri]

Summary

The device plugin can advertise MPS-shared resources to kubelet before the MPS daemon has finished configuring per-device memory limits and thread percentages. A pod scheduled in that window runs against MPS with the daemon defaults (no pinned memory limit, 100% threads) instead of the configured limits — silently bypassing the intended isolation.

This gates readiness on the MPS daemon's .ready file, closing the two existing TODOs in waitForDaemon.

Contributed by @Meiri28 on behalf of @runatom-ai.

The race

internal/plugin/server.go:Start() calls plugin.mps.waitForDaemon() before Serve() + Register(). Previously waitForDaemon ran a single AssertHealthy():

// TODO: Check the .ready file here.
// TODO: Have some retry strategy here.
if err := m.daemon.AssertHealthy(); err != nil { ... }

AssertHealthy() issues get_default_active_thread_percentage, which only proves the control pipe is responsive. In Daemon.Start() the ordering is:

1. mpsControlBin -d starts → pipe becomes responsive (AssertHealthy passes here)
2. per-device pinned memory limits applied
3. active thread percentage applied
4. .ready file created (after all daemons' Start() return)

If the single check lands between (1) and (4), the plugin registers as ready while the configured MPS limits are not yet in place.

The MPS control daemon already creates a node-global .ready file only after full initialization, but nothing consumed it — the two TODOs noted exactly this gap. Both components share the same /mps hostPath mount, so the file is visible to the device plugin.

Changes

• mps/root.go: add Root.ReadyFilePath() so the marker path has a single definition.
• mps-control-daemon/main.go: create/remove .ready via ReadyFilePath() instead of the hardcoded /mps/.ready, keeping writer and reader in sync.
• mps/daemon.go: add Daemon.Ready(), reporting whether the .ready file exists.
• internal/plugin/mps.go: rewrite waitForDaemon to poll checkDaemonReady (Ready() and AssertHealthy()) every 5s up to a 5m bound. On timeout the caller fails and is retried by the plugin manager, so the bound is per-attempt.

Test plan

• go test ./... — full suite passes; new TestReadyFilePath and TestDaemonReady cover the readiness primitive (readiness reflects .ready file existence).
• make build, make check-modules, gofmt — clean.

Commits are DCO-signed.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[myeolenv]

Thanks for the contribution. We will review this soon.