feat(disk, helm, docker): host-namespace disk inspection for containerized gpud (#1253)

## Summary

Make the GPUd disk component report the **node's real disks** when GPUd
runs inside a container (e.g. a privileged DaemonSet), and replace the
prior overlay-root workaround with a proper, configurable fix.

## Problem

Inside a container, the disk component's host tools run in the
**container's** mount namespace, so they describe the container's
overlay rootfs instead of the node:

- `findmnt --target / --json --df` returns nothing (overlay is a pseudo
filesystem that `--df` filters out) and **exits 1**, failing the disk
check.
- `lsblk` misses the node's data disks and reports empty fstypes.
- `statfs(path)` falls through to the overlay — e.g. reporting **124G
for a 2.8T** data disk.

## Change

Add configurable command overrides so the disk component can run its
host tools in the **host mount namespace** (via `nsenter`), mirroring
the existing `--reboot-commands` pattern.

**`gpud run` flags (new):** `--findmnt-commands`, `--lsblk-commands`,
`--blockdev-usage-commands`
Plumbed `cmd/gpud` → `pkg/config` → `pkg/server` → `GPUdInstance` → disk
component → `pkg/disk`.

**`pkg/disk`:**
- `FindMntWithCommand` and an `lsblk` command override (run the
configured invocation prefix; GPUd appends the flags it controls).
- A `df`-based partitions/usage path (`df -T -B1 -P`) that replaces
gopsutil enumeration + `statfs` when the override is set.

**Helm chart:**
- New `gpud.findmntCommands` / `gpud.lsblkCommands` /
`gpud.blockdevUsageCommands` values, defaulted to `nsenter --target 1
--mount -- {findmnt,lsblk,df}` and wired into the DaemonSet startup (env
+ flags), gated like `rebootCommands`.
- Document DaemonSet **reboot** and **disk inspection** configuration in
the chart README; bump chart to **v0.12.2**; note the BYOK machine-id
node label.

**Docker:** keep a runtime-stage guard verifying `findmnt` is present in
the image.

## Backward compatibility (strict invariant)

When a flag is **unset** (the default), every code path falls through to
the **exact legacy behavior** — locate `findmnt`/`lsblk` on `PATH` and
run them directly, enumerate via gopsutil, and measure usage via
`statfs`. Behavior on bare-metal / host GPUd is unchanged. Helm renders
**no** disk env/flags when a value is empty.

## Revert of the `--all` band-aid

This branch previously added `findmnt --df --all` to force the container
overlay into `--df` output. With the host-namespace fix, `findmnt
--target X --json --df` (matching `main`) works on real filesystems, and
`--all` only masked the container-overlay case with **misleading data**
(`overlay` instead of `/dev/sda1 ext4`). The default `findmnt` command
is restored to `main`'s.

## Tests

- `pkg/disk` unit tests: `df` output parser, partition filtering, option
setters, command builders, and the **empty-flag == legacy-path**
invariant.
- Validated on a live BYOK (AKS) cluster: `nsenter`-wrapped
`findmnt`/`lsblk`/`df` report the node's real ext4 root and 2.8T data
disk; `df -T -B1 -P` column order matches the parser.
- `go build` / `go vet` clean; touched-package tests pass; `helm lint`
and `helm template` (default + empty-override) verified.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Signed-off-by: Gyuho Lee <gyuhol@nvidia.com>

Author: gyuho

.gitignore

@@ -42,3 +42,6 @@ custom-gcl
 
 /gpud
 
+
+# Private helm values overlay (nvcr.io image + pull secret) -- internal only
+deployments/helm/gpud/values-nvcr.yaml

Dockerfile

@@ -137,6 +137,17 @@ RUN echo "# APT Package Sources" > /apt-sources/APT_SOURCES.txt && \
 FROM nvidia/cuda:${CUDA_VERSION}-runtime-${OS_NAME}${OS_VERSION}
 WORKDIR /
 
+# OCI image metadata. Mirrors the Helm chart's Chart.yaml/README so registries
+# (e.g. nvcr.io) surface the same description, source, and license for the image.
+# ref. https://github.com/opencontainers/image-spec/blob/main/annotations.md
+LABEL org.opencontainers.image.title="gpud" \
+      org.opencontainers.image.description="GPUd automates monitoring, diagnostics, and issue identification for GPUs" \
+      org.opencontainers.image.source="https://github.com/leptonai/gpud" \
+      org.opencontainers.image.url="https://github.com/leptonai/gpud" \
+      org.opencontainers.image.documentation="https://github.com/leptonai/gpud" \
+      org.opencontainers.image.vendor="leptonai" \
+      org.opencontainers.image.licenses="Apache-2.0"
+
 # Install required runtime dependencies not included in the NVIDIA CUDA runtime image.
 # NOTE: gnupg is installed temporarily for Docker GPG key verification, then purged
 # to address CVE-2025-68973 (out-of-bounds write in GnuPG armor_filter before 2.4.9)
@@ -161,6 +172,9 @@ RUN apt-get update && \
   # Remove gnupg and related packages to address CVE-2025-68973
   # These are only needed for GPG key verification during build, not at runtime
   apt-get purge -y --auto-remove gnupg gnupg-l10n gnupg-utils gpg gpg-agent gpg-wks-client gpg-wks-server gpgconf gpgsm dirmngr && \
+  # util-linux provides findmnt, which the disk component shells out to.
+  command -v findmnt >/dev/null && \
+  findmnt --version >/dev/null && \
   rm -rf /var/lib/apt/lists/*
 
 # Copy the gpud binary

cmd/gpud/command/command.go

@@ -204,6 +204,26 @@ sudo rm /etc/systemd/system/gpud.service
 					Usage: "bash script to run when the control plane sends a reboot request (leave empty to use the built-in sudo reboot behavior)",
 					Value: "",
 				},
+				cli.StringFlag{
+					Name:  "findmnt-commands",
+					Usage: "command prefix used to invoke findmnt for the disk component (e.g. 'nsenter --target 1 --mount -- findmnt' to read host mounts from inside a container); leave empty to locate and run findmnt directly",
+					Value: "",
+				},
+				cli.StringFlag{
+					Name:  "lsblk-commands",
+					Usage: "command prefix used to invoke lsblk for the disk component (e.g. 'nsenter --target 1 --mount -- lsblk' to read host block devices from inside a container); leave empty to locate and run lsblk directly",
+					Value: "",
+				},
+				cli.StringFlag{
+					Name:  "blockdev-usage-commands",
+					Usage: "command prefix used to collect block device usage for the disk component (e.g. 'nsenter --target 1 --mount -- df' to read host disk usage from inside a container); leave empty to use the built-in gopsutil/statfs behavior",
+					Value: "",
+				},
+				cli.StringFlag{
+					Name:  "containerd-service-active-commands",
+					Usage: "command used to check whether the containerd service is active (e.g. 'nsenter --target 1 --mount -- systemctl is-active containerd' to query the host service manager from inside a container; exit code 0 means active); leave empty to use the built-in systemd check",
+					Value: "",
+				},
 				cli.StringFlag{
 					Name:  "version-file",
 					Usage: "specifies the version file to use for auto update (leave empty to disable auto update)",

cmd/gpud/run/command.go

@@ -130,6 +130,10 @@ func Command(cliContext *cli.Context) error {
 	enableAutoUpdate := cliContext.Bool("enable-auto-update")
 	autoUpdateExitCode := cliContext.Int("auto-update-exit-code")
 	rebootCommands := cliContext.String("reboot-commands")
+	findmntCommands := cliContext.String("findmnt-commands")
+	lsblkCommands := cliContext.String("lsblk-commands")
+	blockdevUsageCommands := cliContext.String("blockdev-usage-commands")
+	containerdServiceActiveCommands := cliContext.String("containerd-service-active-commands")
 	versionFile := cliContext.String("version-file")
 	versionFileSet := cliContext.IsSet("version-file")
 	pluginSpecsFile := cliContext.String("plugin-specs-file")
@@ -339,6 +343,10 @@ func Command(cliContext *cli.Context) error {
 	cfg.EnableAutoUpdate = enableAutoUpdate
 	cfg.AutoUpdateExitCode = autoUpdateExitCode
 	cfg.RebootCommands = rebootCommands
+	cfg.FindmntCommands = findmntCommands
+	cfg.LsblkCommands = lsblkCommands
+	cfg.BlockdevUsageCommands = blockdevUsageCommands
+	cfg.ContainerdServiceActiveCommands = containerdServiceActiveCommands
 	if !versionFileSet {
 		versionFile = config.VersionFilePath(cfg.DataDir)
 	}

components/containerd/component.go

@@ -109,7 +109,16 @@ func New(gpudInstance *components.GPUdInstance) (components.Component, error) {
 		checkDependencyInstalledFunc: checkContainerdInstalled,
 		checkSocketExistsFunc:        checkSocketExistsFunc,
 		checkContainerdRunningFunc:   CheckContainerdRunning,
-		checkServiceActiveFunc: func(_ context.Context) (bool, error) {
+		// checkServiceActiveFunc defaults to the in-namespace systemd.IsActive
+		// check. When ContainerdServiceActiveCommands is set (e.g. wrapping
+		// "systemctl is-active containerd" with nsenter), the check runs against
+		// the host's service manager instead -- required when gpud runs inside a
+		// container, where the container's own systemd does not manage containerd.
+		// Empty preserves the legacy behavior byte-for-byte.
+		checkServiceActiveFunc: func(ctx context.Context) (bool, error) {
+			if gpudInstance.ContainerdServiceActiveCommands != "" {
+				return CheckServiceActiveWithCommand(ctx, gpudInstance.ContainerdServiceActiveCommands)
+			}
 			return systemd.IsActive("containerd")
 		},
 		getContainerdUptimeFunc: func() (*time.Duration, error) {

components/containerd/cri.go

@@ -249,6 +249,27 @@ func CheckContainerdRunning(ctx context.Context) bool {
 	return false
 }
 
+// CheckServiceActiveWithCommand runs the given command and reports whether the
+// containerd service is active based on its exit code (0 = active). It overrides
+// the default in-namespace systemd.IsActive check, e.g. by wrapping
+// "systemctl is-active containerd" with nsenter so the check queries the host's
+// service manager from inside a container. A non-zero exit means "not active"
+// (not an error); only a failure to execute the command itself returns an error.
+func CheckServiceActiveWithCommand(ctx context.Context, command string) (bool, error) {
+	// #nosec G204 -- command is an operator-provided override (same trust model as reboot-commands).
+	out, err := exec.CommandContext(ctx, "bash", "-c", command).CombinedOutput()
+	if err != nil {
+		var exitErr *exec.ExitError
+		if errors.As(err, &exitErr) {
+			log.Logger.Debugw("containerd service-active command reported inactive",
+				"command", command, "exitCode", exitErr.ExitCode(), "output", string(out))
+			return false, nil
+		}
+		return false, fmt.Errorf("failed to run containerd service-active command %q: %w", command, err)
+	}
+	return true, nil
+}
+
 // GetVersion gets the version of the containerd runtime.
 func GetVersion(ctx context.Context, endpoint string) (string, error) {
 	conn, err := connect(ctx, endpoint)

components/containerd/cri_test.go

@@ -802,3 +802,22 @@ func TestIsErrUnimplemented(t *testing.T) {
 		})
 	}
 }
+
+func TestCheckServiceActiveWithCommand(t *testing.T) {
+	ctx := context.Background()
+
+	// exit code 0 => active
+	active, err := CheckServiceActiveWithCommand(ctx, "true")
+	require.NoError(t, err)
+	assert.True(t, active, "expected active=true for exit-0 command")
+
+	// emulates "systemctl is-active containerd" returning active
+	active, err = CheckServiceActiveWithCommand(ctx, "echo active")
+	require.NoError(t, err)
+	assert.True(t, active, "expected active=true for 'echo active'")
+
+	// non-zero exit => inactive, but NOT an error (mirrors systemctl exit 3)
+	active, err = CheckServiceActiveWithCommand(ctx, "false")
+	require.NoError(t, err, "non-zero exit should not be an error")
+	assert.False(t, active, "expected active=false for exit-non-zero command")
+}

components/disk/component.go

@@ -141,17 +141,26 @@ func newComponent(
 		getExt4PartitionsFunc: func(ctx context.Context) (disk.Partitions, error) {
 			timeoutCtx, cancel := context.WithTimeout(ctx, getPartitionsTimeout)
 			defer cancel()
-			return disk.GetPartitions(timeoutCtx, disk.WithFstype(disk.DefaultExt4FsTypeFunc), disk.WithMountPoint(disk.DefaultMountPointFunc))
+			// WithBlockdevUsageCommand is empty by default (legacy gopsutil + statfs
+			// path); when configured it runs the override (e.g. nsenter df) so usage
+			// is read from the host mount namespace. The getPartitionsTimeout context
+			// bounds the command, so a hung filesystem cannot block indefinitely.
+			return disk.GetPartitions(timeoutCtx, disk.WithFstype(disk.DefaultExt4FsTypeFunc), disk.WithMountPoint(disk.DefaultMountPointFunc), disk.WithBlockdevUsageCommand(gpudInstance.BlockdevUsageCommands))
 		},
 		getNFSPartitionsFunc: func(ctx context.Context) (disk.Partitions, error) {
 			// statfs on nfs can incur network I/O or impact disk I/O performance
 			// do not track usage for nfs partitions
 			timeoutCtx, cancel := context.WithTimeout(ctx, getPartitionsTimeout)
 			defer cancel()
-			return disk.GetPartitions(timeoutCtx, disk.WithFstype(disk.DefaultNFSFsTypeFunc), disk.WithMountPoint(disk.DefaultMountPointFunc))
+			return disk.GetPartitions(timeoutCtx, disk.WithFstype(disk.DefaultNFSFsTypeFunc), disk.WithMountPoint(disk.DefaultMountPointFunc), disk.WithBlockdevUsageCommand(gpudInstance.BlockdevUsageCommands))
 		},
 
-		findMntFunc: disk.FindMnt,
+		// findMntFunc defaults to the legacy in-namespace findmnt when
+		// FindmntCommands is empty (FindMntWithCommand(ctx, target, "") == FindMnt),
+		// and runs the override (e.g. nsenter findmnt) when configured.
+		findMntFunc: func(ctx context.Context, target string) (*disk.FindMntOutput, error) {
+			return disk.FindMntWithCommand(ctx, target, gpudInstance.FindmntCommands)
+		},
 
 		// Initialize file operation function field with real implementation
 		statWithTimeoutFunc: pkgfile.StatWithTimeout,
@@ -167,11 +176,17 @@ func newComponent(
 		c.getBlockDevicesFunc = func(ctx context.Context) (disk.BlockDevices, error) {
 			timeoutCtx, cancel := context.WithTimeout(ctx, getBlockDevicesTimeout)
 			defer cancel()
+			// WithLsblkCommand/WithFindmntCommand are empty by default (legacy
+			// in-namespace lsblk + findmnt back-fill); when configured they run the
+			// overrides (e.g. nsenter lsblk/findmnt) so block devices and fstypes
+			// are read from the host mount namespace.
 			return disk.GetBlockDevicesWithLsblk(
 				timeoutCtx,
 				disk.WithFstype(disk.DefaultFsTypeFunc),
 				disk.WithDeviceType(disk.DefaultDeviceTypeFunc),
 				disk.WithMountPoint(disk.DefaultMountPointFunc),
+				disk.WithLsblkCommand(gpudInstance.LsblkCommands),
+				disk.WithFindmntCommand(gpudInstance.FindmntCommands),
 			)
 		}
 	}
@@ -576,12 +591,17 @@ func (c *component) Check() components.CheckResult {
 		// e.g.,
 		// "unexpected end of JSON input"
 		prevFailed := false
-		for range 5 {
+		findMntSucceeded := false
+		for attempt := range 5 {
 			cctx, ccancel := context.WithTimeout(c.ctx, time.Minute)
 			mntOut, err := c.findMntFunc(cctx, target)
 			ccancel()
 			if err != nil {
-				log.Logger.Errorw("failed to find mnt", "error", err)
+				// findmnt is occasionally flaky and returns empty output, which
+				// parses as "unexpected end of JSON input". This is transient, so
+				// warn and retry; only a fully-exhausted retry budget (handled
+				// after the loop) is a real error.
+				log.Logger.Warnw("failed to find mnt, will retry", "target", target, "attempt", attempt+1, "error", err)
 
 				select {
 				case <-c.ctx.Done():
@@ -600,10 +620,14 @@ func (c *component) Check() components.CheckResult {
 			}
 			cr.MountTargetUsages[target] = *mntOut
 			if prevFailed {
-				log.Logger.Infow("successfully ran findmnt after retries")
+				log.Logger.Infow("successfully ran findmnt after retries", "target", target)
 			}
+			findMntSucceeded = true
 			break
 		}
+		if !findMntSucceeded {
+			log.Logger.Errorw("failed to find mnt after retries", "target", target)
+		}
 	}
 
 	if len(cr.BlockDevices) > 0 {

components/registry.go

@@ -43,6 +43,33 @@ type GPUdInstance struct {
 	MountPoints  []string
 	MountTargets []string
 
+	// FindmntCommands overrides how the disk component invokes "findmnt".
+	// Empty (the default) preserves the legacy behavior of locating "findmnt" on
+	// PATH and running it in the current namespace. When set (e.g.
+	// "nsenter --target 1 --mount -- findmnt"), the command runs in the host
+	// mount namespace so it reports the host's mounts instead of the container's.
+	FindmntCommands string
+
+	// LsblkCommands overrides how the disk component invokes "lsblk".
+	// Empty (the default) preserves the legacy behavior. When set (e.g.
+	// "nsenter --target 1 --mount -- lsblk"), the command runs in the host mount
+	// namespace.
+	LsblkCommands string
+
+	// BlockdevUsageCommands overrides how the disk component collects partition
+	// usage. Empty (the default) preserves the legacy behavior of enumerating
+	// mounts via gopsutil and measuring usage via the statfs syscall. When set
+	// (e.g. "nsenter --target 1 --mount -- df"), partitions and usage are read
+	// from that command's output in the host mount namespace.
+	BlockdevUsageCommands string
+
+	// ContainerdServiceActiveCommands overrides how the containerd component
+	// checks whether the containerd service is active. Empty (the default)
+	// preserves the legacy in-namespace systemd.IsActive behavior. When set (e.g.
+	// "nsenter --target 1 --mount -- systemctl is-active containerd"), the command
+	// runs against the host's service manager; exit code 0 means active.
+	ContainerdServiceActiveCommands string
+
 	FailureInjector *FailureInjector
 }
 

deployments/helm/gpud/.helmignore

@@ -21,3 +21,6 @@
 .idea/
 *.tmproj
 .vscode/
+
+# Private values overlay -- internal only
+values-nvcr.yaml