feat(bind-mount): add validated bind mount for pre-provisioned local NVMe (#89)

## Description

**TL;DR.** New standalone package `bind-mount`: it exposes one existing
host mount at a second stable path through a persistent, owner-marked
systemd bind mount, with a fail-closed safety contract on every
destructive path. It does no device discovery, no RAID assembly, and no
mkfs; it only aliases a mount that platform bootstrap already created,
and it survives reboots (which a raw `mount --bind` in user data does
not).

### What the package does

- Validates the source in the host mount namespace: it must be an exact,
read-write mount point, with optional fstype (`expected-source-fstype`)
and device-prefix (`expected-source-device-prefix`) assertions.
- Renders and enables a systemd `.mount` unit (bind) tagged with an
owner marker, then relies on the package's controlled interrupt (reboot)
to activate it; the post-interrupt check proves the active mount binds
the desired source before the node returns to service.
- Persists root-only receipts under
`/var/lib/nodewright-packages/bind-mount/` before any systemd operation,
so uninstall works even when the operator runs uninstall without the
original ConfigMap.
- All 10 lifecycle modes are wired in `config.json` and idempotent;
`make validate-standalone PACKAGE=bind-mount` passes against the agent
v1 schemas.

### Fail-closed guarantees

- Refuses a non-empty target directory (never hides existing data), a
symlinked target, a target owned by `/etc/fstab`, and any pre-existing
unit file with different content (foreign units are never overwritten or
removed; removal requires the owner marker).
- Refuses source/target overlap, a read-only source or target mount, and
a config change while the old target is still mounted (explicit
remediation message instead of silent re-pointing).
- Uninstall never unmounts a live target: it removes persistence only
and tells the operator to recycle the node, so a running workload is
never yanked out from under.
- A unit rejected by `systemd-analyze verify` is rolled back; a failed
`systemctl enable` retains the receipt so cleanup still works.

### Included hardening (folded in after an adversarial review of the
initial draft)

1. **ConfigMap-less uninstall of a never-installed package is now a
clean no-op.** Previously, if install had never succeeded (no receipts)
and the operator sent an uninstall without the package ConfigMap (both
the legacy stub-uninstall lineage and the current faux-package path do
this), `load_desired_config` hard-failed on the missing `source` key and
package removal wedged on that node forever. Now that state degrades to
a best-effort sweep: remove any owner-marked `.mount` units, succeed,
write nothing. Covered by a new lifecycle test
(`test_uninstall_of_never_installed_package_is_clean_noop`): failed
install (wrong source fstype), ConfigMaps deleted, then `uninstall` and
`uninstall-check` both succeed.
2. **Kubelet ordering parity in the rendered unit.** The unit now
renders `Before=local-fs.target kubelet.service
snap.kubelet-eks.daemon.service`, matching the equivalent
launch-template user-data variant, so on reboot the mount orders before
kubelet on both EKS AMI flavors. systemd ignores `Before=` entries for
units that do not exist on the node, so this is inert on non-Kubernetes
hosts. This is ordering hygiene, not a success gate; the post-interrupt
check is the gate.

### Consumer context

First consumer is DGXC AV TestStudio (Skyhook/NodeWright on EKS,
internal tracker NKX-12808): moving BuildBarn worker file pools from the
root EBS volume onto pre-provisioned local NVMe (`/mnt/k8s-disks/0`
RAID0 from the EKS AMI's `setup-local-disks`, exposed at `/mnt/data`).
Publication of `ghcr.io/nvidia/nodewright-packages/bind-mount:0.1.0`
unblocks that fleet rollout; the consumer-side Skyhook config is already
staged behind a default-off flag pending this image.

### Testing

- `make test-package PACKAGE=bind-mount`: 32 passed (16 test functions x
ubuntu:22.04/24.04), including the new never-installed uninstall test.
- `make validate-standalone PACKAGE=bind-mount`: schema, step files, and
executable-bit validation all pass against
`ghcr.io/nvidia/skyhook/agent:latest`.
- `shellcheck` (stable) is clean on `bind_mount.sh`.
- Lifecycle tests run the real script in containers against fakes for
`findmnt`/`nsenter`/`stat`/`systemctl`/`systemd-analyze`/`systemd-escape`;
no real systemd or mounts are exercised. On-node validation on an EKS
GPU node is planned on the consumer side before any fleet enablement.

## Checklist
- [x] I am familiar with the [Contributing
Guidelines](https://github.com/NVIDIA/nodewright-packages/blob/HEAD/CONTRIBUTING.md).
- [x] New or existing tests cover these changes.
- [x] The documentation is up to date with these changes.

Signed-off-by: Gregory Giecold <ggiecold@nvidia.com>

Author: NVGreg

.github/ISSUE_TEMPLATE/bug_report_form.yml

@@ -39,6 +39,7 @@ body:
     attributes:
       label: Which package does this relate to?
       options:
+        - bind-mount
         - shellscript
         - tuning
         - tuned

.github/ISSUE_TEMPLATE/feature_request_form.yml

@@ -53,6 +53,7 @@ body:
     attributes:
       label: Which package does this relate to?
       options:
+        - bind-mount
         - shellscript
         - tuning
         - tuned

.github/labeler.yml

@@ -27,6 +27,11 @@ package/copy-fail:
       - any-glob-to-any-file:
           - 'copy-fail/**'
 
+package/bind-mount:
+  - changed-files:
+      - any-glob-to-any-file:
+          - 'bind-mount/**'
+
 package/kdump:
   - changed-files:
       - any-glob-to-any-file:

.github/labels.yml

@@ -130,6 +130,10 @@
   color: "1d76db"
   description: "Relates to the copy-fail package"
 
+- name: "package/bind-mount"
+  color: "1d76db"
+  description: "Relates to the bind-mount package"
+
 - name: "package/kdump"
   color: "1d76db"
   description: "Relates to the kdump package"

.github/workflows/triage.yaml

@@ -76,6 +76,7 @@ jobs:
               'nvidia-setup',
               'copy-fail',
               'shellscript',
+              'bind-mount',
               'kdump',
               'tuned',
               'tuning',

AGENTS.md

@@ -15,11 +15,11 @@ This repo holds the pre-built **packages** consumed by the [NVIDIA NodeWright op
 
 This repo does **not** contain the operator, the agent, or any Go code. It is shell scripts (could be python, or any executable, but currently is all shell), `Dockerfile`s, package config, and Python-based tests/tooling.
 
-**Rename status (Skyhook to NodeWright):** the project is being renamed from Skyhook to NodeWright. Many public names still use `skyhook` on purpose: the published image path (`ghcr.io/nvidia/skyhook-packages/...`), the CRD group (`skyhook.nvidia.com/v1alpha1`), the on-host `SKYHOOK_DIR`/`skyhook_dir` names, and the CLI. The GitHub repo itself is now `nodewright-packages` (with `skyhook-packages` redirecting). Do not "fix" `skyhook` to `nodewright` or vice versa; match whatever the surrounding code already uses.
+**Rename status (Skyhook to NodeWright):** the project is being renamed from Skyhook to NodeWright. New package releases publish at `ghcr.io/nvidia/nodewright-packages/...`; older package versions may still exist only under the legacy `ghcr.io/nvidia/skyhook-packages/...` namespace. The CRD group (`skyhook.nvidia.com/v1alpha1`), the on-host `SKYHOOK_DIR`/`skyhook_dir` names, and the CLI still use `skyhook` on purpose. The GitHub repo itself is now `nodewright-packages` (with `skyhook-packages` redirecting). Do not rename remaining `skyhook` references without checking their specific compatibility contract.
 
 ## Repository layout
 
-- **Each top-level directory that contains a `Dockerfile` is a package.** Current packages: `shellscript`, `tuning`, `tuned`, `kdump`, `nvidia-setup`, `nvidia-tuning-gke`, `nvidia-tuned`, `copy-fail`.
+- **Each top-level directory that contains a `Dockerfile` is a package.** Current packages: `shellscript`, `tuning`, `tuned`, `kdump`, `nvidia-setup`, `nvidia-tuning-gke`, `nvidia-tuned`, `copy-fail`, `bind-mount`.
 - `scripts/`: repo tooling (`validate.py`, `sync_labels.py`).
 - `tests/integration/`: Docker-based pytest suites, one subdir per package that has tests (with hyphens converted to underscores, e.g. `nvidia-setup` to `nvidia_setup`; not every package has a subdir).
 - `.github/`: workflows, the `build-package` composite action, issue/PR templates, `labels.yml`, `CODEOWNERS`.
@@ -105,7 +105,7 @@ Almost all package logic is bash. The existing scripts are not yet consistent (s
 
 - **On PR (`pr_build.yaml`):** detects which package directories changed, then for each changed package runs validate, test, and a dev build tagged `<version>-dev.sha<short>`. Changed-package detection keys on a directory having a `Dockerfile`; `scripts/` and hidden dirs are skipped.
 - **On tag push `<package>/<version>`:** two independent workflows run. `build_container.yaml` builds and pushes the multi-arch image (`linux/amd64,linux/arm64`) with a build-provenance attestation; `release.yml` creates the GitHub Release with `git-cliff` notes scoped to that package. The image build (here and in `pr_build.yaml`) goes through the `.github/actions/build-package` composite action. (`build_package.yml` is a reusable `workflow_call` workflow that exists but is not currently wired up.)
-- **Published image path:** `ghcr.io/nvidia/skyhook-packages/<package>:<version>` (the `skyhook-packages` segment is retained pending the rename).
+- **Published image path for new releases:** `ghcr.io/nvidia/nodewright-packages/<package>:<version>`. Verify legacy versions before changing an existing consumer because some older tags remain only under `ghcr.io/nvidia/skyhook-packages/`.
 - An optional `<package>/preprocess.sh` is run before the image build and can emit `BUILD_ARGS`.
 
 ## Code review (CodeRabbit)

README.md

@@ -118,9 +118,19 @@ A temporary mitigation package for **CVE-2026-31431 ("Copy Fail")** that disable
 
 **Key features:**
 - Distro-agnostic (works anywhere `kmod`/`modprobe` is honored)
-- No reboot, no configmap, no interrupts — minimal surface area
+- No reboot, no configmap, no interrupts: minimal surface area
 - Designed to be retired once distros ship a fixed kernel
 
+### 8. Bind Mount Package (`bind-mount/`)
+
+A fail-closed package that exposes an existing host mount at a stable second path through a persistent systemd bind mount.
+
+**Capabilities:**
+- Validates the exact source mount, filesystem type, device prefix, and read-write state
+- Refuses to hide nonempty target data or replace foreign mount definitions
+- Activates through a controlled NodeWright interrupt and verifies the host mount afterward
+- Removes persistence on uninstall without live-unmounting active workloads
+
 ## Package Structure
 
 Each package follows the standard NodeWright package structure:
@@ -261,6 +271,7 @@ This validation step is crucial as the agent uses JSON schema validation to ensu
 
 1. **Choose a package** that fits your use case:
    - `shellscript` for custom scripts and automation
+   - `bind-mount` for a persistent alias to an existing host filesystem
    - `tuning` for system-level configuration management  
    - `tuned` for automated performance tuning with the tuned daemon
    - `kdump` for kernel crash dump collection and debugging
@@ -278,6 +289,7 @@ This validation step is crucial as the agent uses JSON schema validation to ensu
 - [Kdump Package](./kdump/README.md) - Usage guide for the kdump package
 - [NVIDIA Setup Package](./nvidia-setup/README.md) - Node setup (EFA, Lustre, chrony, local disks) per service/accelerator
 - [Copy-Fail Package](./copy-fail/README.md) - CVE-2026-31431 modprobe-blacklist mitigation
+- [Bind Mount Package](./bind-mount/README.md) - Persistent, fail-closed bind mounts for existing host storage
 - [NVIDIA NodeWright Documentation](https://github.com/NVIDIA/skyhook) - Main NodeWright operator documentation
 
 ## Contributing
@@ -286,4 +298,4 @@ See [CONTRIBUTING.md](./CONTRIBUTING.md) for guidelines on contributing to this
 
 ## License
 
-This project is licensed under the terms specified in [LICENSE](./LICENSE).
+This project is licensed under the terms specified in [LICENSE](./LICENSE).

bind-mount/CHANGELOG.md

@@ -0,0 +1,9 @@
+# Changelog
+
+All notable changes to this package will be documented in this file.
+
+## [0.1.0]
+
+### New Features
+
+- Initial release of a fail-closed, systemd-managed bind mount package.

bind-mount/Dockerfile

@@ -0,0 +1,20 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+FROM busybox:latest
+
+RUN mkdir -p /skyhook-package/skyhook_dir
+COPY . /skyhook-package

bind-mount/README.md

@@ -0,0 +1,53 @@
+# Bind Mount Package
+
+The `bind-mount` package exposes one existing host mount at a second stable path through a persistent systemd bind mount. It does not discover, format, assemble, or delete storage.
+
+The package is intended for nodes where platform bootstrap owns the source filesystem and a workload needs a stable path. It validates the source in the host mount namespace, writes and enables a systemd mount unit, and relies on a NodeWright interrupt to activate the unit before workloads resume.
+
+## Configuration
+
+```yaml
+apiVersion: skyhook.nvidia.com/v1alpha1
+kind: Skyhook
+metadata:
+  name: local-data-bind
+spec:
+  runtimeRequired: true
+  interruptionBudget:
+    count: 1
+  packages:
+    bind-mount:
+      version: 0.1.0
+      image: ghcr.io/nvidia/nodewright-packages/bind-mount
+      interrupt:
+        type: reboot
+      configMap:
+        source: /mnt/k8s-disks/0
+        target: /mnt/data
+        expected-source-fstype: xfs
+        expected-source-device-prefix: /dev/md
+```
+
+`source` and `target` are required. `expected-source-fstype` and `expected-source-device-prefix` are optional fail-closed assertions.
+
+The image may be referenced under multiple package map keys on the same node to
+manage multiple bind mounts. Each instance has its own ownership receipt and
+owner marker, scoped to the Skyhook resource UID and package map key. Targets
+must be distinct: two instances that name the same systemd mount unit fail
+closed rather than sharing or replacing it.
+
+## Safety contract
+
+- The source must already be an exact, read-write mount point.
+- The target must be empty, or already be the same bind mount backed by this package instance's owned unit; otherwise the package fails.
+- The systemd unit rechecks target emptiness at activation and fails if the target is already non-empty, reducing the risk of silently hiding newly written data.
+- Existing symlinks, foreign systemd units, and `/etc/fstab` ownership are rejected.
+- Source and target paths must be disjoint; neither may contain the other.
+- Prepared and active checks reject a read-only target mount.
+- A source or target configuration change fails while that instance's old target remains mounted. Remove the package and recycle the node before changing paths.
+- Uninstall disables and removes the persistent unit but deliberately does not unmount a live target. Recycle the node to complete rollback safely.
+- A root-only ownership receipt supports ConfigMap-less uninstall on operators older than v0.16.0 and preserves cleanup after an interrupted install on newer operators. The receipt is removed after `uninstall-check` confirms that no owned unit remains.
+- Uninstall of a package instance that never installed succeeds as a clean no-op, even when the uninstall runs without a ConfigMap. It only sweeps units carrying that exact Skyhook-resource/package-key owner marker and cannot remove another instance's unit.
+- The mount unit orders before `local-fs.target` and the kubelet units (`kubelet.service`, `snap.kubelet-eks.daemon.service`) so a rebooting node mounts the target before workloads start. Ordering is hygiene, not a success gate; the post-interrupt check is the gate.
+
+The apply and config checks validate the prepared unit before an interrupt. The post-interrupt check validates the active host mount. Consumers should use `runtimeRequired: true` and a bounded interruption budget.