Fix inotify pre-flight check for PROM_ENABLED on macOS (llm-d#1654)

* Evaluate inotify limit on the kind node, not the host

The Prometheus config-reloader inotify pre-flight check ran an
unconditional cat of /proc/sys/fs/inotify/max_user_instances on the
host, before the cluster existed. On macOS that proc file is absent
(kind runs inside the container-runtime VM), so the check aborted
env-dev-kind with PROM_ENABLED=true.

The limit that applies to pods is the kind node's kernel: the host
kernel on Linux, the runtime VM kernel on macOS. Move the check to
after the node container is created and read the value from inside it,
which is correct on both platforms. Update the remediation message to
cover raising the limit on the host (Linux) and in the runtime VM
(macOS).

Signed-off-by: roytman <roytman@il.ibm.com>

* fix copilot comments

Signed-off-by: roytman <roytman@il.ibm.com>

---------

Signed-off-by: roytman <roytman@il.ibm.com>

Author: roytman

scripts/kind-dev-env.sh

@@ -198,17 +198,34 @@ for cmd in kind kubectl ${CONTAINER_RUNTIME}; do
     fi
 done
 
-# Prometheus config-reloader needs sufficient inotify resources
-if [ "${PROM_ENABLED}" == "true" ]; then
-  INOTIFY_INSTANCES=$(cat /proc/sys/fs/inotify/max_user_instances)
-  if [ "${INOTIFY_INSTANCES}" -lt 512 ]; then
-    echo "Error: fs.inotify.max_user_instances is ${INOTIFY_INSTANCES} (need >= 512) for Prometheus."
+# The Prometheus config-reloader needs a sufficient inotify instance limit. The
+# argument is the limit read from the kernel backing the kind node: the host on
+# Linux, the runtime VM on macOS. A non-numeric value means the limit could not
+# be read; warn and skip rather than fail or compare a garbage string.
+check_inotify_limit() {
+  local limit="$1"
+  if ! [[ "${limit}" =~ ^[0-9]+$ ]]; then
+    echo "Warning: could not read fs.inotify.max_user_instances; skipping Prometheus inotify check." >&2
+    return 0
+  fi
+  if [ "${limit}" -lt 512 ]; then
+    echo "Error: fs.inotify.max_user_instances is ${limit} (need >= 512) for Prometheus."
     echo ""
+    echo "Raise it on the kind node's kernel. On Linux this is the host:"
     echo "  sudo sysctl -w fs.inotify.max_user_instances=512"
-    echo ""
     echo "To persist: echo 'fs.inotify.max_user_instances=512' | sudo tee /etc/sysctl.d/99-inotify.conf"
+    echo ""
+    echo "On macOS the kind node runs in the container runtime VM; raise it there, e.g.:"
+    echo "  ${CONTAINER_RUNTIME} exec ${CLUSTER_NAME}-control-plane sysctl -w fs.inotify.max_user_instances=512"
     exit 1
   fi
+}
+
+# On Linux the host kernel is shared with kind's node containers, so the limit
+# can be verified before creating the cluster to fail fast. Hosts where kind
+# runs inside a VM have no inotify proc file and are checked from the node below.
+if [ "${PROM_ENABLED}" == "true" ] && [ -r /proc/sys/fs/inotify/max_user_instances ]; then
+  check_inotify_limit "$(cat /proc/sys/fs/inotify/max_user_instances)"
 fi
 
 # TARGET_PORTS is substituted directly into the `targetPorts: ${TARGET_PORTS}` field
@@ -265,6 +282,13 @@ set -x
 CONTAINER_NAME="${CLUSTER_NAME}-control-plane"
 ${CONTAINER_RUNTIME} exec ${CONTAINER_NAME} /bin/bash -c "sysctl net.ipv4.conf.all.arp_ignore=0"
 
+# Hosts where kind runs inside a VM (e.g. macOS) have no inotify proc file, so
+# the limit is read from the node, whose kernel is the runtime VM's. The host
+# case is handled by the fast-fail check before cluster creation.
+if [ "${PROM_ENABLED}" == "true" ] && [ ! -r /proc/sys/fs/inotify/max_user_instances ]; then
+  check_inotify_limit "$(${CONTAINER_RUNTIME} exec "${CONTAINER_NAME}" cat /proc/sys/fs/inotify/max_user_instances 2>/dev/null || true)"
+fi
+
 # Wait for all pods to be ready
 kubectl --context ${KUBE_CONTEXT} -n kube-system wait --for=condition=Ready --all pods --timeout=300s