osdc/hf-cache: fix prepare-host-mount init crash-loop on shadowed /mnt/hf_cache

[huydhn]

Problem

#876's self-heal init (prepare-host-mount) can crash-loop instead of healing, leaving the rclone cache mount silently absent on a node whose node-init.osdc.io/hf-cache startup taint has already been cleared by an earlier healthy incarnation. Nothing re-adds the taint, so CI job pods keep landing on the node and read an empty /mnt/hf_cache.

Observed on meta-prod-aws-ue2: 20 hf-cache mounts wedged (init CrashLoopBackOff, 70–92 restarts ≈ 6h), 29 job/workflow pods running on those broken-cache nodes — including a failed pytorch inductor job that read the empty cache instead of the model bucket.

Root cause (all in the prepare-host-mount init)

1. mount --bind /mnt /mnt is non-recursive, so re-binding /mnt shadows a pre-existing /mnt/hf_cache submount (e.g. one created by an earlier init version that bound /mnt/hf_cache directly on the root fs). The shadowed mount becomes an unreachable orphan (parent = root fs, hidden under the /mnt bind).
2. The guard grep -q " /mnt/hf_cache " /proc/mounts matches that shadowed orphan (a raw table scan sees hidden mounts), so the re-bind is skipped.
3. mount --make-rshared /mnt/hf_cache then runs on a live path that is not a mountpoint → not mount point or bad option → set -e → the init exits non-zero → CrashLoopBackOff forever.

Because the init never completes, the rclone container never starts, so /mnt/hf_cache is just a bare dir. Verified on the host: the stale mount was an xfs self-bind (/dev/nvme0n1p1), and umount/make-rshared by path failed because the orphan had no reachable live path.

Fix

• Guard on the live path with mountpoint -q, never a /proc/mounts scan — a shadowed orphan can no longer make us skip the bind.
• Use mount --rbind for /mnt so an existing submount is carried into the new bind instead of being shadowed.
• Tolerate make-rshared failures (|| true) so a transient state can't become a crash-loop.
• Peel any occupant of the live /mnt/hf_cache (dead FUSE or stale xfs bind) with a bounded loop, not one gated on stat (a stale bind stats fine, so the old loop missed it).

Testing

• just lint (all 13) ✅ and just test (98.78% cov) ✅
• Smoke test's make-rshared assertion still holds.
• Not yet deployed. Needs a soak on staging + a rolling redeploy to ue1/ue2.

Follow-up (separate PR)

Fail-closed taint: have the init re-add node-init.osdc.io/hf-cache on start and only let the taint-remover clear it once this incarnation's rclone is confirmed mounted, so a future unhealthy mount keeps jobs off the node instead of relying on the mount always recovering. (Requires an add-taint mode in taint_remover.py.)