osdc/hf-cache: cap rclone Go heap with GOMEMLIMIT (#892)

rclone is a Go binary, and its OOMs are a **GC** problem, not a genuine
memory requirement: under bursty `--vfs-cache-mode full` buffering the
Go heap grows past the cgroup limit before the GC runs, so the kernel
OOM-kills rclone and the `/mnt/hf_cache` FUSE mount dies node-wide (→
job-pod `CreateContainerError`).

## What
Give the Go runtime a soft heap ceiling (`GOMEMLIMIT`) at **~90% of the
reserved limit** so the GC reclaims buffering before the kernel does —
no limit changes.

`deploy.sh` derives `GOMEMLIMIT` from each tier's memory (the value it
already renders for `__RCLONE_MEMORY_LIMIT__`) and injects it as a
**literal env value**:

```yaml
env:
  - name: GOMEMLIMIT
    value: "__GOMEMLIMIT__"   # e.g. 460MiB for the 512Mi tier
```

So there's:
- no runtime shell arithmetic (a `bytes * 90` intermediate overflows
32-bit busybox);
- no downward-API indirection to resolve at runtime;
- a literal you can read straight out of the rendered manifest.

Go needs `MiB`/`GiB` suffixes (it rejects k8s `Mi`/`Gi`), so the value
is converted in `deploy.sh`, not reused verbatim.

Per tier: 256Mi→230MiB, 512Mi→460MiB, 1Gi→921MiB, 2Gi→1843MiB,
4Gi→3686MiB.

## Verification
- `just lint` 13/13, `just test` pass.
- Rendered each tier via `deploy.sh`'s logic and validated with
`kubeconform -strict`; confirmed the manifest carries a literal
`GOMEMLIMIT` (no `resourceFieldRef`) that sits below the container
limit.

Author: huydhn

osdc/integration-tests/workflows/integration-test.yaml.tpl

@@ -349,7 +349,7 @@ jobs:
             echo "FAIL: $MODEL not in the cache ($CACHE_DIR missing) — seed it with scripts/hf-cache-seed.py"
             exit 1
           fi
-          pip install --no-cache-dir torch  # default CUDA build for the GPU runner
+          pip install --no-cache-dir torch==2.12.1  # default CUDA build for the GPU runner
           pip install --no-cache-dir 'transformers>=4.45' 'huggingface_hub>=0.24' accelerate
           # device_map='cuda' loads weights to VRAM (see runs-on note); greedy decode.
           MODEL="$MODEL" python3 -c "
@@ -452,7 +452,7 @@ jobs:
     steps:
       - name: Install deps (torch, transformers, hub, awscli)
         run: |
-          pip install --no-cache-dir torch  # default build; runs on CPU for a tiny model
+          pip install --no-cache-dir torch==2.12.1  # default build; runs on CPU for a tiny model
           # transformers <5: bert-tiny's config.json has no model_type key (5.x rejects
           # it; 4.x infers "bert"). hf_hub <1 stays compatible with transformers 4.x.
           pip install --no-cache-dir 'transformers>=4.45,<5' 'huggingface_hub>=0.24,<1' awscli
@@ -716,7 +716,7 @@ jobs:
       - name: Verify torch CPU install (pip)
         run: |
           echo "=== Torch CPU Install (pip) ==="
-          pip install --no-cache-dir torch==2.11.0
+          pip install --no-cache-dir torch==2.12.1
           python3 -c "
           import torch, sys
           cuda = torch.version.cuda
@@ -731,7 +731,7 @@ jobs:
         run: |
           echo "=== Torch CPU Install (uv) ==="
           pip uninstall -y torch >/dev/null 2>&1 || true
-          uv pip install --system --no-cache torch==2.11.0
+          uv pip install --system --no-cache torch==2.12.1
           python3 -c "
           import torch, sys
           cuda = torch.version.cuda
@@ -1076,7 +1076,7 @@ jobs:
       - name: Verify torch CPU install (pip)
         run: |
           echo "=== Torch CPU Install (pip) ==="
-          pip install --no-cache-dir torch==2.11.0
+          pip install --no-cache-dir torch==2.12.1
           python3 -c "
           import torch, sys
           cuda = torch.version.cuda
@@ -1091,7 +1091,7 @@ jobs:
         run: |
           echo "=== Torch CPU Install (uv) ==="
           pip uninstall -y torch >/dev/null 2>&1 || true
-          uv pip install --system --no-cache torch==2.11.0
+          uv pip install --system --no-cache torch==2.12.1
           python3 -c "
           import torch, sys
           cuda = torch.version.cuda
@@ -1467,7 +1467,7 @@ jobs:
       - name: Verify torch CUDA install (pip)
         run: |
           echo "=== Torch CUDA Install (pip) ==="
-          pip install --no-cache-dir torch==2.11.0
+          pip install --no-cache-dir torch==2.12.1
           python3 -c "
           import torch, sys
           cuda = torch.version.cuda
@@ -1482,7 +1482,7 @@ jobs:
         run: |
           echo "=== Torch CUDA Install (uv) ==="
           pip uninstall -y torch >/dev/null 2>&1 || true
-          uv pip install --system --no-cache torch==2.11.0
+          uv pip install --system --no-cache torch==2.12.1
           python3 -c "
           import torch, sys
           cuda = torch.version.cuda

osdc/modules/hf-cache/deploy.sh

@@ -87,6 +87,17 @@ render_mount_ds() {
   # $1 = DaemonSet name, $2 = affinity operator, $3 = gpu-count CSV,
   # $4 = rclone memory (rendered as both request and limit → reserved)
   local values="[\"${3//,/\",\"}\"]"
+  # GOMEMLIMIT ~= 90% of the reserved limit, in Go's byte-suffix format (MiB/GiB — Go
+  # rejects k8s's Mi/Gi). rclone is a Go binary that OOMs from lazy GC, not real need;
+  # a soft heap ceiling below the cgroup limit makes the GC run before the kernel kills
+  # the mount. Computed here (bash) so it renders to a literal — no runtime arithmetic.
+  local mib
+  case "$4" in
+    *Gi) mib=$((${4%Gi} * 1024)) ;;
+    *Mi) mib=${4%Mi} ;;
+    *) mib=0 ;;
+  esac
+  local gomemlimit="$((mib * 90 / 100))MiB"
   sed -e "s|__NAMESPACE__|${NAMESPACE}|g" \
     -e "s|__BUCKET__|${BUCKET}|g" \
     -e "s|__REGION__|${BUCKET_REGION}|g" \
@@ -97,6 +108,7 @@ render_mount_ds() {
     -e "s|__GPU_OP__|${2}|g" \
     -e "s|__MULTI_GPU_COUNTS__|${values}|g" \
     -e "s|__RCLONE_MEMORY_LIMIT__|${4}|g" \
+    -e "s|__GOMEMLIMIT__|${gomemlimit}|g" \
     "$MODULE_DIR/kubernetes/mount-daemonset.yaml.tpl"
 }
 

osdc/modules/hf-cache/kubernetes/mount-daemonset.yaml.tpl

@@ -8,7 +8,7 @@
 # the tiers mutually exclusive (exactly one mount per node).
 #
 # Placeholders (deploy.sh): __NAMESPACE__ __BUCKET__ __REGION__ __RCLONE_IMAGE__
-# __VFS_CACHE_MAX_SIZE__ __TAINT_REMOVER_IMAGE__ __RCLONE_MEMORY_LIMIT__
+# __VFS_CACHE_MAX_SIZE__ __TAINT_REMOVER_IMAGE__ __RCLONE_MEMORY_LIMIT__ __GOMEMLIMIT__
 # __DS_NAME__ __GPU_OP__ __MULTI_GPU_COUNTS__
 apiVersion: apps/v1
 kind: DaemonSet
@@ -85,6 +85,13 @@ spec:
           # FUSE + Bidirectional propagation need privileged.
           securityContext:
             privileged: true
+          env:
+            # rclone (Go) OOMs from lazy GC, not a real memory need: cap the heap below the
+            # cgroup limit so the GC runs before the kernel OOM-kills the mount node-wide.
+            # deploy.sh derives this from the tier's memory (~90%, in Go's MiB/GiB format —
+            # not k8s Mi/Gi), so it tracks __RCLONE_MEMORY_LIMIT__ without runtime arithmetic.
+            - name: GOMEMLIMIT
+              value: "__GOMEMLIMIT__"
           command:
             - /bin/sh
             - -c
@@ -157,7 +164,8 @@ spec:
             timeoutSeconds: 10
             failureThreshold: 3
           # An rclone OOM drops the mount node-wide. Memory is tiered by GPU count and
-          # reserved (request == limit) — see deploy.sh MOUNT_TIERS.
+          # reserved (request == limit) — see deploy.sh MOUNT_TIERS. GOMEMLIMIT (env above)
+          # is derived from this limit so the Go GC caps the heap first.
           resources:
             requests:
               cpu: 100m

osdc/modules/hf-cache/tests/smoke/test_hf_cache.py

@@ -84,6 +84,23 @@ def test_mount_is_read_only(self, mount_pod_spec: dict) -> None:
         cmd = " ".join(mount_pod_spec["containers"][0].get("command", []))
         assert "--read-only" in cmd, "rclone mount must be --read-only — job pods must not write the shared cache."
 
+    def test_gomemlimit_below_container_limit(self, mount_pod_spec: dict) -> None:
+        """deploy.sh renders GOMEMLIMIT from the tier's memory limit (~90%, in Go's MiB
+        format) so the Go GC caps the heap before the kernel OOM-kills the mount. It must be
+        set and sit below the container limit (headroom for non-heap memory)."""
+        rclone = mount_pod_spec["containers"][0]
+        gomemlimit = next((e.get("value") for e in rclone.get("env", []) if e["name"] == "GOMEMLIMIT"), None)
+        assert gomemlimit is not None, "rclone must set a literal GOMEMLIMIT env."
+        assert gomemlimit.endswith("MiB"), f"GOMEMLIMIT must be Go MiB format (not k8s Mi); got {gomemlimit!r}."
+        gml_mib = int(gomemlimit[:-3])
+
+        limit = rclone.get("resources", {}).get("limits", {}).get("memory", "")
+        limit_mib = int(limit[:-2]) * (1024 if limit.endswith("Gi") else 1)
+        assert 0 < gml_mib < limit_mib, (
+            f"GOMEMLIMIT ({gml_mib}MiB) must be >0 and below the container limit ({limit}) — "
+            "a soft ceiling with headroom, not the hard cap."
+        )
+
     def test_hostpath_bidirectional(self, mount_pod_spec: dict) -> None:
         mounts = mount_pod_spec["containers"][0].get("volumeMounts", [])
         hf = [m for m in mounts if m.get("mountPath") == MOUNT_PATH]