cleanup-hf-model: target any HF cache volume by name and run in a separate compose project

[lloydmak99]

Problem

Inference CVMs carry two HuggingFace cache volumes because prod/small-models.yaml and prod/dsv4-qwen36-gemma4.yaml spell the volume hugginface_cache. Stale checkpoints accumulate in whichever volume a retired pack used. On gpu23 the misspelled work_hugginface_cache holds about 1.1 TB that nothing mounts (zai-org/GLM-5-FP8, sgl-project/DeepSeek-V4-Flash-FP8, google/gemma-4-31B-it, Qwen/Qwen3.6-27B-FP8) and the guest is at 1,669 GB of 2,122 GB.

cleanup-hf-model.yaml could not reach that volume: it declared a plain huggingface_cache volume, so it only resolved to <project>_huggingface_cache, and in any project other than work it silently created a new empty volume. Running it inside work also lets compose-manager's --remove-orphans stop the live model. The two alternatives do not work today: compose-manager's /docker/evict fails closed on every prod CVM (its docker inspect parser rejects "Cmd": null), and /docker/clean runs docker volume prune -f, which skips named volumes on the guest's Docker 25.0.3.

Changes

• Declare the volume external: true with name: ${HF_CACHE_VOLUME:-work_huggingface_cache}, so the utility targets an existing volume by its full name and Compose fails fast instead of creating an empty one.
• MODEL_NAME accepts one or more org/repo values separated by commas or whitespace. Every token is validated (exactly one /, no leading slash, no ..) before any listing or deletion.
• Per-model removal loop with a Removed N of M requested model(s). summary; absent models are reported and skipped, not treated as errors.
• Header comment and README document the recommended invocation with "project":"cleanup" and the HF_CACHE_VOLUME override, and keep the warning about running in work.

No file under prod/ or experiments/ changes.

Validation

• MODEL_NAME=org/repo docker compose -f cleanup-hf-model.yaml config renders external: true and name: work_huggingface_cache; with HF_CACHE_VOLUME=work_hugginface_cache it renders name: work_hugginface_cache.
• Functional test on a scratch volume seeded with models--org--a, models--org--b, models--org--keep: MODEL_NAME="org/a, org/b org/missing" removed a and b, left keep, logged one Not found line and Removed 2 of 3 requested model(s)., exit 0.
• MODEL_NAME="../etc" exits 1 with Invalid model token and deletes nothing.
• PyYAML parse of the file passes; the OTel validator continues to exclude the file by basename.

Rollout Notes

Nothing deploys on merge. After the auto-tag, the gpu23 cleanup is one call, run in its own project so the GLM-5.2 stack in work is untouched:

{"tag":"<tag>","file":"cleanup-hf-model.yaml","project":"cleanup","force_recreate":true,
 "env":{"MODEL_NAME":"zai-org/GLM-5-FP8, sgl-project/DeepSeek-V4-Flash-FP8, google/gemma-4-31B-it, Qwen/Qwen3.6-27B-FP8",
        "HF_CACHE_VOLUME":"work_hugginface_cache"}}

The live GLM-5.2 W4AFP8 weights are in work_huggingface_cache and are not selected. Expected result is about 1.1 TB freed inside the gpu23 guest; the host-side qcow2 does not shrink. gpu03 and gpu13 carry similar leftovers in their misspelled volumes and can use the same call.

[github-actions]

🔍 OpenCodeReview found 1 issue(s) in this PR.

• ✅ 1 posted as inline comment(s)
• 📝 0 posted as summary

[github-actions]

set -f (noglob) is set here and never re-enabled, which silently breaks the downstream "Cached model weights" diagnostics at lines 74–75:

if ls "$$HUB"/models--* >/dev/null 2>&1; then
du -sh "$$HUB"/models--* | sort -rh

Because pathname expansion is disabled, the models--* pattern is passed literally to ls/du. Since no file is literally named models--*, the ls check always fails and the script prints (none) even when real model directories exist — so operators can no longer verify what is cached before deletion. This is a regression introduced by this change; the deletion logic itself (rm -rf "$$TARGET") is unaffected because it uses fully-qualified paths.

Glob disabling is only needed around set -- $$MODELS (to stop a token like * from expanding to filenames). Re-enable it immediately afterward so the existing globs work as intended.

Suggestion:

Suggested change

	set -f
	MODELS=$$(printf '%s' "$$MODEL_NAME" | tr ',' ' ')
	set -- $$MODELS
	set -f
	MODELS=$$(printf '%s' "$$MODEL_NAME" | tr ',' ' ')
	set -- $$MODELS
	set +f

[PierreLeGuen]

The external-volume declaration and multi-model validation loop are sound, but set -f (line 47) is never cleared, so the pre-deletion models--* listing always prints (none) — the one in-run confirmation that the right volume is mounted before an irreversible rm -rf. Adding set +f after validation fixes it.

Optional follow-ups:

• cleanup-hf-model.yaml:74 — set -f is enabled at line 47 so that set -- $MODELS does not glob-expand model tokens, but it is never turned back off. Fix: Add set +f immediately after the validation loop closes (line 66, before the '=== Disk usage ===' section).
• cleanup-hf-model.yaml:37 — The PR makes "project":"cleanup" the recommended invocation, but the service keeps the fixed container_name. Fix: Drop the container_name: hf-cleanup pin so Compose derives project-scoped names (cleanup-hf-cleanup-1`); nothing resolves this container…

Checks: PyYAML parse of cleanup-hf-model.yaml: passed. git diff --check: clean.