fix: cache Wan2.2 weights on the fleet's persistent volume

[crowecawcaw]

Problem

Follow-up to #269, addressing review feedback on the Wan2.2 sample's model cache.

HFCacheDir defaulted to /var/tmp/wan22_hf_cache, which is on the worker's root volume. Deadline Cloud only preserves a service-managed fleet's persistent volume across worker lifecycle events — the root volume is discarded. So a customer who enabled persistent storage still re-downloaded the ~34 GiB checkpoint on every new worker.

There was a sharper version of the same bug: the script set HF_HOME explicitly, which opted out of persistence it would otherwise have inherited. Deadline Cloud points the worker's home directory at the persistent volume, so HuggingFace's own ~/.cache/huggingface default would already have landed there.

The tmpfs warning was also misattributed. Mounting /tmp as a RAM-backed tmpfs capped at 50% of RAM is an Amazon Linux 2023 default, not something Deadline Cloud does. That cap explains the original failure: this job requires a 64 GiB worker, so /tmp tops out near 32 GiB and cannot hold a 34 GiB checkpoint. AL2023 keeps /var/tmp on the root file system, so the old default was right for space and wrong for persistence.

Change

HFCacheDir now defaults to empty and the script resolves the location at runtime:

1. An explicit HFCacheDir / --hf-cache-dir always wins (local runs, custom fleet layouts)
2. Else $DEADLINE_PERSISTENT_MOUNT/wan22_hf_cache — reused across workers
3. Else /var/tmp/wan22_hf_cache on the root volume, logging that each new worker will re-download

Fleets without persistent storage keep working; they just log the cost. HFCacheDir stays a STRING rather than a PATH, so weights remain out of job attachments.

Testing

Submitted to a GPU queue (NVIDIA L4, AL2023) with identical job parameters and no template edits between runs:

Fleet config	Volume seen	Cache path
No persistent storage	249.9 GiB total	/var/tmp/wan22_hf_cache (root EBS)
Persistent storage (200 GiB)	199.9 GiB total	/mnt/persistent/wan22_hf_cache

Both jobs SUCCEEDED and produced a valid H.264 MP4 (640x352, 17 frames, 24 fps). The reported volume sizes confirm each run cached on the intended disk. Job attachments uploaded 1 file (the script only), so weights stayed out of S3.

Also: python3 scripts/validate_repository.py (35 tests) and openjd check pass.

Not verified on a live worker: cross-worker cache reuse. Both runs got a fresh worker and empty volume (0.0 GiB already cached), so the warm-volume path is covered by the existing _snapshot_is_complete logic and local testing rather than an observed run.

[github-actions]

Please verify that Deadline Cloud actually sets an environment variable named DEADLINE_PERSISTENT_MOUNT on service-managed fleet workers. I could not find this variable referenced anywhere else in this repo or in the Deadline Cloud user/developer guides. Notably, the sibling hf_finetune_lora sample points its HF cache at a hardcoded /mnt/persistent/hf_cache path rather than reading any such variable, which suggests the actual persistent-volume mount path is a fixed location, not an env var.

If the variable is never set, _resolve_cache_dir always takes the /var/tmp fallback branch, the entire persistent-volume feature this PR adds is dead code, and every worker silently re-downloads ~34 GiB — the opposite of the PR's stated goal, with only a log line to indicate it. Since the whole change hinges on this name, it is worth confirming against a live worker or the Deadline Cloud docs before merging (and aligning with how hf_finetune_lora locates the volume).