Persistent Model Cache

LLMKube includes a persistent model cache that avoids re-downloading models when InferenceServices are deleted and recreated.

Overview

Without persistent caching, models are downloaded via init container every time a pod starts. For large models (13B-70B), this means 26-40GB+ downloads taking 10-30+ minutes each time you recreate a deployment.

With persistent caching:

A PVC is created automatically in each namespace where you deploy models
Models are downloaded once to the namespace's PVC
Subsequent pods mount the cache and skip download
Delete/recreate cycles complete in seconds

Architecture

LLMKube uses per-namespace PVCs for model caching. This provides:

Namespace isolation: Each namespace has its own cache
No cross-namespace dependencies: Models work independently
Simple RBAC: No need for cross-namespace access

┌─────────────────────────────────────────────────────────────┐
│                 Namespace: production                        │
│  ┌─────────────────────────────────────────────────────┐    │
│  │          llmkube-model-cache PVC                     │    │
│  │  /models/<cache-key>/model.gguf                     │    │
│  └─────────────────────────────────────────────────────┘    │
│           ▲                              ▲                   │
│           │ (init container writes)      │ (read-only)       │
│           │                              │                   │
│  ┌────────┴────────┐        ┌────────────┴──────────────┐   │
│  │  First Pod      │        │  Subsequent Pods          │   │
│  │  - Downloads    │        │  - Mount cache read-only  │   │
│  │  - Caches model │        │  - Skip download          │   │
│  └─────────────────┘        └───────────────────────────┘   │
└─────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────┐
│                 Namespace: staging                           │
│  ┌─────────────────────────────────────────────────────┐    │
│  │          llmkube-model-cache PVC                     │    │
│  │  /models/<cache-key>/model.gguf                     │    │
│  └─────────────────────────────────────────────────────┘    │
│                          ▲                                   │
│                          │                                   │
│                 ┌────────┴────────┐                         │
│                 │  Pods in staging │                         │
│                 └─────────────────┘                         │
└─────────────────────────────────────────────────────────────┘

Deploying to Any Namespace

You can deploy models to any namespace using the CLI:

# Deploy to production namespace
llmkube deploy llama-3.1-8b --gpu -n production

# Deploy to staging namespace
llmkube deploy llama-3.1-8b --gpu -n staging

# Deploy to default namespace
llmkube deploy llama-3.1-8b --gpu

The controller will automatically:

Create a llmkube-model-cache PVC in the target namespace (if it doesn't exist)
Configure the pod's init-container to download the model to the PVC
Mount the PVC read-only for the main container

Note: Each namespace has its own PVC, so the same model deployed to multiple namespaces will be downloaded once per namespace.

Cache Key

Models are cached using a SHA256 hash of the source URL (first 16 characters). This means:

Models with the same source URL share cache entries
Changing the source URL creates a new cache entry
The cache key is stored in Model.Status.CacheKey

Example:

Source: https://huggingface.co/TheBloke/Llama-2-7B-GGUF/resolve/main/llama-2-7b.Q4_K_M.gguf
Cache Key: a3b8c9d4e5f67890
Path: /models/a3b8c9d4e5f67890/model.gguf

Multi-file cache validation and shared volumes

For multi-file downloads using IfNotPresent, an unpinned cached file is checked against its recorded staged size before it is reused. Older entries without a valid size record are kept without contacting the origin, preserving the existing revision even if upstream has changed. Their current local size is recorded on a best-effort basis, except on FUSE where serving pods do not write to warm caches; there the prefetch stager records it, so a FUSE cache staged before size records existed stops warning once a stager has run. This baseline detects subsequent size changes; it cannot prove that a legacy file was complete before the upgrade. A mismatch with a valid record triggers a fresh staging attempt; a failed download fails the init container. This also applies to unpinned files in a partially pinned spec.fileSha256 manifest.

After a failed refresh, OnChange checks the cached file against its own size record, not the current origin size: a different origin size means the cache is stale, not corrupt. Without a valid record it keeps the cached copy with a warning. A mismatch with a valid local record still fails the init container. Checksum-pinned files must still pass their local digest check. Size checks cannot detect corruption of the same length; use spec.fileSha256 for content verification.

The downloader checks the size again after publishing and removes the destination, checksum stamp, and staged size record if publication left the wrong size. Size records are written using a temporary file and rename so concurrent readers cannot observe a partially written number. The same writer produces checksum stamps and rejection markers; its temporary files are named <file>.llmkube-tmp.XXXXXX. A pod that dies mid-write can leave one behind, so every downloader allowed to write the cache (any pod off FUSE, the prefetch stager on FUSE) removes such files once they are more than ten minutes old, along with the older <file>.llmkube-size.XXXXXX and <marker>.sha256-rejected.XXXXXX names. A stamp temporary from before this naming, <file>.sha256.XXXXXX, is not swept because that pattern can match a file a repository ships; delete one by hand if you find it.

OnChange still revalidates each file against the current origin. A mutable source can therefore change shards independently during a refresh. Pin the source to an immutable revision (for example an HF commit) or pin every file with spec.fileSha256 when all shards must come from the same revision.

Concurrent staging requires filesystem locks that work across writer nodes. ReadWriteMany alone does not guarantee this. BlobFuse lists flock as an unsupported operation. Behavior change: serving pods refuse to download missing or mismatched artifacts on FUSE mounts, for single-file models as well as multi-file ones (single-file models since #2254). Warm caches remain usable, including when the origin is offline; the serving downloader does not sweep temporary files or create a lock file on these mounts. Under RefreshPolicy: OnChange, a serving pod that finds the upstream changed keeps a cached copy that passes its local checks and logs Upstream changed for <file>; a serving pod does not stage on a FUSE cache, so it kept the cached copy; it serves the old revision until a prefetch stager refreshes the cache. Detection uses the Linux FUSE magic value, which also covers virtiofs (including Kata Containers volumes). It reads the magic with stat -f; a custom init image whose stat lacks -f cannot detect FUSE, so it stages under the directory lock as before and logs a warning, which leaves a FUSE cache unprotected across nodes. The filesystem type alone cannot establish cross-node locking support. Upgrading to a release with this change rolls every cache-backed serving pod once, because the downloader's command changes. A refusal sets ModelTransferHealthy=False with reason ModelCacheLockUnsupported. The condition message carries the same remedy the downloader logs, which depends on the source kind (prefetch for HTTP/HF sources; storage with cross-node flock or a pvc:// source for s3:// and local sources).

For HTTP/HF sources, set spec.modelCache.prefetch: true on every InferenceService using such a claim. The operator's prefetch Job is allowed to stage on FUSE without relying on flock. It waits for a failed pod before creating a replacement, so a terminating writer is not overlapped by its retry. That wait uses the Job's podReplacementPolicy, which needs the JobPodReplacementPolicy feature gate (on by default from Kubernetes 1.29); with the gate off, the API server drops the field and a retry can start while the failed pod is still terminating. Operator-driven Job replacement waits for the old Job's pods to be deleted either way. If an old pod is stuck on a NotReady node, recover the node or ensure its writer has stopped before force-deleting that pod to unblock replacement. Serving pods wait for successful staging into their claim, then mount the cache read-only. See Letting the service require a prefetch.

s3:// and local sources do not support prefetch. Use storage with cross-node flock support, or stage the files separately and use a pvc:// source. The operator runs one prefetch stager at a time per cache directory and claim, so separate Models sharing a cache key take turns (see Prefetch); avoid writers it does not manage, such as a hand-run Job mounting the claim, while a prefetch Job stages its directory. Other filesystems still use flock; their cross-node lock semantics must be supported by the storage provider.

Integrity Verification (spec.sha256)

Setting spec.sha256 on a Model makes the download init container verify the artifact's bytes against the expected digest before anything becomes the cache. A mismatch fails the init container, so the pod never starts the inference container with bad weights.

The transfer lands in a partial file and is renamed onto the final path only after the hash matches. On a mismatch the partial is discarded and the start fails, so bad bytes never become the cache.
A rejected publish leaves a <model>.<sha256>.sha256-rejected marker. The next start fails before any transfer, so a wrong pin cannot re-download a multi-gigabyte artifact on every kubelet backoff. The marker is keyed on the expected digest: correcting spec.sha256 makes it inert, and another Model sharing the cache directory cannot clear it. Delete it to force a retry. This mirrors the metal-agent's digestMismatchMemo, without its in-process lifetime.
On RefreshPolicy: OnChange, an upstream that has moved past the pin keeps the pinned cached copy and exits 0; the marker then makes later starts skip the re-fetch, so a moved upstream does not re-download on every rollout. The log names the digest mismatch rather than blaming reachability.
A verified artifact gets a <model>.sha256 stamp recording the digest, the file size and its modification time, so a stamp can only vouch for the exact bytes that were hashed. A later start skips re-hashing gigabytes only while all three still agree; a missing, stale or differently-sized file is hashed once and re-stamped. The stamp is written to a temp file and renamed, so a concurrent reader never sees a truncated one. The metal agent's model store writes and reads the same triple, so a stamp from either side is legible to the other, and a bare-digest stamp from an older agent release is hashed once and rewritten.
A cached file that fails the re-hash was corrupted outside any download. It is left in place (the cache directory is keyed on the source alone and may be shared with another Model) and the same start replaces it with a fresh verified download.
The gates fail closed: a download built for a digest-pinned Model aborts if the digest does not reach the container instead of transferring unchecked.
Verification also runs on prefetch Jobs, which reuse the same init container.
A pre-staged pvc:// source with spec.sha256 verifies the mounted file in an init container at pod start. The mount is read-only, so no stamp is written: the file is hashed on every pod start. A mismatch fails the pod. The hash is added to every pod start, rollout, reschedule and scale-out: expect roughly (file size / hash throughput), which for a 100-400 GB artifact on a network PVC can be many minutes. oci:// sources stay digest-addressed and are rejected with spec.sha256.
Combining spec.sha256 with multi-file staging (spec.files / spec.mmproj) or an oci:// source is rejected at admission; use spec.fileSha256 for a multi-file set. This is a behavior change for a Model that set such a combination before this release: the digest was accepted and silently ignored then. Remove the ignored spec.sha256 (or the multi-file / oci:// source) to keep the Model admissible.

Multi-file staging (`spec.fileSha256`)

spec.sha256 attests to a single artifact, so it cannot pin a multi-file staging set. Use spec.fileSha256, a map from repo-relative path to digest:

spec:
  source: hf://org/repo-GGUF
  files:
    - Model-00001-of-00002.gguf
    - Model-00002-of-00002.gguf
  fileSha256:
    Model-00001-of-00002.gguf: 9f2c...   # 64 hex characters
    Model-00002-of-00002.gguf: 1ab3...

Only listed files are verified; a file with no entry is staged as before, so a partially pinned set is allowed. An unpinned shard in a partially pinned set gets no protection, so pinning the whole set is recommended.
spec.files is capped at 256 entries of 512 characters each.
Each pinned file is hashed before it is renamed onto the cache. A mismatch discards that file's partial and records its own <file>.<sha256>.sha256-rejected marker, and the loop stops. Once the upload is fixed, delete the marker (<file>.<sha256>.sha256-rejected) to let the next start re-download.
The per-file stamp and fail-closed behavior match the single-file path.
Every key must name a spec.files entry or spec.mmproj. Globs in spec.files are rejected when fileSha256 is set, because a digest cannot be keyed to an unknown expanded name. spec.sha256 and spec.fileSha256 are mutually exclusive, and fileSha256 is rejected on an oci:// source.
With spec.downloadConcurrency above 1, each hash is logged: the file and the expected digest when it starts, then the computed digest with the file's size and the hashing time, and the rename's time once the file is published. A stamp hit names the file it skipped. Without it the per-hash lines are absent, while the stamp-hit and mismatch lines stay what they were, so the logging adds no template change of its own.

Staging files concurrently (`spec.downloadConcurrency`)

By default the downloader stages spec.files one at a time: each file is downloaded, closed, hashed if pinned, and renamed into place before the next one starts. Set spec.downloadConcurrency (1 to 16) to stage that many files at once, so one file's close, hash and rename overlap the next file's transfer:

spec:
  source: hf://org/sharded-checkpoint
  files:
    - model-00001-of-00048.safetensors
    # ...
  downloadConcurrency: 4

It requires spec.files and a source the downloader stages. A single-object Model, and a pvc:// or oci:// source, which is mounted rather than downloaded, are rejected at admission.
Each progress line names its file and the files in flight at that moment, for example Downloaded 1906 MiB of 7452 MiB of model-00004-of-00048.safetensors (25%) (concurrency: 3). The pool logs its limits when it starts.
The first file that fails stops new files from starting. The files already in flight finish, then the init fails, and the failed files' errors are logged again as the last lines, where the container's termination message and the ModelTransferHealthy condition read them. Partials stay for the next attempt to resume.

On a FUSE cache that keeps each open file on the node's disk, such as blobfuse2's file cache, every file in flight needs its full size free on the node's disk. A file count alone cannot fit shards of different sizes, so spec.downloadMaxInFlightSize caps the bytes in flight:

spec:
  downloadConcurrency: 4
  downloadMaxInFlightSize: 90Gi

A file starts only while the size in flight plus its own fits the cap. A file larger than the cap counts as the cap, so it starts once nothing else is in flight and stages alone. Files start in list order: one that does not fit holds back the ones after it, and the pool logs Waiting to start <file>: ... with the sizes.
How a file is counted:
- Under IfNotPresent, a file already in the cache whose size record agrees with it counts its size on disk, with no request to the origin; blobfuse2 holds a file it only reads, too. One without an agreeing record may be replaced in full, so it counts the whole cap.
- Under OnChange, every file counts the origin's Content-Length, since a changed file is replaced at that size.
- A missing file counts the origin's Content-Length, from a HEAD request limited to 30 seconds. A size the origin does not report in time counts the whole cap.
The value takes up to 15 digits of bytes, or up to 6 digits with a suffix up to Ti (90Gi, 100G), and requires downloadConcurrency above 1. Set it to the node disk the cache may fill, below the disk's free space with the node's other writers in mind.
Changing the value changes the downloader's pod template, so serving pods that stage the Model roll. Unset and 1 build the same template.

Prefetch (Eager Download)

By default a Model with a remote source is only a declaration: nothing is downloaded until the first InferenceService referencing it starts, and the download then runs in that pod's init container, on the serving node. Set spec.prefetch: true to have the operator stage the artifact into a model cache claim ahead of the serving pods:

apiVersion: inference.llmkube.dev/v1alpha1
kind: Model
metadata:
  name: llama-3b-prefetch
spec:
  source: hf://bartowski/Llama-3.2-3B-Instruct-GGUF/Llama-3.2-3B-Instruct-Q4_K_M.gguf
  format: gguf
  prefetch: true

The controller runs an owner-referenced download Job that reuses the same init-container downloader the serving path uses. status.phase moves Downloading -> Ready; once Ready, the first InferenceService starts from a cache hit instead of a cold download. A glob in spec.files is expanded from the repository listing before the Job is built, exactly as it is for the serving pods, so a multi-file checkpoint such as files: [config.json, "*.json", "*.safetensors"] prefetches too. With --file-listing=pod that listing runs in a short-lived Job labeled and placed like the prefetch Job, so the controller needs no egress to the model host (see "Listing from a pod instead of the controller" in the model cache guide).

Where the download lands

The Job stages into one claim, chosen in this order:

spec.modelCache.claimName on the Model, when set. The Job writes there and never falls back, whatever services exist.
Otherwise the single spec.modelCache.claimName the referencing InferenceServices agree on (#1676).
Otherwise the namespace's shared cache (llmkube-model-cache, created on demand).

spec.modelCache on the Model takes the same fields as an InferenceService's: claimName, persistence, storageClassName and prefetch (an alias of spec.prefetch). It is also the default for the services that reference the Model: a service with no modelCache block of its own resolves the Model's, so the claim is declared once. A service that sets its own block still wins. persistence: Ephemeral cannot be combined with prefetch; the API server rejects it.

Every claim the Model has been staged into is recorded in status.prefetch, one entry per claim with the Job name, the phase (Pending, Running, Succeeded, Failed) and a fingerprint of what was staged. A spec change that alters the files (the source, files, a revision, fileSha256) runs a new prefetch; one that does not (placement, tolerations) only refreshes the entry. Deleting a finished Job retries it.

Prefetch stagers run one at a time per cache directory and claim. The cache directory is keyed on the source alone, so two Models on one source share it, and on a FUSE claim a stager writes without flock. While another prefetch Job (or a pod left from one) for the same directory and claim is still active, the operator does not create the next one: that Model's entry stays Pending, PrefetchWaiting=True (reason PrefetchWaitingForStager) names the Job and the Model it waits for, and the Model checks again every 15 seconds. Once the first Job has finished, the second starts and finds the files it already staged.

Two conditions report what the field could not do:

PrefetchSkipped=True (reason SourceNotPrefetchable): spec.prefetch is set on a source the downloader does not stage (local paths, pvc://, oci://, s3://).
PrefetchTargetMismatch=True (reason ClaimMismatch): a referencing service mounts a claim the Model was not staged into, so that service downloads at serve time. Once a download has completed the operator does not move it: a service that appears later with another claim is reported here, and gets a PrefetchTargetMismatch warning event of its own. Set modelCache.prefetch on that service, or modelCache.claimName on the Model, to stage there. While the Job is still running, a service that appears with another claim retargets it instead.

Letting the service require a prefetch

An InferenceService can ask for its own staging with spec.modelCache.prefetch: true, whether or not the Model sets spec.prefetch:

apiVersion: inference.llmkube.dev/v1alpha1
kind: Model
metadata:
  name: qwen2-5-0-5b-instruct
spec:
  source: Qwen/Qwen2.5-0.5B-Instruct   # about 1 GiB; a gated repo adds sourceSecretRef
  format: safetensors
  files: [config.json, "*.json", "*.safetensors"]
  modelCache:
    claimName: model-cache
  prefetchAffinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - {key: nvidia.com/gpu.present, operator: DoesNotExist}
---
apiVersion: inference.llmkube.dev/v1alpha1
kind: InferenceService
metadata:
  name: qwen
spec:
  modelRef: qwen2-5-0-5b-instruct
  runtime: vllm
  modelCache:
    claimName: model-cache
    prefetch: true

With the field set:

The Model controller runs a prefetch Job for this Model into the claim this service's pods mount (claimName, else the Model's modelCache.claimName, else the operator cache), placed by the Model's prefetchNodeSelector, prefetchTolerations and prefetchAffinity. The download happens on the nodes meant for downloads, never on the serving nodes. One Model can be staged into several claims; each has its own Job (<model>-prefetch-<claim hash>) and status.prefetch entry.
The service creates no pods until that prefetch has succeeded for the Model's current spec, and reports ModelPrefetched=False with reason WaitingForPrefetch (also in status.schedulingStatus) meanwhile, or WaitingForModel while the Model has not derived its cache key yet. A later change that needs new weights holds the same way: the live Deployment keeps serving the old template until the new prefetch completes, then rolls. A failed prefetch shows as PrefetchFailed and the workload stays as it was; delete the Job to retry. While a hold is on, a replica change other than to zero waits with the template, and a multiNode group that loses a member is recreated only once the prefetch finishes.
Once staged, the serving pods only consume the cache: the claim is mounted read-only, there is no model-cache-prep or model-downloader init container, and a model-verify init container checks the files the Job recorded on the entry are present (and hashes a single file against spec.sha256 when one is pinned). If a file is missing the pod fails with PrefetchedFilesMissing (surfaced on the ModelTransferHealthy condition) rather than downloading on the serving node. The prefetch sets no Hub-offline variable: set spec.hubOffline: true when the runtime must not reach the Hub for anything the Job did not stage (see the air-gapped guide).
The Job writes the cache as the init image's user (--init-container-user) in the operator's group (--default-fsgroup), whatever the requesting service's podSecurityContext says, and its prep hands the cache root to that group. Because the serving pods mount the claim read-only, the kubelet does not apply their own fsGroup to it. So a pod that reads a prefetched cache also runs in the Job's group: unless that group is already the pod's fsGroup, its runAsGroup or one of its supplementalGroups, the operator adds it to the pod's supplementalGroups. That covers a podSecurityContext with its own fsGroup and one with none. The service's spec is not changed. With --default-fsgroup 0 nothing is added, because the prep then hands the root to group 0. If model-verify still cannot read a staged file or enter its directory, it fails with PrefetchedFilesUnreadable, naming the pod's uid and groups and the blocking path's owner and mode. That is an ownership problem, so unlike PrefetchedFilesMissing it does not trigger a re-stage.
Use a ReadWriteMany (or ReadOnlyMany) claim when the serving pods run on other nodes than the Job. A ReadWriteOnce claim attaches to one node: the first staging works because the Job finishes before the pods attach, but a later re-stage or re-verification while pods mount the claim elsewhere stalls on Multi-Attach and is reported as PrefetchStalled. On a single-writer claim the Model therefore skips the routine re-verification while a pod mounts the claim (the entry says so and stays usable), and a re-stage after PrefetchedFilesMissing still runs but needs the consuming service suspended (spec.suspend) to release the attachment when the Job lands on another node.
prefetch: true means "serve only from a prefetched cache". A request the pods could not consume is refused, not ignored: the service reports ModelPrefetched=False with reason PrefetchUnsupported, creates no pods (or keeps its live Deployment as it is), and no Job runs. That covers model caching disabled on the operator, a runtime the operator does not stage (TGI, generic without stageModel, skipModelInit: true), a spec.speculativeDecoding.draftModelRef or a spec.vllmConfig.speculative.model (the draft is not prefetched, so its downloader would run on the serving node), a multiNode member with its own modelCache.claimName (that member's pod mounts a claim the prefetch does not fill), and a source the downloader does not stage ahead of time (s3://). The field is PrefetchNotApplicable, and not waited for, only where nothing downloads at serve time: Metal, pvc://, oci:// and local sources.

Applied together by GitOps, the pair above is safe in either order: the Model names the claim, the service inherits or repeats it, and the service waits for the staging it asked for. The request itself is not inherited: modelCache.prefetch on the Model is spec.prefetch, which stages the Model's own claim and holds no service, so a service that must wait sets modelCache.prefetch in its own block.

Placing the prefetch pod

A node-pinned cache (a WaitForFirstConsumer local-path claim) binds to whichever node the prefetch pod schedules to. spec.prefetchTolerations, spec.prefetchNodeSelector and spec.prefetchAffinity steer that placement so the model lands on the node that will serve it, or stays off the GPU nodes while it downloads:

spec:
  prefetch: true
  prefetchTolerations:
    - key: example-gpu
      operator: Exists
  prefetchNodeSelector:
    example.com/node-pool: gpu
  prefetchAffinity:
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 100
          preference:
            matchExpressions:
              - {key: example.com/node-pool, operator: In, values: [downloads]}

All three govern only where the download runs, never where the model is served.

Placement has to reach the claim. A claim on one node's local disk (a WaitForFirstConsumer local-path volume that is already bound, a hostPath PV) can only be filled from that node, so a rule that keeps the Job off the GPU nodes leaves it unschedulable when the claim lives on one. The Job pod requests no GPU, so pinning it to the claim's node with prefetchNodeSelector still keeps the download out of the serving pods; an off-GPU rule is for a claim every node can reach (an RWX class, a blob mount). A Job whose pod cannot be scheduled never completes and never fails: the Model stays Downloading, its status.prefetch entry and Progressing condition carry the scheduler's message (reason PrefetchUnschedulable, for example volume node affinity conflict), and a PrefetchUnschedulable warning event is recorded. A pod that was scheduled but has not started within five minutes (an image pull, a volume that cannot attach) is reported as PrefetchStalled, with the waiting reason of the container that blocks. A downloader init container that is running is not a stall: Kubernetes keeps a pod with init containers in phase Pending until all of them finish, so a download in progress reads as downloading. Fix the placement or the claim, then delete the Job to retry.

Limitations:

Prefetch applies to remote sources (https://, hf://, and a bare repository ID with spec.files). Local paths, pvc://, oci:// and s3:// sources set PrefetchSkipped instead.
In perService cache mode the operator-created <isvc>-model-cache claim binds where the first consumer schedules. A Model-level spec.prefetch targets the shared cache there and is reported as a PrefetchTargetMismatch; modelCache.prefetch on the service stages into the per-service claim, with the Job as that first consumer, so set the Model's prefetch placement to the node the service will run on.
A failed prefetch Job sets status.phase: Failed on a Model that asked for it; check the Job's pod logs (the name is in status.prefetch). The Job self-cleans 24 hours after finishing and is garbage-collected with the Model.
There is no timeout on a prefetch Job. A pod that cannot be scheduled is reported as above; a pod that is scheduled but stalls (a slow or hung source) is visible only in its logs.

When the staged files disappear

status.prefetch records that a claim was filled; it cannot see the claim afterwards. If the files go away (llmkube cache clear, a claim deleted and recreated under the same name, a wiped volume), two things bring them back without a rollout:

The serving pods of a service with modelCache.prefetch fail model-verify with PrefetchedFilesMissing, which the service reports on ModelTransferHealthy. The service then deletes the Model's prefetch Job for its claim, the Model stages again, and the failing pods pass model-verify on their next init retry. This happens once per failure episode: a claim the serving node cannot reach at all stays reported rather than re-staged in a loop.
The Model re-verifies a claim whenever its prefetch Job is gone: a day after it finished (the Job's TTL), or when someone deletes it. With the files still in place that is a cache hit and completes in seconds. The entry stays Succeeded and the Model Ready while the check runs, so no service is held for it; the entry's message and the Model's Progressing condition (reason PrefetchReverifying) show it, and only a failed check moves the entry to Failed. So "delete the Job" is always an available retry, and a wiped claim is refilled within a day even with no pods to notice.

See config/samples/model_prefetch.yaml and config/samples/model_prefetch_gitops.yaml for complete examples.

Configuration

Helm Values

modelCache:
  # Enable persistent model cache (default: true)
  enabled: true

  # Provisioning mode: shared or perService.
  # shared (default): a single cluster-wide PVC that all InferenceServices share.
  # perService: operator-managed per-InferenceService PVC that binds on the node
  #   where the inference pod schedules. A separate user-managed override is
  #   available via spec.modelCache.claimName on individual InferenceServices.
  mode: shared

  # Storage size for model cache
  size: 100Gi

  # Storage class (leave empty for default)
  storageClass: ""

  # Access mode
  # - ReadWriteOnce: Single-node clusters
  # - ReadWriteMany: Multi-node clusters (requires NFS, EFS, etc.)
  accessMode: ReadWriteOnce

  # Mount path inside controller pod
  mountPath: /models

  # PVC annotations (e.g., for backup policies)
  annotations: {}

Multi-Node Clusters

For multi-node clusters where pods may run on different nodes, you need a storage class that supports ReadWriteMany:

AWS EKS (EFS):

modelCache:
  storageClass: efs-sc
  accessMode: ReadWriteMany

GKE (Filestore):

modelCache:
  storageClass: filestore-standard
  accessMode: ReadWriteMany

Azure AKS (Azure Files):

modelCache:
  storageClass: azurefile-premium
  accessMode: ReadWriteMany

On-Premise (NFS):

modelCache:
  storageClass: nfs-client
  accessMode: ReadWriteMany

The volume must support file locking. Downloaders that share a cache directory take turns through flock on .llmkube-download.lock, so pods on different nodes rely on the volume to enforce that lock across nodes. NFSv4 and CephFS provide it. An NFSv3 mount with nolock does not, so two pods downloading the same model at once can corrupt it. Init images that override the default must provide flock.

Strictly tainted GPU nodes

When a GPU node carries a hard NoSchedule taint, dynamic provisioning can fail for reasons unrelated to the inference pod's tolerations. The safest approach is to pre-provision a PVC bound to the tainted node and reference it through spec.modelCache.claimName.

The inference pod already receives the GPU toleration derived from the Model's hardware.gpu configuration. The model-downloader runs as an init container in that same pod, so a pre-existing PVC does not require a separate download Job.

The claim must already exist and be node-aligned by the cluster administrator's storage setup. claimName is user-owned: LLMKube mounts it through the same prep and download init containers as the built-in cache, but never creates, mutates, or deletes it. If the named PVC is missing, the InferenceService is marked Degraded rather than silently falling back to the shared cache.

Note: claimName is ignored for pvc:// model sources because those weights are already staged on the cluster and mounted read-only; no download occurs.

Some dynamic provisioners create a per-node helper pod to provision volumes. This helper pod is separate from the LLMKube inference pod and typically carries only the default not-ready/unreachable tolerations. When a GPU node has a hard NoSchedule taint, the helper pod may remain Pending with an untolerated-taint event — even though the inference pod itself has the correct GPU toleration.

Strict-taint choices:

Pre-provision a static, node-aligned claim and use claimName.
Use a provisioner whose helper configuration supports the taint.
Apply a narrowly scoped cluster-level policy maintained by the cluster administrator.

Recognising a volume the provisioner never stamped

The failure above is worth calling out separately because it does not look like a storage failure. Some provisioners publish a PersistentVolume even when their helper never ran, carrying a node affinity that no node satisfies:

$ kubectl get pv <pv> -o jsonpath='{.spec.nodeAffinity}'
{"required":{"nodeSelectorTerms":[{"matchExpressions":[
  {"key":"kubernetes.io/hostname","operator":"In","values":[""]}]}]}}

An empty value in that In term is the fingerprint: node label values are never empty, so the volume can never be scheduled anywhere. The PVC still reports Bound, which is why this reads as success.

The consuming pod then sits Pending forever, and the scheduler describes it in one of two ways depending on whether the pod had already been assigned a node:

0/4 nodes are available: 1 node(s) didn't match PersistentVolume's node affinity

running PreBind plugin "VolumeBinding": binding volumes: pv "pvc-..."
node affinity doesn't match node "gpu-node-1": no matching NodeSelectorTerms

Both name node affinity, which sends you to the scheduler. The scheduler is not at fault: it chose correctly, and the PVC's volume.kubernetes.io/selected-node annotation agrees with it. Provisioning is what failed.

LLMKube reports this directly rather than passing the scheduler's wording through:

$ kubectl get inferenceservice <name> -o jsonpath='{.status.schedulingStatus}'
UnbindableModelCache

$ kubectl get inferenceservice <name> -o jsonpath='{.status.schedulingMessage}'
model cache volume was never provisioned: PersistentVolume "pvc-..." has a node
affinity that matches no node. Claim: <name>-model-cache. ...

To recover, delete the claim so its volume is released, then provision on a storage class whose helper can run on the tainted node. Which objects you have to remove depends on the cache mode:

Cache mode	Deleting the InferenceService clears it
`perService`	Yes, the claim is owner-ref'd and garbage-collected with it
`shared` (default)	No, the shared claim intentionally outlives any one service
`claimName` (bring your own)	No, LLMKube never deletes a user-owned claim

In the latter two cases the unusable volume survives the InferenceService, so delete the claim explicitly before retrying. Recreating the service against a claim that is still bound to a broken volume reproduces the same Pending state.

Choosing a provisioner that tolerates taints

Whether this is fixable by configuration depends on the provisioner:

Provisioner	Helper tolerations configurable
`rancher.io/local-path`	Yes, via `helperPod.yaml` in its ConfigMap
`microk8s.io/hostpath`	No, the helper template is built into the image

For local-path, add tolerations to the helper template so it can run wherever a volume is legitimately requested. This mirrors what a CSI node plugin does:

# local-path-config ConfigMap, helperPod.yaml key
apiVersion: v1
kind: Pod
metadata:
  name: helper-pod
spec:
  priorityClassName: system-node-critical
  tolerations:
    - operator: Exists
  containers:
    - name: helper-pod
      image: busybox

Then point model caches at it:

modelCache:
  storageClass: local-path

microk8s.io/hostpath exposes no equivalent knob, so on a tainted node the remaining options are the pre-provisioned claimName route above or a different storage class.

Per-InferenceService Cache PVC (Bring Your Own)

The cache backend above is an operator-global choice. To point a single InferenceService at its own pre-existing, user-owned PVC — for example a node-local volume for a large model pinned to one node, while everything else rides the shared cache — set spec.modelCache.claimName:

apiVersion: inference.llmkube.dev/v1alpha1
kind: InferenceService
metadata:
  name: llama-3.1-70b
spec:
  modelRef: llama-3.1-70b
  modelCache:
    claimName: my-model-cache   # pre-existing PVC in the same namespace

Behavior:

The named PVC becomes the writable model cache for this workload only: the same model-downloader init container runs against it, and, under modelCache.prep: always, the model-cache-prep one before it. Weights land under the usual <cacheKey>/ subdirectory, and the serving container mounts it read-only. RefreshPolicy and cache-key semantics are unchanged, so multiple models can safely share one claim.
The block can live on the Model instead: a service with no modelCache of its own resolves Model.spec.modelCache, so a claim shared by every service of a model is declared once (see Prefetch above). A service's own block still wins.
The operator never creates or deletes the claim — you own it end-to-end (unlike perService mode, where the operator provisions and garbage-collects <isvc>-model-cache). If the claim does not exist, the InferenceService is marked Degraded with a ModelCachePVCNotFound event instead of silently falling back to the shared cache.
claimName targets the download path, so it is ignored — with a ModelCacheClaimIgnored warning event — for pre-staged pvc:// model sources (mounted read-only, no download), and whenever caching is inactive (chart modelCache.enabled: false, a local file:// source, or a remote model not yet fingerprinted); in the inactive-caching cases the pod falls back to an ephemeral emptyDir and re-downloads on every restart.
claimName is mutable: changing it rolls the Deployment onto the new claim, but the old claim (and the weights on it) is not garbage-collected — clean it up yourself if it is no longer needed.
Node alignment is your responsibility: for an RWO or node-local claim, use nodeSelector so the pod lands where the PVC binds (a WaitForFirstConsumer local class binds on the first consumer; a pre-bound RWO PVC pins the pod).
llmkube cache list discovers the shared cache and operator-managed per-service cache PVCs. A user-managed spec.modelCache.claimName PVC is outside the operator's cache label/discovery contract and may not appear in the listing. Each claim is read through a short-lived read-only pod the command creates and deletes, including a Pending WaitForFirstConsumer claim, which that pod binds. For a ReadWriteOnce or ReadWriteOncePod claim a pod holds, the pod is pinned to that pod's node, so a strict RWO serving pod does not make it stall on Multi-Attach; a ReadWriteMany or ReadOnlyMany claim is never pinned, and a claim nothing holds is not pinned either: the pod is its first consumer. cache list needs permission to list, get, create, exec into and delete pods in the namespace, and exits nonzero when it cannot read any discovered claim. The inspector copies the holder's tolerations, so a tainted node that runs the claim does not reject it.

Per-Service Storage Class

spec.modelCache.storageClassName overrides the operator-global --model-cache-storage-class flag for the cache claim the operator creates, so a mixed cluster can cache on local disk where the cluster's default class cannot attach, while the operator still creates and garbage-collects the PVC:

apiVersion: inference.llmkube.dev/v1alpha1
kind: InferenceService
metadata:
  name: onprem-llama
spec:
  modelRef: llama-3b
  modelCache:
    persistence: Cached
    storageClassName: local-path   # overrides --model-cache-storage-class

Behavior:

In perService mode the field is authoritative for the claim the operator creates, <isvc>-model-cache.
In shared mode the namespace has one claim shared by every service, so the first creator chooses its class.
The class of an existing claim never changes: storageClassName is immutable once bound and the operator never mutates a claim. Editing the field against a claim that already exists (a differing later request in shared mode, or an edit on your own claim in perService mode) is ignored and raises a ModelCacheStorageClassIgnored warning event; delete the claim to reprovision on the requested class.
The field cannot combine with claimName (the claim is user-owned) or with persistence: Ephemeral (no PVC is created to class); both are rejected at admission.
There is no allowlist of storage classes: anyone who can edit an InferenceService can pin its cache claim to any storage class in the cluster. Scope who may edit InferenceServices if storage-class choice is restricted in your cluster.

CLI Commands

List Cached Models

# List cached models in default namespace
llmkube cache list

# List from all namespaces
llmkube cache list -A

Output:

Model Cache Entries
═══════════════════════════════════════════════════════════════════════════════
CACHE KEY         SIZE      MODELS              SOURCE
a3b8c9d4e5f67890  4.1 GiB   llama-2-7b          ...TheBloke/Llama-2-7B-GGUF/...
f1c314277254a2fd  7.2 GiB   llama-3.1-8b        ...meta-llama/Meta-Llama-3.1-8B/...

Total: 2 cache entries, 2 models

Clear Cache

# Clear cache for a specific model
llmkube cache clear --model llama-2-7b

# Remove every entry that no Model references
llmkube cache clear --orphaned

# Clear all cache (with confirmation)
llmkube cache clear

# Force clear without confirmation
llmkube cache clear --force

Entries are removed by a short-lived busybox:1.37.0 pod that mounts the cache claim read-write, one pod per claim, deleted as soon as it is done. On a cluster that cannot pull that image, pass --cache-image <ref> (or set LLMKUBE_CACHE_IMAGE) to cache list, cache clear and delete --purge-cache; the flag wins over the variable. The image needs sh, sleep, ls, rm and a du that takes -sb, as busybox and GNU coreutils do.

A ReadWriteOncePod claim another pod holds cannot be mounted a second time on any node, so cache list and cache clear refuse it at once, naming the access mode and the holding pod, instead of waiting for a pod that can never start. A pod that is already being deleted releases the claim, and so does a cache pod another running llmkube command deletes when it ends, so both are waited for, not refused. A cache pod llmkube left behind at a name the command does not use is named as such, since the command does not delete it: delete it and retry. For this check cache clear needs list pods; without it, it falls back to the old wait, while cache list reports the pod-list failure as it always has. delete --purge-cache waits for every holder, as it did before: the service it deleted takes its pods with it. cache list pins its inspector pod only for a ReadWriteOnce or ReadWriteOncePod claim: a ReadWriteMany or ReadOnlyMany claim is never pinned.

cache clear with no flag and cache clear --orphaned act on the claims cache list discovers: llmkube-model-cache and the labeled per-service <isvc>-model-cache. cache clear --model and delete --purge-cache target only the claims that can hold that model's files: the spec.modelCache.claimName of the InferenceServices that serve the model, the <isvc>-model-cache of those services in perService mode, and the shared cache, which is where their pods mount in the default shared mode (and the only home when the Model has no InferenceService). Two things follow from that:

It needs your own permission to get, create and delete pods in the namespace, and to exec into the one it creates. On a restricted-profile namespace the pod is rejected, the same way cache list is.
A strict RWO claim that a serving pod still holds cannot be attached a second time on another node, so the purge fails with a timeout naming the claim. Delete the InferenceService first, then purge.

The namespace-wide modes remove only directories named like a cache key (16 hex characters). --model and delete --purge-cache validate the key as a single path segment first. Either way the command removes /models/<key> and never data the operator did not write.

Clearing the entry of a model that is still serving removes the file from the volume, but the running process keeps its open handle and serves the old bytes until it restarts; the next start re-downloads.

delete --purge-cache differs from cache clear --model in one case: a perService <isvc>-model-cache is owner-ref'd to the InferenceService, so the delete reports that the claim goes with the service instead of purging it.

Preload Models

Pre-download models before deploying them:

# Preload a catalog model
llmkube cache preload llama-3.1-8b

# Preload to a specific namespace
llmkube cache preload llama-3.1-8b -n production

This is useful for:

Air-gapped environments (pre-populate cache on a connected machine)
Reducing deployment time (model already cached)
Bandwidth management (download during off-peak hours)

Air-Gapped Deployments

For air-gapped environments:

On a connected machine, preload models:

llmkube cache preload llama-3.1-8b
llmkube cache preload mistral-7b

Export the PVC. The weights live in the cache PVC of the namespace you preloaded into, not in the controller pod. The controller's own /models is scratch space, and it never mounts a workload namespace's claim, so copying out of llmkube-system cannot reach them.

Find the claim first. In shared mode (the default) it is llmkube-model-cache; in perService mode it is <inferenceservice>-model-cache:
```
kubectl -n <namespace> get pvc
```
Then mount it in a throwaway pod and copy from there:
```
kubectl -n <namespace> run cache-export --restart=Never --image=busybox:1.36 \
  --overrides='{"spec":{"containers":[{"name":"cache-export","image":"busybox:1.36","command":["sleep","3600"],"volumeMounts":[{"name":"cache","mountPath":"/models"}]}],"volumes":[{"name":"cache","persistentVolumeClaim":{"claimName":"llmkube-model-cache"}}]}}'

kubectl -n <namespace> wait --for=condition=Ready pod/cache-export --timeout=120s
kubectl cp <namespace>/cache-export:/models ./model-cache
kubectl -n <namespace> delete pod cache-export
```
A ReadWriteOnce claim can only be mounted where it is already attached, so scale the InferenceService to zero first if the export pod will not schedule.

On the air-gapped cluster, import the cache by reversing the copy against a helper pod mounting the destination claim:

kubectl -n <namespace> run cache-import --restart=Never --image=busybox:1.36 \
  --overrides='{"spec":{"containers":[{"name":"cache-import","image":"busybox:1.36","command":["sleep","3600"],"volumeMounts":[{"name":"cache","mountPath":"/models"}]}],"volumes":[{"name":"cache","persistentVolumeClaim":{"claimName":"llmkube-model-cache"}}]}}'

kubectl -n <namespace> wait --for=condition=Ready pod/cache-import --timeout=120s
kubectl cp ./model-cache/. <namespace>/cache-import:/models
kubectl -n <namespace> delete pod cache-import

The destination claim must exist before the copy. Deploy the model once and let it fail to download, or create the PVC yourself, so the operator has a claim to bind.

Deploy models (they'll be found in cache):
```
llmkube deploy llama-3.1-8b --gpu
```

Troubleshooting

Model Not Using Cache

If models are still being downloaded via init container:

Check if the Model has a CacheKey:

kubectl get model llama-3.1-8b -n <namespace> -o jsonpath='{.status.cacheKey}'

Verify the controller has cache enabled:

kubectl get deploy -n llmkube-system llmkube-controller-manager -o yaml | grep model-cache

Check PVC exists in your namespace:

kubectl get pvc llmkube-model-cache -n <namespace>

Check if model is cached in the PVC:

kubectl exec -n <namespace> <pod-name> -- ls -la /models/

Cache PVC Full

If the cache PVC runs out of space:

List cache entries:
```
llmkube cache list -n <namespace>
```

Clear unused models:

llmkube cache clear --orphaned -n <namespace>

Then, if that is not enough, a specific model:

llmkube cache clear --model <unused-model> -n <namespace>

Or resize the PVC (if your storage class supports it):

kubectl patch pvc llmkube-model-cache -n <namespace> \
  -p '{"spec":{"resources":{"requests":{"storage":"200Gi"}}}}'

Cache Corruption

If you suspect cache corruption:

Clear the specific cache entry, which removes the directory from the PVC:
```
llmkube cache clear --model <model-name> --force -n <namespace>
```

Delete and recreate the InferenceService to trigger re-download:

kubectl delete inferenceservice llama-3.1-8b -n <namespace>
kubectl apply -f inferenceservice.yaml

Or delete and recreate the Model:

kubectl delete model llama-3.1-8b -n <namespace>
kubectl apply -f model.yaml

Performance Considerations

Storage Performance: Use SSD-backed storage for faster model loading
Network: For ReadWriteMany, ensure low-latency network between nodes and storage
Cache Size: Plan for 1.5-2x your total model sizes to allow for cache rotation

Disabling Cache

To disable persistent caching (not recommended):

# values.yaml
modelCache:
  enabled: false

This will revert to the legacy behavior where each pod downloads the model via init container.

Security Considerations

Local and hostPath model sources (host-path allowlist)

Besides remote downloads, the model storage machinery can serve node-local sources: a Model whose spec.source is an absolute path (/srv/models/model.gguf) or a file:// URL is mounted into the inference pod via a hostPath volume. As of v0.9.0, these sources are gated by an operator-configured allowlist (GHSA-jw3m-8q7m-f35r).

The allowlist is empty by default, which disables local and hostPath sources entirely. With the default configuration, a Model with a local source is rejected: it goes to phase Failed with a SourceNotAllowed condition, nothing is fetched, and no hostPath volume is created. Remote sources (https://, pvc://, hf://) are unaffected.

To keep serving models from node-local disk, list the absolute directory roots you trust:

# values.yaml
modelSource:
  allowedHostPathRoots:
    - /srv/models

The equivalent controller flag is --allowed-host-path-roots (comma-separated). A source is permitted only when its cleaned path is within one of the configured roots; .. escapes are rejected. The check is lexical and does not resolve symlinks, so only allowlist roots whose contents you control.

Automatic shared cache on `fsGroupPolicy: None` backends

Whether Kubernetes applies the pod fsGroup to a volume depends on the CSI driver's fsGroupPolicy (kubectl get csidriver <name> -o jsonpath='{.spec.fsGroupPolicy}'), not on the filesystem. ceph-csi's CephFS driver defaults to File, so fsGroup is applied; a deployment can set its driver to None (some Rook CephFS installs do). NFS drivers vary, so check yours.

The automatic shared-model-cache workflow is a first-class supported path, including on fsGroupPolicy: None CSIs. On those backends Kubernetes does not apply the pod fsGroup to the volume, so the PVC root stays root:root and the non-root model downloader cannot write to it. The model-cache-prep init container fixes the ownership for you so the cache just works. You do not need to pre-stage models or hand-manage a PVC.

The prep is the compatibility implementation for these backends, and the operator adds it on every cache-backed pod and prefetch Job by default (modelCache.prep: always). On a File driver kubelet has already applied the ownership and the prep only re-asserts it, so it is redundant there: set modelCache.prep: never to drop it (see below). It runs as non-privileged root (uid 0) with ALL capabilities dropped and only CHOWN+FOWNER added (no privilege escalation, read-only rootfs, seccomp RuntimeDefault). Root here is a backend-driven requirement, not an avoidable misconfiguration: chowning a root-owned mount needs CAP_CHOWN, the prep shells out to chown, and a non-root process loses its capabilities across execve (Kubernetes has no ambient-capability field), so the exec'd chown would fail EPERM. Root retains its capabilities across exec. (Running the prep non-root broke the shared cache on these backends in 0.8.20; reverted in 0.8.21. A future change may move the chown into a small dedicated helper that performs the syscall in its own entrypoint, which would let the prep run non-root again without the exec capability loss.)

What the prep sets the cache root to

The prep changes only the mount root (/models), never recursively. What it sets depends on whether the pod has an fsGroup, resolved as the InferenceService's own spec.podSecurityContext.fsGroup when set, otherwise --default-fsgroup:

fsGroup	Prep command	Who can write the root
`> 0` (default `102`)	`chown 0:<fsGroup> /models && chmod g+rwX /models`	every container in the pod, through the `fsGroup` supplementary group
`<= 0` (the OpenShift preset, `--default-fsgroup=0`)	`chown <uid>:0 /models && chmod 770 /models`	the downloader, as owner, and any container whose group is `0`

<uid> is the uid the model downloader actually runs as: the InferenceService's spec.podSecurityContext.runAsUser when it sets one (the downloader inherits it), otherwise --init-container-user (chart value controllerManager.initContainer.user, default 101, curl_user in the default curlimages/curl image). The operator cannot read an image's user without a registry call, so if you mirror a different --init-container-image, set --init-container-user to the uid that image runs as (docker run --rm <image> id). The flag refuses 0 and negative values. Group 0 is the group OpenShift runs SCC-assigned arbitrary uids under, so a container whose uid comes from the namespace range still reaches the root through the group bits. Mode 770 grants nothing to other users.

Upgrade note: caches prepared as `100:100` (#2204)

Before this change the fsGroup <= 0 branch ran chown 100:100 /models && chmod 770 /models, but the default init image runs as uid 101, so on storage that does not apply an fsGroup the downloader could not write the root it had just been given. On upgrade, with fsGroup <= 0 and modelCache.prep: always:

A cache root previously prepared as 100:100 mode 770 is re-owned to <uid>:0 (default 101:0), mode 770, on the next prep run: the next start of a cache-backed serving pod or prefetch Job. Nothing on the cache is migrated or deleted, and files and directories under the root keep their current owners.
Cache-backed serving pods on that profile roll once, because the prep command is part of the pod template. Installs with fsGroup > 0 render the same pods as before and do not roll.
After the re-own, uid 100 and group 100 have no access to the root. Anything that relied on that owner (a pre-provisioned volume or a custom image running as uid 100) must reach the cache another way. Setting --init-container-user=100 keeps uid 100 as the owner, with group 0 instead of 100. On storage that applies an fsGroup, a container that relied only on the fsGroup group also has no access after the prep sets the root's group to 0, exactly as it had none under 100:100.

Turning the prep off (`modelCache.prep`)

modelCache.prep (controller flag --model-cache-prep) selects whether the prep is injected:

Value	Effect
`always` (default)	The prep runs in every cache-backed pod and every prefetch Job.
`never`	The prep is never injected. The downloader still runs non-root; if the volume's ownership is wrong it fails with `Permission denied` and the `ModelCacheNotWritable` diagnosis names this setting, and `--init-container-user` for the case where the prep is on with fsGroup disabled.

Use never only when you know the cache's ownership is right without the prep: a CSI driver with fsGroupPolicy: File and a pod that sets fsGroup (kubelet applies only the pod's fsGroup), or a claim you pre-provisioned with the downloader's ownership. A supplied spec.podSecurityContext replaces the operator's default, so fsGroup must be restated there. The one thing the operator still adds to it is the prefetch Job's group, in the supplementalGroups of a pod that reads a prefetched cache (see the prefetch section). Then the cache-backed pods contain no root container at all, which is what a restricted namespace needs, and the prefetch Job additionally carries runAsNonRoot with a numeric uid, allowPrivilegeEscalation: false, ALL dropped, and a pod-level seccompProfile: RuntimeDefault.

Changing the value rolls every cache-backed serving pod once, because the pod template changes. A prefetch Job whose pod has already started finishes under the setting it started with. One that has not started a pod yet keeps the template it was created with, so under restricted its pods are refused until the Job is deleted and rebuilt (or its ttlSecondsAfterFinished expires). Under never the prefetch Job's containers run as --init-container-user (default 101, the default curlimages/curl user), so set that flag to your image's uid when you change --init-container-image; otherwise the prefetch writes as a different user from the serving downloader.

auto (resolve the storage class and skip the prep only when the driver applies fsGroup itself) is a planned follow-up; the flag and the chart schema reject it today.

PSA `restricted`

The prep needs root plus CHOWN/FOWNER, so a cache-backed pod with modelCache.prep: always cannot satisfy Pod Security Admission restricted (which forbids running as root and adding any capability except NET_BIND_SERVICE). On storage that applies fsGroup itself, set modelCache.prep: never: the root container is gone and the pod can pass restricted.

Each InferenceService sharing that cache must also supply, in spec.podSecurityContext:

spec:
  podSecurityContext:
    runAsNonRoot: true
    runAsUser: 101        # numeric; the downloader inherits it
    fsGroup: 102          # the --default-fsgroup value; a supplied context replaces the operator default
    fsGroupChangePolicy: OnRootMismatch
    seccompProfile:
      type: RuntimeDefault

fsGroup has to be in the snippet because the operator's default fsGroup is not merged into a user-supplied context: without it kubelet applies no group to the cache volume and a fsGroupPolicy: File driver, which is why never was chosen, changes nothing. OnRootMismatch is the operator's own default for the same reason (see fsGroupChangePolicy below).

A numeric runAsUser is required, not optional: the default init image (curlimages/curl) declares the named user curl_user, and the kubelet refuses runAsNonRoot on an image whose user it cannot resolve numerically (container has runAsNonRoot and image has non-numeric user (curl_user)).

Inference pods deliberately do not get runAsNonRoot by default, because runtime images that run as root (vLLM, for example) would break. Without never, the alternatives are an emptyDir model store (no shared persistent cache) or a baseline namespace policy.

`fsGroupChangePolicy` and pod start time on large caches

On storage where Kubernetes applies the pod fsGroup (CSI drivers with fsGroupPolicy: File, and under the default ReadWriteOnceWithFSType a ReadWriteOnce volume with an fsType), the default fsGroupChangePolicy is Always: before every pod start, kubelet walks the entire volume and changes the group and permissions of every file and directory. A shared model cache is the worst case for that walk, since it grows with every model you add, and on a network filesystem each change is a metadata round trip.

The operator therefore sets fsGroupChangePolicy: OnRootMismatch wherever it sets the default fsGroup (inference pods and the prefetch Job):

securityContext:
  fsGroup: 102
  fsGroupChangePolicy: OnRootMismatch
  seccompProfile:
    type: RuntimeDefault

With OnRootMismatch, kubelet skips the walk when the volume root already has the fsGroup GID, group rwx and the setgid bit, which is the state the first full pass leaves and, under modelCache.prep: always, the model-cache-prep step (chown 0:<fsGroup>, chmod g+rwX) keeps. If anything changes the root's group, the next start does the full walk again. On fsGroupPolicy: None backends kubelet changes no ownership at all, so the setting has no effect there.

A user-supplied spec.podSecurityContext replaces the operator default entirely, so it does not get this setting. If you supply your own context on a large cache, add fsGroupChangePolicy: OnRootMismatch next to your fsGroup. When the default fsGroup is disabled (--default-fsgroup=0, the OpenShift preset), the operator sets neither field.

One consequence of skipping the walk: kubelet no longer re-grants group write on every file at each start, so files keep the modes their writer gave them. That is fine for the operator's own paths, because every downloader runs as the same init-image user and the serving containers mount the cache read-only. If you switch --init-container-image to an image with a different UID on a cache that is already populated, a resumed download into a directory the old UID created can fail with permission denied; delete that model's cache directory (or fix its ownership once) and let it download again.

Shared-cache group-write multi-tenancy

In shared mode, the cache-prep init container (with its default modelCache.prep: always) sets chmod g+rwX /models, making the cache root writable by any pod in the same fsGroup. On a shared cache PVC, multiple InferenceServices share that fsGroup and can write to each other's cached model files.

Within one operator's trust domain this is fine. For a multi-tenant cache where distrusting tenants share the same namespace, this is a real consideration. Use per-service caches (modelCache.mode: perService) for isolation between distrusting tenants.

FilesExpand file tree

MODEL-CACHE.md

Latest commit

History