KServe Custom Storage Initializer Problem with Models Registered on MinIO

[MattiaSarti]

Describe the bug

I have uploaded a model to MinIO and registered it in the Model Registry, and after defining a custom storage initializer (a ClusterStorageContainer) for KServe's InferenceServices to recognize Model Registry URIs, I am trying to create an InferenceService with an URI that points to my registered model (model-registry://) so that it downloads such a model from MinIO by fetching its metadata from the Model Registry first. Nevertheless, the InferenceService storage initializer fails with weird logs complaining about the bucket object name. What is strange is that, if I instead just directly pass the S3-based MinIO model URI to the InferenceService (s3://) and I keep everything else the same, the model is successfully downloaded by the storage initializer and the InferenceService works. In both cases, the storage initializer container gets populated with the same environment variables containing all the information required to download the model, so I am suspecting the two different images for the storage initializer behave incoherently, with the one based on the Model Registry failing because of some inner bug. Can you please help me understand if that is the case or if it's me misconfiguring something?

To Reproduce

Steps to reproduce the behavior:

1. Set up a Kubernetes cluster
2. Deploy a simple MinIO instance (see the "additonal context" section below for YAML definitions)
3. Deploy Kubeflow Model Registry and KServe from the 1.10.2 version of the unified Kubeflow manifests (including the custom storage initializer in the "additonal context" section below)
4. Upload some model to MinIO and register it with the Model Registry (see the "additonal context" section below for relevant model metadata)

Expected behavior

I expect the Model Registry-based URI to work like the S3-based one.

Additional context

Most Relevant Information to Reproduce

MinIO Definition

apiVersion: v1
kind: Namespace
metadata:
  name: minio-dev
  labels:
    name: minio-dev
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: minio-storage
  namespace: minio-dev
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: Pod
metadata:
  labels:
    app: minio
  name: minio
  namespace: minio-dev
spec:
  containers:
  - name: minio
    image: quay.io/minio/minio:latest
    command:
    - /bin/bash
    - -c
    args: 
    - minio server /data --console-address :9090
    volumeMounts:
    - mountPath: /data
      name: minio-data
  volumes:
  - name: minio-data
    persistentVolumeClaim:
      claimName: minio-storage
---
apiVersion: v1
kind: Service
metadata:
  name: minio
  namespace: minio-dev
spec:
  selector:
    app: minio
  ports:
  - name: http-api
    protocol: TCP
    port: 9000
    targetPort: 9000
  - name: http-ui
    protocol: TCP
    port: 9090
    targetPort: 9090

Custom Storage Initializer Definition

apiVersion: serving.kserve.io/v1alpha1
kind: ClusterStorageContainer
metadata:
  name: model-registry-storage-initializer
  namespace: kubeflow
spec:
  container:
    env:
    - name: MODEL_REGISTRY_BASE_URL
      value: model-registry-service.kubeflow.svc.cluster.local:8080
    image: ghcr.io/kubeflow/model-registry/storage-initializer:v0.2.19
    name: storage-initializer
    resources:
      limits:
        memory: 1Gi
      requests:
        cpu: 100m
        memory: 100Mi
  supportedUriFormats:
  - prefix: model-registry://

InferenceService Definition

apiVersion: v1
kind: Secret
metadata:
  name: minio-for-kserve
  namespace: kubeflow-user-example-com
  annotations:
    serving.kserve.io/s3-endpoint: minio.minio-dev.svc.cluster.local:9000
    serving.kserve.io/s3-usehttps: "0"
    serving.kserve.io/s3-region: "minio"
    serving.kserve.io/s3-useanoncredential: "false"
    serving.kserve.io/s3-usevirtualbucket: "false"
type: Opaque
stringData:
  AWS_ACCESS_KEY_ID: minioadmin
  AWS_SECRET_ACCESS_KEY: minioadmin
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: minio-for-kserve
  namespace: kubeflow-user-example-com
secrets:
- name: minio-for-kserve
---
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  annotations:
    sidecar.istio.io/inject: 'false'
  finalizers:
    - inferenceservice.finalizers
  generation: 1
  labels:
    modelregistry/model-version-id: '5'
    modelregistry/registered-model-id: '1'
  name: wonderland
  namespace: kubeflow-user-example-com
spec:
  predictor:
    serviceAccountName: minio-for-kserve
    model:
      modelFormat:
        name: sklearn
        version: '1'
      name: ''
      resources: {}
      storageUri: model-registry://iris-flower-classification/super-latest-version
      # storageUri: s3://my-fancy-bucket/models/whatever

NOTE: changing the URI in the InferenceService definition above to the commented one makes it work

With the S3-based URI:

InferenceService storage-initializer Container Logs

2025-08-27 10:42:55.912 1 kserve INFO [initializer-entrypoint:<module>():16] Initializing, args: src_uri [s3://my-fancy-bucket/models/whatever] dest_path[ [/mnt/models]
2025-08-27 10:42:55.912 1 kserve INFO [storage.py:download():64] Copying contents of s3://my-fancy-bucket/models/whatever to local
2025-08-27 10:42:57.023 1 kserve INFO [storage.py:_download_s3():274] Downloaded object models/whatever/model.joblib to /mnt/models/model.joblib
2025-08-27 10:42:57.024 1 kserve INFO [storage.py:download():110] Successfully copied s3://my-fancy-bucket/models/whatever to /mnt/models
2025-08-27 10:42:57.024 1 kserve INFO [storage.py:download():111] Model downloaded in 1.1125189810009033 seconds.

InferenceService Pod

kind: Pod
apiVersion: v1
metadata:
  name: wonder-wonderland-predictor-00001-deployment-b9c5665dc-2qzr6
  generateName: wonder-wonderland-predictor-00001-deployment-b9c5665dc-
  namespace: kubeflow-user-example-com
  uid: 403e5dd3-5b7b-4800-aa75-ee365ec04f1a
  resourceVersion: '759504'
  creationTimestamp: '2025-08-27T10:42:41Z'
  labels:
    app: wonder-wonderland-predictor-00001
    component: predictor
    modelregistry/model-version-id: '5'
    modelregistry/registered-model-id: '1'
    pod-template-hash: b9c5665dc
    service.istio.io/canonical-name: wonder-wonderland-predictor
    service.istio.io/canonical-revision: wonder-wonderland-predictor-00001
    serving.knative.dev/configuration: wonder-wonderland-predictor
    serving.knative.dev/configurationGeneration: '1'
    serving.knative.dev/configurationUID: c5a6d2f2-30bb-4bc0-acb1-a09e5bc20e25
    serving.knative.dev/revision: wonder-wonderland-predictor-00001
    serving.knative.dev/revisionUID: 0f8e27c4-3d97-41fa-b144-6b4e2fb22161
    serving.knative.dev/service: wonder-wonderland-predictor
    serving.knative.dev/serviceUID: 827f4e30-51c1-49bd-9158-702e8cf1c13e
    serving.kserve.io/inferenceservice: wonder-wonderland
  annotations:
    autoscaling.knative.dev/class: kpa.autoscaling.knative.dev
    autoscaling.knative.dev/min-scale: '1'
    cni.projectcalico.org/containerID: a2316f298bc91d2da0dfdcf329ec751d5aa22a74a7af65a7e1c4a625c2b3c891
    cni.projectcalico.org/podIP: 10.1.43.83/32
    cni.projectcalico.org/podIPs: 10.1.43.83/32
    internal.serving.kserve.io/storage-initializer-sourceuri: s3://my-fancy-bucket/models/whatever
    prometheus.kserve.io/path: /metrics
    prometheus.kserve.io/port: '8080'
    serving.knative.dev/creator: system:serviceaccount:kubeflow:kserve-controller-manager
    serving.kserve.io/enable-metric-aggregation: 'false'
    serving.kserve.io/enable-prometheus-scraping: 'false'
    sidecar.istio.io/inject: 'false'
  ownerReferences:
    - apiVersion: apps/v1
      kind: ReplicaSet
      name: wonder-wonderland-predictor-00001-deployment-b9c5665dc
      uid: 1e2b84b0-ddc3-41a1-9f70-d7fd6ad5b609
      controller: true
      blockOwnerDeletion: true
  managedFields:
    - manager: kubelite
      operation: Update
      apiVersion: v1
      time: '2025-08-27T10:42:41Z'
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            .: {}
            f:autoscaling.knative.dev/class: {}
            f:autoscaling.knative.dev/min-scale: {}
            f:internal.serving.kserve.io/storage-initializer-sourceuri: {}
            f:prometheus.kserve.io/path: {}
            f:prometheus.kserve.io/port: {}
            f:serving.knative.dev/creator: {}
            f:sidecar.istio.io/inject: {}
          f:generateName: {}
          f:labels:
            .: {}
            f:app: {}
            f:component: {}
            f:modelregistry/model-version-id: {}
            f:modelregistry/registered-model-id: {}
            f:pod-template-hash: {}
            f:service.istio.io/canonical-name: {}
            f:service.istio.io/canonical-revision: {}
            f:serving.knative.dev/configuration: {}
            f:serving.knative.dev/configurationGeneration: {}
            f:serving.knative.dev/configurationUID: {}
            f:serving.knative.dev/revision: {}
            f:serving.knative.dev/revisionUID: {}
            f:serving.knative.dev/service: {}
            f:serving.knative.dev/serviceUID: {}
            f:serving.kserve.io/inferenceservice: {}
          f:ownerReferences:
            .: {}
            k:{"uid":"1e2b84b0-ddc3-41a1-9f70-d7fd6ad5b609"}: {}
        f:spec:
          f:affinity:
            .: {}
            f:podAntiAffinity:
              .: {}
              f:preferredDuringSchedulingIgnoredDuringExecution: {}
          f:containers:
            k:{"name":"kserve-container"}:
              .: {}
              f:args: {}
              f:env:
                .: {}
                k:{"name":"K_CONFIGURATION"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"K_REVISION"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"K_SERVICE"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"PORT"}:
                  .: {}
                  f:name: {}
                  f:value: {}
              f:image: {}
              f:imagePullPolicy: {}
              f:lifecycle:
                .: {}
                f:preStop:
                  .: {}
                  f:httpGet:
                    .: {}
                    f:path: {}
                    f:port: {}
                    f:scheme: {}
              f:name: {}
              f:ports:
                .: {}
                k:{"containerPort":8080,"protocol":"TCP"}:
                  .: {}
                  f:containerPort: {}
                  f:name: {}
                  f:protocol: {}
              f:resources:
                .: {}
                f:limits:
                  .: {}
                  f:cpu: {}
                  f:memory: {}
                f:requests:
                  .: {}
                  f:cpu: {}
                  f:memory: {}
              f:securityContext:
                .: {}
                f:allowPrivilegeEscalation: {}
                f:capabilities:
                  .: {}
                  f:drop: {}
                f:runAsNonRoot: {}
              f:terminationMessagePath: {}
              f:terminationMessagePolicy: {}
            k:{"name":"queue-proxy"}:
              .: {}
              f:env:
                .: {}
                k:{"name":"CONTAINER_CONCURRENCY"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"ENABLE_HTTP2_AUTO_DETECTION"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"ENABLE_HTTP_FULL_DUPLEX"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"ENABLE_MULTI_CONTAINER_PROBES"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"ENABLE_PROFILING"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"HOST_IP"}:
                  .: {}
                  f:name: {}
                  f:valueFrom:
                    .: {}
                    f:fieldRef: {}
                k:{"name":"METRICS_COLLECTOR_ADDRESS"}:
                  .: {}
                  f:name: {}
                k:{"name":"METRICS_DOMAIN"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"QUEUE_SERVING_PORT"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"QUEUE_SERVING_TLS_PORT"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"REVISION_IDLE_TIMEOUT_SECONDS"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"REVISION_RESPONSE_START_TIMEOUT_SECONDS"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"REVISION_TIMEOUT_SECONDS"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"ROOT_CA"}:
                  .: {}
                  f:name: {}
                k:{"name":"SERVING_CONFIGURATION"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"SERVING_ENABLE_PROBE_REQUEST_LOG"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"SERVING_ENABLE_REQUEST_LOG"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"SERVING_LOGGING_CONFIG"}:
                  .: {}
                  f:name: {}
                k:{"name":"SERVING_LOGGING_LEVEL"}:
                  .: {}
                  f:name: {}
                k:{"name":"SERVING_NAMESPACE"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"SERVING_POD"}:
                  .: {}
                  f:name: {}
                  f:valueFrom:
                    .: {}
                    f:fieldRef: {}
                k:{"name":"SERVING_POD_IP"}:
                  .: {}
                  f:name: {}
                  f:valueFrom:
                    .: {}
                    f:fieldRef: {}
                k:{"name":"SERVING_READINESS_PROBE"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"SERVING_REQUEST_LOG_TEMPLATE"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"SERVING_REQUEST_METRICS_BACKEND"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"SERVING_REQUEST_METRICS_REPORTING_PERIOD_SECONDS"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"SERVING_REVISION"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"SERVING_SERVICE"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"SYSTEM_NAMESPACE"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"TRACING_CONFIG_BACKEND"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"TRACING_CONFIG_DEBUG"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"TRACING_CONFIG_SAMPLE_RATE"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"TRACING_CONFIG_ZIPKIN_ENDPOINT"}:
                  .: {}
                  f:name: {}
                k:{"name":"USER_PORT"}:
                  .: {}
                  f:name: {}
                  f:value: {}
              f:image: {}
              f:imagePullPolicy: {}
              f:name: {}
              f:ports:
                .: {}
                k:{"containerPort":8012,"protocol":"TCP"}:
                  .: {}
                  f:containerPort: {}
                  f:name: {}
                  f:protocol: {}
                k:{"containerPort":8022,"protocol":"TCP"}:
                  .: {}
                  f:containerPort: {}
                  f:name: {}
                  f:protocol: {}
                k:{"containerPort":8112,"protocol":"TCP"}:
                  .: {}
                  f:containerPort: {}
                  f:name: {}
                  f:protocol: {}
                k:{"containerPort":9090,"protocol":"TCP"}:
                  .: {}
                  f:containerPort: {}
                  f:name: {}
                  f:protocol: {}
                k:{"containerPort":9091,"protocol":"TCP"}:
                  .: {}
                  f:containerPort: {}
                  f:name: {}
                  f:protocol: {}
              f:readinessProbe:
                .: {}
                f:failureThreshold: {}
                f:httpGet:
                  .: {}
                  f:httpHeaders: {}
                  f:path: {}
                  f:port: {}
                  f:scheme: {}
                f:periodSeconds: {}
                f:successThreshold: {}
                f:timeoutSeconds: {}
              f:resources:
                .: {}
                f:requests:
                  .: {}
                  f:cpu: {}
              f:securityContext:
                .: {}
                f:allowPrivilegeEscalation: {}
                f:capabilities:
                  .: {}
                  f:drop: {}
                f:readOnlyRootFilesystem: {}
                f:runAsNonRoot: {}
                f:seccompProfile:
                  .: {}
                  f:type: {}
              f:terminationMessagePath: {}
              f:terminationMessagePolicy: {}
          f:dnsPolicy: {}
          f:enableServiceLinks: {}
          f:restartPolicy: {}
          f:schedulerName: {}
          f:securityContext: {}
          f:serviceAccount: {}
          f:serviceAccountName: {}
          f:terminationGracePeriodSeconds: {}
    - manager: calico
      operation: Update
      apiVersion: v1
      time: '2025-08-27T10:42:43Z'
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            f:cni.projectcalico.org/containerID: {}
            f:cni.projectcalico.org/podIP: {}
            f:cni.projectcalico.org/podIPs: {}
      subresource: status
    - manager: kubelite
      operation: Update
      apiVersion: v1
      time: '2025-08-27T10:43:10Z'
      fieldsType: FieldsV1
      fieldsV1:
        f:status:
          f:conditions:
            k:{"type":"ContainersReady"}:
              .: {}
              f:lastProbeTime: {}
              f:lastTransitionTime: {}
              f:status: {}
              f:type: {}
            k:{"type":"Initialized"}:
              .: {}
              f:lastProbeTime: {}
              f:lastTransitionTime: {}
              f:status: {}
              f:type: {}
            k:{"type":"PodReadyToStartContainers"}:
              .: {}
              f:lastProbeTime: {}
              f:lastTransitionTime: {}
              f:status: {}
              f:type: {}
            k:{"type":"Ready"}:
              .: {}
              f:lastProbeTime: {}
              f:lastTransitionTime: {}
              f:status: {}
              f:type: {}
          f:containerStatuses: {}
          f:hostIP: {}
          f:hostIPs: {}
          f:initContainerStatuses: {}
          f:phase: {}
          f:podIP: {}
          f:podIPs:
            .: {}
            k:{"ip":"10.1.43.83"}:
              .: {}
              f:ip: {}
          f:startTime: {}
      subresource: status
spec:
  volumes:
    - name: kube-api-access-4s78l
      projected:
        sources:
          - serviceAccountToken:
              expirationSeconds: 3607
              path: token
          - configMap:
              name: kube-root-ca.crt
              items:
                - key: ca.crt
                  path: ca.crt
          - downwardAPI:
              items:
                - path: namespace
                  fieldRef:
                    apiVersion: v1
                    fieldPath: metadata.namespace
        defaultMode: 420
    - name: kserve-provision-location
      emptyDir: {}
  initContainers:
    - name: storage-initializer
      image: kserve/storage-initializer:v0.15.0
      args:
        - s3://my-fancy-bucket/models/whatever
        - /mnt/models
      env:
        - name: AWS_ACCESS_KEY_ID
          valueFrom:
            secretKeyRef:
              name: minio-for-kserve
              key: AWS_ACCESS_KEY_ID
        - name: AWS_SECRET_ACCESS_KEY
          valueFrom:
            secretKeyRef:
              name: minio-for-kserve
              key: AWS_SECRET_ACCESS_KEY
        - name: S3_USE_HTTPS
          value: '0'
        - name: S3_ENDPOINT
          value: minio.minio-dev.svc.cluster.local:9000
        - name: AWS_ENDPOINT_URL
          value: http://minio.minio-dev.svc.cluster.local:9000
        - name: awsAnonymousCredential
          value: 'false'
        - name: AWS_DEFAULT_REGION
          value: minio
        - name: S3_USER_VIRTUAL_BUCKET
          value: 'false'
      resources:
        limits:
          cpu: '1'
          memory: 1Gi
        requests:
          cpu: 100m
          memory: 100Mi
      volumeMounts:
        - name: kserve-provision-location
          mountPath: /mnt/models
        - name: kube-api-access-4s78l
          readOnly: true
          mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: FallbackToLogsOnError
      imagePullPolicy: IfNotPresent
  containers:
    - name: kserve-container
      image: >-
        index.docker.io/kserve/sklearnserver@sha256:d19adc0a6223d72e371a9cf852c56c910789ef0eac6a8baf96db35d0cc1d8304
      args:
        - '--model_name=wonder-wonderland'
        - '--model_dir=/mnt/models'
        - '--http_port=8080'
      ports:
        - name: user-port
          containerPort: 8080
          protocol: TCP
      env:
        - name: PORT
          value: '8080'
        - name: K_REVISION
          value: wonder-wonderland-predictor-00001
        - name: K_CONFIGURATION
          value: wonder-wonderland-predictor
        - name: K_SERVICE
          value: wonder-wonderland-predictor
      resources:
        limits:
          cpu: '1'
          memory: 2Gi
        requests:
          cpu: '1'
          memory: 2Gi
      volumeMounts:
        - name: kube-api-access-4s78l
          readOnly: true
          mountPath: /var/run/secrets/kubernetes.io/serviceaccount
        - name: kserve-provision-location
          readOnly: true
          mountPath: /mnt/models
      lifecycle:
        preStop:
          httpGet:
            path: /wait-for-drain
            port: 8022
            scheme: HTTP
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: FallbackToLogsOnError
      imagePullPolicy: IfNotPresent
      securityContext:
        capabilities:
          drop:
            - ALL
        runAsNonRoot: true
        allowPrivilegeEscalation: false
    - name: queue-proxy
      image: >-
        gcr.io/knative-releases/knative.dev/serving/cmd/queue@sha256:698ef80ebc698f4d2bb93c1e85684063a0cf253a83faebcbf106cee444181d8e
      ports:
        - name: http-queueadm
          containerPort: 8022
          protocol: TCP
        - name: http-autometric
          containerPort: 9090
          protocol: TCP
        - name: http-usermetric
          containerPort: 9091
          protocol: TCP
        - name: queue-port
          containerPort: 8012
          protocol: TCP
        - name: https-port
          containerPort: 8112
          protocol: TCP
      env:
        - name: SERVING_NAMESPACE
          value: kubeflow-user-example-com
        - name: SERVING_SERVICE
          value: wonder-wonderland-predictor
        - name: SERVING_CONFIGURATION
          value: wonder-wonderland-predictor
        - name: SERVING_REVISION
          value: wonder-wonderland-predictor-00001
        - name: QUEUE_SERVING_PORT
          value: '8012'
        - name: QUEUE_SERVING_TLS_PORT
          value: '8112'
        - name: CONTAINER_CONCURRENCY
          value: '0'
        - name: REVISION_TIMEOUT_SECONDS
          value: '300'
        - name: REVISION_RESPONSE_START_TIMEOUT_SECONDS
          value: '0'
        - name: REVISION_IDLE_TIMEOUT_SECONDS
          value: '0'
        - name: SERVING_POD
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: SERVING_POD_IP
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: status.podIP
        - name: SERVING_LOGGING_CONFIG
        - name: SERVING_LOGGING_LEVEL
        - name: SERVING_REQUEST_LOG_TEMPLATE
          value: >-
            {"httpRequest": {"requestMethod": "{{.Request.Method}}",
            "requestUrl": "{{js .Request.RequestURI}}", "requestSize":
            "{{.Request.ContentLength}}", "status": {{.Response.Code}},
            "responseSize": "{{.Response.Size}}", "userAgent": "{{js
            .Request.UserAgent}}", "remoteIp": "{{js .Request.RemoteAddr}}",
            "serverIp": "{{.Revision.PodIP}}", "referer": "{{js
            .Request.Referer}}", "latency": "{{.Response.Latency}}s",
            "protocol": "{{.Request.Proto}}"}, "traceId": "{{index
            .Request.Header "X-B3-Traceid"}}"}
        - name: SERVING_ENABLE_REQUEST_LOG
          value: 'false'
        - name: SERVING_REQUEST_METRICS_BACKEND
          value: prometheus
        - name: SERVING_REQUEST_METRICS_REPORTING_PERIOD_SECONDS
          value: '5'
        - name: TRACING_CONFIG_BACKEND
          value: none
        - name: TRACING_CONFIG_ZIPKIN_ENDPOINT
        - name: TRACING_CONFIG_DEBUG
          value: 'false'
        - name: TRACING_CONFIG_SAMPLE_RATE
          value: '0.1'
        - name: USER_PORT
          value: '8080'
        - name: SYSTEM_NAMESPACE
          value: knative-serving
        - name: METRICS_DOMAIN
          value: knative.dev/internal/serving
        - name: SERVING_READINESS_PROBE
          value: '{"tcpSocket":{"port":8080,"host":"127.0.0.1"},"successThreshold":1}'
        - name: ENABLE_PROFILING
          value: 'false'
        - name: SERVING_ENABLE_PROBE_REQUEST_LOG
          value: 'false'
        - name: METRICS_COLLECTOR_ADDRESS
        - name: HOST_IP
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: status.hostIP
        - name: ENABLE_HTTP2_AUTO_DETECTION
          value: 'false'
        - name: ENABLE_HTTP_FULL_DUPLEX
          value: 'false'
        - name: ROOT_CA
        - name: ENABLE_MULTI_CONTAINER_PROBES
          value: 'false'
      resources:
        requests:
          cpu: 25m
      volumeMounts:
        - name: kube-api-access-4s78l
          readOnly: true
          mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      readinessProbe:
        httpGet:
          path: /
          port: 8012
          scheme: HTTP
          httpHeaders:
            - name: K-Network-Probe
              value: queue
        timeoutSeconds: 1
        periodSeconds: 10
        successThreshold: 1
        failureThreshold: 3
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      imagePullPolicy: IfNotPresent
      securityContext:
        capabilities:
          drop:
            - ALL
        runAsNonRoot: true
        readOnlyRootFilesystem: true
        allowPrivilegeEscalation: false
        seccompProfile:
          type: RuntimeDefault
  restartPolicy: Always
  terminationGracePeriodSeconds: 300
  dnsPolicy: ClusterFirst
  serviceAccountName: minio-for-kserve
  serviceAccount: minio-for-kserve
  nodeName: mattia-gp63-leopard-8re
  securityContext: {}
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 100
          podAffinityTerm:
            labelSelector:
              matchLabels:
                serving.knative.dev/revision: wonder-wonderland-predictor-00001
            topologyKey: kubernetes.io/hostname
  schedulerName: default-scheduler
  tolerations:
    - key: node.kubernetes.io/not-ready
      operator: Exists
      effect: NoExecute
      tolerationSeconds: 300
    - key: node.kubernetes.io/unreachable
      operator: Exists
      effect: NoExecute
      tolerationSeconds: 300
  priority: 0
  enableServiceLinks: false
  preemptionPolicy: PreemptLowerPriority
status:
  phase: Running
  conditions:
    - type: PodReadyToStartContainers
      status: 'True'
      lastProbeTime: null
      lastTransitionTime: '2025-08-27T10:42:45Z'
    - type: Initialized
      status: 'True'
      lastProbeTime: null
      lastTransitionTime: '2025-08-27T10:42:58Z'
    - type: Ready
      status: 'True'
      lastProbeTime: null
      lastTransitionTime: '2025-08-27T10:43:10Z'
    - type: ContainersReady
      status: 'True'
      lastProbeTime: null
      lastTransitionTime: '2025-08-27T10:43:10Z'
    - type: PodScheduled
      status: 'True'
      lastProbeTime: null
      lastTransitionTime: '2025-08-27T10:42:42Z'
  hostIP: 192.168.1.179
  hostIPs:
    - ip: 192.168.1.179
  podIP: 10.1.43.83
  podIPs:
    - ip: 10.1.43.83
  startTime: '2025-08-27T10:42:42Z'
  initContainerStatuses:
    - name: storage-initializer
      state:
        terminated:
          exitCode: 0
          reason: Completed
          startedAt: '2025-08-27T10:42:45Z'
          finishedAt: '2025-08-27T10:42:57Z'
          containerID: >-
            containerd://f72096b098fb44a3869dd6c74a5730dc8b1d3a2b2e457683222c1ebdd12a9207
      lastState: {}
      ready: true
      restartCount: 0
      image: docker.io/kserve/storage-initializer:v0.15.0
      imageID: >-
        docker.io/kserve/storage-initializer@sha256:72be1c414b11f45788106d6e002c18bdb4ca851048c4ae0621c9d57a17ccc501
      containerID: >-
        containerd://f72096b098fb44a3869dd6c74a5730dc8b1d3a2b2e457683222c1ebdd12a9207
      started: false
      volumeMounts:
        - name: kserve-provision-location
          mountPath: /mnt/models
        - name: kube-api-access-4s78l
          mountPath: /var/run/secrets/kubernetes.io/serviceaccount
          readOnly: true
          recursiveReadOnly: Disabled
  containerStatuses:
    - name: kserve-container
      state:
        running:
          startedAt: '2025-08-27T10:42:58Z'
      lastState: {}
      ready: true
      restartCount: 0
      image: sha256:f14f202c23a5e0a24caadcf3fcda2f7fa3c7825513e04f90d121b5d8e15edf81
      imageID: >-
        docker.io/kserve/sklearnserver@sha256:d19adc0a6223d72e371a9cf852c56c910789ef0eac6a8baf96db35d0cc1d8304
      containerID: >-
        containerd://9bcaad57a8039390e1c1d7bf58bbef09f26fbaba0309392fde4ea4ff7e30ec2d
      started: true
      volumeMounts:
        - name: kube-api-access-4s78l
          mountPath: /var/run/secrets/kubernetes.io/serviceaccount
          readOnly: true
          recursiveReadOnly: Disabled
        - name: kserve-provision-location
          mountPath: /mnt/models
          readOnly: true
          recursiveReadOnly: Disabled
    - name: queue-proxy
      state:
        running:
          startedAt: '2025-08-27T10:42:58Z'
      lastState: {}
      ready: true
      restartCount: 0
      image: sha256:55b72cb821c4ec4a08c3bc7a09ddd5c39784b56d6fc3abc61e34d4fec3fed73d
      imageID: >-
        gcr.io/knative-releases/knative.dev/serving/cmd/queue@sha256:698ef80ebc698f4d2bb93c1e85684063a0cf253a83faebcbf106cee444181d8e
      containerID: >-
        containerd://530b0ac6151c2a0287f57292dbdd90b362d791ce7efbeb898d73028f839ebe73
      started: true
      volumeMounts:
        - name: kube-api-access-4s78l
          mountPath: /var/run/secrets/kubernetes.io/serviceaccount
          readOnly: true
          recursiveReadOnly: Disabled
  qosClass: Burstable

With the Model Registry-based URI:

InferenceService storage-initializer Container Logs

2025/08/27 10:44:28 Initializing, args: src_uri [model-registry://iris-flower-classification/super-latest-version] dest_path[ [/mnt/models]
2025/08/27 10:44:28 Falling back to base url model-registry-service.kubeflow-user-example-com.svc.cluster.local:8080 for model registry service
2025/08/27 10:44:28 Download model indexed in model registry: modelName=, storageUri=model-registry://iris-flower-classification/super-latest-version, modelDir=/mnt/models
2025/08/27 10:44:28 Parsed storageUri=model-registry://iris-flower-classification/super-latest-version as: modelRegistryUrl=model-registry-service.kubeflow-user-example-com.svc.cluster.local:8080, registeredModelName=iris-flower-classification, versionName=0xc000616760
2025/08/27 10:44:28 Fetching model: registeredModelName=iris-flower-classification, versionName=0xc000616760
2025/08/27 10:44:28 Fetching model version: model=&{0xc000092568 <nil> <nil> iris-flower-classification 0xc000600650 0xc000600640 0xc000600660 <nil> <nil> [] [] <nil> <nil> <nil> <nil> <nil> 0xc000600670 0xc000600680}
2025/08/27 10:44:29 Fetching model artifacts: version=&{0xc00062a138 0xc0001a14a0 <nil> super-latest-version 0xc0001a14d0 0xc0001a1390 1 0xc0001a14b0 0xc0001a13a0 0xc0001a14c0}
2025/08/27 10:44:29 Error downloading the model: unable to get batch objects XMinioInvalidObjectName: Object name contains unsupported characters.
 status code: 400, request id: 185F99C92F2B5BA4, host id: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8

InferenceService Pod

kind: Pod
apiVersion: v1
metadata:
  name: wonderland-predictor-00001-deployment-66f66c6c68-559bk
  generateName: wonderland-predictor-00001-deployment-66f66c6c68-
  namespace: kubeflow-user-example-com
  uid: 1d23a6fd-908c-4d69-8844-7961baa9b14e
  resourceVersion: '762963'
  creationTimestamp: '2025-08-27T10:49:50Z'
  labels:
    app: wonderland-predictor-00001
    component: predictor
    modelregistry/model-version-id: '5'
    modelregistry/registered-model-id: '1'
    pod-template-hash: 66f66c6c68
    service.istio.io/canonical-name: wonderland-predictor
    service.istio.io/canonical-revision: wonderland-predictor-00001
    serving.knative.dev/configuration: wonderland-predictor
    serving.knative.dev/configurationGeneration: '1'
    serving.knative.dev/configurationUID: bbf81c5e-cda3-4f2d-a00d-916acb388235
    serving.knative.dev/revision: wonderland-predictor-00001
    serving.knative.dev/revisionUID: c7023e6a-4b86-4af2-b28b-df7409bc4ca1
    serving.knative.dev/service: wonderland-predictor
    serving.knative.dev/serviceUID: bd04f2b0-a0b4-43df-89d5-6a57514641a0
    serving.kserve.io/inferenceservice: wonderland
  annotations:
    autoscaling.knative.dev/class: kpa.autoscaling.knative.dev
    autoscaling.knative.dev/min-scale: '1'
    cni.projectcalico.org/containerID: 8fab5aaebb2e3bf9a9d5782c3d7434b2c27f0e1b6c9b9a505f16b440db9200cf
    cni.projectcalico.org/podIP: 10.1.43.106/32
    cni.projectcalico.org/podIPs: 10.1.43.106/32
    internal.serving.kserve.io/storage-initializer-sourceuri: model-registry://iris-flower-classification/super-latest-version
    prometheus.kserve.io/path: /metrics
    prometheus.kserve.io/port: '8080'
    serving.knative.dev/creator: system:serviceaccount:kubeflow:kserve-controller-manager
    serving.kserve.io/enable-metric-aggregation: 'false'
    serving.kserve.io/enable-prometheus-scraping: 'false'
    sidecar.istio.io/inject: 'false'
  ownerReferences:
    - apiVersion: apps/v1
      kind: ReplicaSet
      name: wonderland-predictor-00001-deployment-66f66c6c68
      uid: d28efe56-e2b7-4287-844a-48f81399c660
      controller: true
      blockOwnerDeletion: true
  managedFields:
    - manager: kubelite
      operation: Update
      apiVersion: v1
      time: '2025-08-27T10:49:50Z'
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            .: {}
            f:autoscaling.knative.dev/class: {}
            f:autoscaling.knative.dev/min-scale: {}
            f:internal.serving.kserve.io/storage-initializer-sourceuri: {}
            f:prometheus.kserve.io/path: {}
            f:prometheus.kserve.io/port: {}
            f:serving.knative.dev/creator: {}
            f:sidecar.istio.io/inject: {}
          f:generateName: {}
          f:labels:
            .: {}
            f:app: {}
            f:component: {}
            f:modelregistry/model-version-id: {}
            f:modelregistry/registered-model-id: {}
            f:pod-template-hash: {}
            f:service.istio.io/canonical-name: {}
            f:service.istio.io/canonical-revision: {}
            f:serving.knative.dev/configuration: {}
            f:serving.knative.dev/configurationGeneration: {}
            f:serving.knative.dev/configurationUID: {}
            f:serving.knative.dev/revision: {}
            f:serving.knative.dev/revisionUID: {}
            f:serving.knative.dev/service: {}
            f:serving.knative.dev/serviceUID: {}
            f:serving.kserve.io/inferenceservice: {}
          f:ownerReferences:
            .: {}
            k:{"uid":"d28efe56-e2b7-4287-844a-48f81399c660"}: {}
        f:spec:
          f:affinity:
            .: {}
            f:podAntiAffinity:
              .: {}
              f:preferredDuringSchedulingIgnoredDuringExecution: {}
          f:containers:
            k:{"name":"kserve-container"}:
              .: {}
              f:args: {}
              f:env:
                .: {}
                k:{"name":"K_CONFIGURATION"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"K_REVISION"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"K_SERVICE"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"PORT"}:
                  .: {}
                  f:name: {}
                  f:value: {}
              f:image: {}
              f:imagePullPolicy: {}
              f:lifecycle:
                .: {}
                f:preStop:
                  .: {}
                  f:httpGet:
                    .: {}
                    f:path: {}
                    f:port: {}
                    f:scheme: {}
              f:name: {}
              f:ports:
                .: {}
                k:{"containerPort":8080,"protocol":"TCP"}:
                  .: {}
                  f:containerPort: {}
                  f:name: {}
                  f:protocol: {}
              f:resources:
                .: {}
                f:limits:
                  .: {}
                  f:cpu: {}
                  f:memory: {}
                f:requests:
                  .: {}
                  f:cpu: {}
                  f:memory: {}
              f:securityContext:
                .: {}
                f:allowPrivilegeEscalation: {}
                f:capabilities:
                  .: {}
                  f:drop: {}
                f:runAsNonRoot: {}
              f:terminationMessagePath: {}
              f:terminationMessagePolicy: {}
            k:{"name":"queue-proxy"}:
              .: {}
              f:env:
                .: {}
                k:{"name":"CONTAINER_CONCURRENCY"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"ENABLE_HTTP2_AUTO_DETECTION"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"ENABLE_HTTP_FULL_DUPLEX"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"ENABLE_MULTI_CONTAINER_PROBES"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"ENABLE_PROFILING"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"HOST_IP"}:
                  .: {}
                  f:name: {}
                  f:valueFrom:
                    .: {}
                    f:fieldRef: {}
                k:{"name":"METRICS_COLLECTOR_ADDRESS"}:
                  .: {}
                  f:name: {}
                k:{"name":"METRICS_DOMAIN"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"QUEUE_SERVING_PORT"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"QUEUE_SERVING_TLS_PORT"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"REVISION_IDLE_TIMEOUT_SECONDS"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"REVISION_RESPONSE_START_TIMEOUT_SECONDS"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"REVISION_TIMEOUT_SECONDS"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"ROOT_CA"}:
                  .: {}
                  f:name: {}
                k:{"name":"SERVING_CONFIGURATION"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"SERVING_ENABLE_PROBE_REQUEST_LOG"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"SERVING_ENABLE_REQUEST_LOG"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"SERVING_LOGGING_CONFIG"}:
                  .: {}
                  f:name: {}
                k:{"name":"SERVING_LOGGING_LEVEL"}:
                  .: {}
                  f:name: {}
                k:{"name":"SERVING_NAMESPACE"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"SERVING_POD"}:
                  .: {}
                  f:name: {}
                  f:valueFrom:
                    .: {}
                    f:fieldRef: {}
                k:{"name":"SERVING_POD_IP"}:
                  .: {}
                  f:name: {}
                  f:valueFrom:
                    .: {}
                    f:fieldRef: {}
                k:{"name":"SERVING_READINESS_PROBE"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"SERVING_REQUEST_LOG_TEMPLATE"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"SERVING_REQUEST_METRICS_BACKEND"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"SERVING_REQUEST_METRICS_REPORTING_PERIOD_SECONDS"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"SERVING_REVISION"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"SERVING_SERVICE"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"SYSTEM_NAMESPACE"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"TRACING_CONFIG_BACKEND"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"TRACING_CONFIG_DEBUG"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"TRACING_CONFIG_SAMPLE_RATE"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"TRACING_CONFIG_ZIPKIN_ENDPOINT"}:
                  .: {}
                  f:name: {}
                k:{"name":"USER_PORT"}:
                  .: {}
                  f:name: {}
                  f:value: {}
              f:image: {}
              f:imagePullPolicy: {}
              f:name: {}
              f:ports:
                .: {}
                k:{"containerPort":8012,"protocol":"TCP"}:
                  .: {}
                  f:containerPort: {}
                  f:name: {}
                  f:protocol: {}
                k:{"containerPort":8022,"protocol":"TCP"}:
                  .: {}
                  f:containerPort: {}
                  f:name: {}
                  f:protocol: {}
                k:{"containerPort":8112,"protocol":"TCP"}:
                  .: {}
                  f:containerPort: {}
                  f:name: {}
                  f:protocol: {}
                k:{"containerPort":9090,"protocol":"TCP"}:
                  .: {}
                  f:containerPort: {}
                  f:name: {}
                  f:protocol: {}
                k:{"containerPort":9091,"protocol":"TCP"}:
                  .: {}
                  f:containerPort: {}
                  f:name: {}
                  f:protocol: {}
              f:readinessProbe:
                .: {}
                f:failureThreshold: {}
                f:httpGet:
                  .: {}
                  f:httpHeaders: {}
                  f:path: {}
                  f:port: {}
                  f:scheme: {}
                f:periodSeconds: {}
                f:successThreshold: {}
                f:timeoutSeconds: {}
              f:resources:
                .: {}
                f:requests:
                  .: {}
                  f:cpu: {}
              f:securityContext:
                .: {}
                f:allowPrivilegeEscalation: {}
                f:capabilities:
                  .: {}
                  f:drop: {}
                f:readOnlyRootFilesystem: {}
                f:runAsNonRoot: {}
                f:seccompProfile:
                  .: {}
                  f:type: {}
              f:terminationMessagePath: {}
              f:terminationMessagePolicy: {}
          f:dnsPolicy: {}
          f:enableServiceLinks: {}
          f:restartPolicy: {}
          f:schedulerName: {}
          f:securityContext: {}
          f:serviceAccount: {}
          f:serviceAccountName: {}
          f:terminationGracePeriodSeconds: {}
    - manager: calico
      operation: Update
      apiVersion: v1
      time: '2025-08-27T10:49:51Z'
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            f:cni.projectcalico.org/containerID: {}
            f:cni.projectcalico.org/podIP: {}
            f:cni.projectcalico.org/podIPs: {}
      subresource: status
    - manager: kubelite
      operation: Update
      apiVersion: v1
      time: '2025-08-27T10:49:54Z'
      fieldsType: FieldsV1
      fieldsV1:
        f:status:
          f:conditions:
            k:{"type":"ContainersReady"}:
              .: {}
              f:lastProbeTime: {}
              f:lastTransitionTime: {}
              f:message: {}
              f:reason: {}
              f:status: {}
              f:type: {}
            k:{"type":"Initialized"}:
              .: {}
              f:lastProbeTime: {}
              f:lastTransitionTime: {}
              f:message: {}
              f:reason: {}
              f:status: {}
              f:type: {}
            k:{"type":"PodReadyToStartContainers"}:
              .: {}
              f:lastProbeTime: {}
              f:lastTransitionTime: {}
              f:status: {}
              f:type: {}
            k:{"type":"Ready"}:
              .: {}
              f:lastProbeTime: {}
              f:lastTransitionTime: {}
              f:message: {}
              f:reason: {}
              f:status: {}
              f:type: {}
          f:containerStatuses: {}
          f:hostIP: {}
          f:hostIPs: {}
          f:initContainerStatuses: {}
          f:podIP: {}
          f:podIPs:
            .: {}
            k:{"ip":"10.1.43.106"}:
              .: {}
              f:ip: {}
          f:startTime: {}
      subresource: status
spec:
  volumes:
    - name: kube-api-access-7hshv
      projected:
        sources:
          - serviceAccountToken:
              expirationSeconds: 3607
              path: token
          - configMap:
              name: kube-root-ca.crt
              items:
                - key: ca.crt
                  path: ca.crt
          - downwardAPI:
              items:
                - path: namespace
                  fieldRef:
                    apiVersion: v1
                    fieldPath: metadata.namespace
        defaultMode: 420
    - name: kserve-provision-location
      emptyDir: {}
  initContainers:
    - name: storage-initializer
      image: ghcr.io/kubeflow/model-registry/storage-initializer:v0.2.19
      args:
        - model-registry://iris-flower-classification/super-latest-version
        - /mnt/models
      env:
        - name: MODEL_REGISTRY_BASE_URL
          value: >-
            model-registry-service.kubeflow-user-example-com.svc.cluster.local:8080
        - name: AWS_ACCESS_KEY_ID
          valueFrom:
            secretKeyRef:
              name: minio-for-kserve
              key: AWS_ACCESS_KEY_ID
        - name: AWS_SECRET_ACCESS_KEY
          valueFrom:
            secretKeyRef:
              name: minio-for-kserve
              key: AWS_SECRET_ACCESS_KEY
        - name: S3_USE_HTTPS
          value: '0'
        - name: S3_ENDPOINT
          value: minio.minio-dev.svc.cluster.local:9000
        - name: AWS_ENDPOINT_URL
          value: http://minio.minio-dev.svc.cluster.local:9000
        - name: awsAnonymousCredential
          value: 'false'
        - name: AWS_DEFAULT_REGION
          value: minio
        - name: S3_USER_VIRTUAL_BUCKET
          value: 'false'
      resources:
        limits:
          cpu: '1'
          memory: 1Gi
        requests:
          cpu: 100m
          memory: 100Mi
      volumeMounts:
        - name: kserve-provision-location
          mountPath: /mnt/models
        - name: kube-api-access-7hshv
          readOnly: true
          mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: FallbackToLogsOnError
      imagePullPolicy: IfNotPresent
  containers:
    - name: kserve-container
      image: >-
        index.docker.io/kserve/sklearnserver@sha256:d19adc0a6223d72e371a9cf852c56c910789ef0eac6a8baf96db35d0cc1d8304
      args:
        - '--model_name=wonderland'
        - '--model_dir=/mnt/models'
        - '--http_port=8080'
      ports:
        - name: user-port
          containerPort: 8080
          protocol: TCP
      env:
        - name: PORT
          value: '8080'
        - name: K_REVISION
          value: wonderland-predictor-00001
        - name: K_CONFIGURATION
          value: wonderland-predictor
        - name: K_SERVICE
          value: wonderland-predictor
      resources:
        limits:
          cpu: '1'
          memory: 2Gi
        requests:
          cpu: '1'
          memory: 2Gi
      volumeMounts:
        - name: kube-api-access-7hshv
          readOnly: true
          mountPath: /var/run/secrets/kubernetes.io/serviceaccount
        - name: kserve-provision-location
          readOnly: true
          mountPath: /mnt/models
      lifecycle:
        preStop:
          httpGet:
            path: /wait-for-drain
            port: 8022
            scheme: HTTP
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: FallbackToLogsOnError
      imagePullPolicy: IfNotPresent
      securityContext:
        capabilities:
          drop:
            - ALL
        runAsNonRoot: true
        allowPrivilegeEscalation: false
    - name: queue-proxy
      image: >-
        gcr.io/knative-releases/knative.dev/serving/cmd/queue@sha256:698ef80ebc698f4d2bb93c1e85684063a0cf253a83faebcbf106cee444181d8e
      ports:
        - name: http-queueadm
          containerPort: 8022
          protocol: TCP
        - name: http-autometric
          containerPort: 9090
          protocol: TCP
        - name: http-usermetric
          containerPort: 9091
          protocol: TCP
        - name: queue-port
          containerPort: 8012
          protocol: TCP
        - name: https-port
          containerPort: 8112
          protocol: TCP
      env:
        - name: SERVING_NAMESPACE
          value: kubeflow-user-example-com
        - name: SERVING_SERVICE
          value: wonderland-predictor
        - name: SERVING_CONFIGURATION
          value: wonderland-predictor
        - name: SERVING_REVISION
          value: wonderland-predictor-00001
        - name: QUEUE_SERVING_PORT
          value: '8012'
        - name: QUEUE_SERVING_TLS_PORT
          value: '8112'
        - name: CONTAINER_CONCURRENCY
          value: '0'
        - name: REVISION_TIMEOUT_SECONDS
          value: '300'
        - name: REVISION_RESPONSE_START_TIMEOUT_SECONDS
          value: '0'
        - name: REVISION_IDLE_TIMEOUT_SECONDS
          value: '0'
        - name: SERVING_POD
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: SERVING_POD_IP
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: status.podIP
        - name: SERVING_LOGGING_CONFIG
        - name: SERVING_LOGGING_LEVEL
        - name: SERVING_REQUEST_LOG_TEMPLATE
          value: >-
            {"httpRequest": {"requestMethod": "{{.Request.Method}}",
            "requestUrl": "{{js .Request.RequestURI}}", "requestSize":
            "{{.Request.ContentLength}}", "status": {{.Response.Code}},
            "responseSize": "{{.Response.Size}}", "userAgent": "{{js
            .Request.UserAgent}}", "remoteIp": "{{js .Request.RemoteAddr}}",
            "serverIp": "{{.Revision.PodIP}}", "referer": "{{js
            .Request.Referer}}", "latency": "{{.Response.Latency}}s",
            "protocol": "{{.Request.Proto}}"}, "traceId": "{{index
            .Request.Header "X-B3-Traceid"}}"}
        - name: SERVING_ENABLE_REQUEST_LOG
          value: 'false'
        - name: SERVING_REQUEST_METRICS_BACKEND
          value: prometheus
        - name: SERVING_REQUEST_METRICS_REPORTING_PERIOD_SECONDS
          value: '5'
        - name: TRACING_CONFIG_BACKEND
          value: none
        - name: TRACING_CONFIG_ZIPKIN_ENDPOINT
        - name: TRACING_CONFIG_DEBUG
          value: 'false'
        - name: TRACING_CONFIG_SAMPLE_RATE
          value: '0.1'
        - name: USER_PORT
          value: '8080'
        - name: SYSTEM_NAMESPACE
          value: knative-serving
        - name: METRICS_DOMAIN
          value: knative.dev/internal/serving
        - name: SERVING_READINESS_PROBE
          value: '{"tcpSocket":{"port":8080,"host":"127.0.0.1"},"successThreshold":1}'
        - name: ENABLE_PROFILING
          value: 'false'
        - name: SERVING_ENABLE_PROBE_REQUEST_LOG
          value: 'false'
        - name: METRICS_COLLECTOR_ADDRESS
        - name: HOST_IP
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: status.hostIP
        - name: ENABLE_HTTP2_AUTO_DETECTION
          value: 'false'
        - name: ENABLE_HTTP_FULL_DUPLEX
          value: 'false'
        - name: ROOT_CA
        - name: ENABLE_MULTI_CONTAINER_PROBES
          value: 'false'
      resources:
        requests:
          cpu: 25m
      volumeMounts:
        - name: kube-api-access-7hshv
          readOnly: true
          mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      readinessProbe:
        httpGet:
          path: /
          port: 8012
          scheme: HTTP
          httpHeaders:
            - name: K-Network-Probe
              value: queue
        timeoutSeconds: 1
        periodSeconds: 10
        successThreshold: 1
        failureThreshold: 3
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      imagePullPolicy: IfNotPresent
      securityContext:
        capabilities:
          drop:
            - ALL
        runAsNonRoot: true
        readOnlyRootFilesystem: true
        allowPrivilegeEscalation: false
        seccompProfile:
          type: RuntimeDefault
  restartPolicy: Always
  terminationGracePeriodSeconds: 300
  dnsPolicy: ClusterFirst
  serviceAccountName: minio-for-kserve
  serviceAccount: minio-for-kserve
  nodeName: mattia-gp63-leopard-8re
  securityContext: {}
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 100
          podAffinityTerm:
            labelSelector:
              matchLabels:
                serving.knative.dev/revision: wonderland-predictor-00001
            topologyKey: kubernetes.io/hostname
  schedulerName: default-scheduler
  tolerations:
    - key: node.kubernetes.io/not-ready
      operator: Exists
      effect: NoExecute
      tolerationSeconds: 300
    - key: node.kubernetes.io/unreachable
      operator: Exists
      effect: NoExecute
      tolerationSeconds: 300
  priority: 0
  enableServiceLinks: false
  preemptionPolicy: PreemptLowerPriority
status:
  phase: Pending
  conditions:
    - type: PodReadyToStartContainers
      status: 'True'
      lastProbeTime: null
      lastTransitionTime: '2025-08-27T10:49:52Z'
    - type: Initialized
      status: 'False'
      lastProbeTime: null
      lastTransitionTime: '2025-08-27T10:49:50Z'
      reason: ContainersNotInitialized
      message: 'containers with incomplete status: [storage-initializer]'
    - type: Ready
      status: 'False'
      lastProbeTime: null
      lastTransitionTime: '2025-08-27T10:49:50Z'
      reason: ContainersNotReady
      message: 'containers with unready status: [kserve-container queue-proxy]'
    - type: ContainersReady
      status: 'False'
      lastProbeTime: null
      lastTransitionTime: '2025-08-27T10:49:50Z'
      reason: ContainersNotReady
      message: 'containers with unready status: [kserve-container queue-proxy]'
    - type: PodScheduled
      status: 'True'
      lastProbeTime: null
      lastTransitionTime: '2025-08-27T10:49:50Z'
  hostIP: 192.168.1.179
  hostIPs:
    - ip: 192.168.1.179
  podIP: 10.1.43.106
  podIPs:
    - ip: 10.1.43.106
  startTime: '2025-08-27T10:49:50Z'
  initContainerStatuses:
    - name: storage-initializer
      state:
        waiting:
          reason: CrashLoopBackOff
          message: >-
            back-off 10s restarting failed container=storage-initializer
            pod=wonderland-predictor-00001-deployment-66f66c6c68-559bk_kubeflow-user-example-com(1d23a6fd-908c-4d69-8844-7961baa9b14e)
      lastState:
        terminated:
          exitCode: 1
          reason: Error
          message: "2025/08/27 10:49:52 Initializing, args: src_uri [model-registry://iris-flower-classification/super-latest-version] dest_path[ [/mnt/models]\n2025/08/27 10:49:52 Falling back to base url model-registry-service.kubeflow-user-example-com.svc.cluster.local:8080 for model registry service\n2025/08/27 10:49:52 Download model indexed in model registry: modelName=, storageUri=model-registry://iris-flower-classification/super-latest-version, modelDir=/mnt/models\n2025/08/27 10:49:52 Parsed storageUri=model-registry://iris-flower-classification/super-latest-version as: modelRegistryUrl=model-registry-service.kubeflow-user-example-com.svc.cluster.local:8080, registeredModelName=iris-flower-classification, versionName=0xc0005204c0\n2025/08/27 10:49:52 Fetching model: registeredModelName=iris-flower-classification, versionName=0xc0005204c0\n2025/08/27 10:49:52 Fetching model version: model=&{0xc0005d20f0 <nil> <nil> iris-flower-classification 0xc00051c480 0xc00051c470 0xc00051c490 <nil> <nil> [] [] <nil> <nil> <nil> <nil> <nil> 0xc00051c4a0 0xc00051c4b0}\n2025/08/27 10:49:53 Fetching model artifacts: version=&{0xc00012c4a0 0xc000060240 <nil> super-latest-version 0xc000060270 0xc000060130 1 0xc000060250 0xc000060140 0xc000060260}\n2025/08/27 10:49:53 Error downloading the model: unable to get batch objects XMinioInvalidObjectName: Object name contains unsupported characters.\n\tstatus code: 400, request id: 185F9A149D1EE54A, host id: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8\n"
          startedAt: '2025-08-27T10:49:52Z'
          finishedAt: '2025-08-27T10:49:53Z'
          containerID: >-
            containerd://e62e291c66b8e39e2b3a21426a9bc0cb61201bd39aca9028b602b8e3e8766a2d
      ready: false
      restartCount: 1
      image: ghcr.io/kubeflow/model-registry/storage-initializer:v0.2.19
      imageID: >-
        ghcr.io/kubeflow/model-registry/storage-initializer@sha256:1a8613a7a49857d5658a3239f72042a53863cfb19137fad2745de1914283fa18
      containerID: >-
        containerd://e62e291c66b8e39e2b3a21426a9bc0cb61201bd39aca9028b602b8e3e8766a2d
      started: false
      volumeMounts:
        - name: kserve-provision-location
          mountPath: /mnt/models
        - name: kube-api-access-7hshv
          mountPath: /var/run/secrets/kubernetes.io/serviceaccount
          readOnly: true
          recursiveReadOnly: Disabled
  containerStatuses:
    - name: kserve-container
      state:
        waiting:
          reason: PodInitializing
      lastState: {}
      ready: false
      restartCount: 0
      image: >-
        index.docker.io/kserve/sklearnserver@sha256:d19adc0a6223d72e371a9cf852c56c910789ef0eac6a8baf96db35d0cc1d8304
      imageID: ''
      started: false
      volumeMounts:
        - name: kube-api-access-7hshv
          mountPath: /var/run/secrets/kubernetes.io/serviceaccount
          readOnly: true
          recursiveReadOnly: Disabled
        - name: kserve-provision-location
          mountPath: /mnt/models
          readOnly: true
          recursiveReadOnly: Disabled
    - name: queue-proxy
      state:
        waiting:
          reason: PodInitializing
      lastState: {}
      ready: false
      restartCount: 0
      image: >-
        gcr.io/knative-releases/knative.dev/serving/cmd/queue@sha256:698ef80ebc698f4d2bb93c1e85684063a0cf253a83faebcbf106cee444181d8e
      imageID: ''
      started: false
      volumeMounts:
        - name: kube-api-access-7hshv
          mountPath: /var/run/secrets/kubernetes.io/serviceaccount
          readOnly: true
          recursiveReadOnly: Disabled
  qosClass: Burstable

[tarilabs]

to continue from Slack thread, thanks so much for reporting!

am I right to assume your ModelArtifact.uri is

s3://my-fancy-bucket/models/whatever

in Model Registry?

the error looks to me it's thrown by the KServe CSI when called here

model-registry/internal/csi/storage/modelregistry_provider.go

Lines 97 to 112 in a1b414d

	// Call appropriate kserve provider based on the indexed model artifact URI
	if modelArtifact.Uri == nil {
	return fmt.Errorf("%w %s", ErrModelArtifactEmptyURI, *modelArtifact.Id)
	}
	protocol, err := p.extractProtocol(*modelArtifact.Uri)
	if err != nil {
	return err
	}
	provider, err := kserve.GetProvider(p.Providers, protocol)
	if err != nil {
	return err
	}
	return provider.DownloadModel(modelDir, "", *modelArtifact.Uri)

so it's definitely something behaving differently, just not sure yet exactly where the bug really lies.
we can definitely continue investigating via Slack but a github issue important not to lose track of this.

[MattiaSarti]

Thank you for following up, @tarilabs! I definitely agree discussing here is better.

About this:

am I right to assume your ModelArtifact.uri is
s3://my-fancy-bucket/models/whatever
in Model Registry?

Yes, you are right. When getting it by manually fetching model version artifacts from the Model Registry, it also has some query parameters, but for the sake of my comparison I am just using that same exact one, without query parameters (you can find a note at the end of the "InferenceService Definition" section in my bug report above).

[rareddy]

Is the code wrong in returning the versionName as a pointer instead of a string here?

model-registry/internal/csi/storage/modelregistry_provider.go

Line 143 in a1b414d

return registeredModelName, versionName, nil

[MattiaSarti]

@rareddy, that explains why the logs print some memory address instead of the string value, so it could be an additional fix to take into account to have proper logs, but then versionName seems to be dereferenced correctly where actually used, that is in fetchModelVersion:

model-registry/internal/csi/storage/modelregistry_provider.go

Line 154 in a1b414d

Name(*versionName).

[tarilabs]

the code of parseModelVersion IS correct in returning a string pointer at line 143, per the doc:

model-registry/internal/csi/storage/modelregistry_provider.go

Lines 115 to 120 in a1b414d

	// Possible URIs:
	// (1) model-registry://{modelName}
	// (2) model-registry://{modelName}/{modelVersion}
	// (3) model-registry://{modelRegistryUrl}/{modelName}
	// (4) model-registry://{modelRegistryUrl}/{modelName}/{modelVersion}
	func (p *ModelRegistryProvider) parseModelVersion(storageUri string) (string, *string, error) {

is meant to return the decomposition of the string which may not have a modelVersion.

I've filed #1540 at least to complement with better logging.l

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically closed because it has not had recent activity. Please comment "/reopen" to reopen it.

[MattiaSarti]

/reopen

[google-oss-prow]

@MattiaSarti: Reopened this issue.

Details

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.