Error when upgrading to 26.03.1

[gioargyr]

Validation Checklist

• I confirm that this is a Kubeflow-related issue.
• I am reporting this in the appropriate repository.
• I have followed the Kubeflow installation guidelines.
• The issue report is detailed and includes version numbers where applicable.
• I have considered adding my company to the adopters page to support Kubeflow and help the community, since I expect help from the community for my issue (see 1. and 2.).
• This issue pertains to Kubeflow development.
• I am available to work on this issue.
• You can join the CNCF Slack and access our meetings at the Kubeflow Community website. Our channel on the CNCF Slack is here #kubeflow-community-distribution.

Version

26.03.1

Detailed Description

Hello,

I tried to upgrade Kubeflow from 26.03 to 26.03.1 using the documentation you have on the README section of this repository and I always get the same error in llmisvc-controller-manager Pod:

{"level":"error","ts":"2026-07-10T09:42:03Z","logger":"controller-runtime.source.EventHandler","msg":"failed to get informer from cache","error":"Timeout: failed waiting for *v1alpha2.LLMInferenceServiceConfig Informer to sync","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1.1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/source/kind.go:76\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.34.5/pkg/util/wait/loop.go:53\nk8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext\n\t/go/pkg/mod/k8s.io/apimachinery@v0.34.5/pkg/util/wait/loop.go:54\nk8s.io/apimachinery/pkg/util/wait.PollUntilContextCancel\n\t/go/pkg/mod/k8s.io/apimachinery@v0.34.5/pkg/util/wait/poll.go:33\nsigs.k8s.io/controller-runtime/pkg/internal/source.(*Kind[...]).Start.func1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.19.7/pkg/internal/source/kind.go:64"}
E0710 09:42:03.464889       1 reflector.go:205] "Failed to watch" err="failed to list *v1alpha2.LLMInferenceServiceConfig: client rate limiter Wait returned an error: context canceled" logger="UnhandledError" reflector="pkg/mod/k8s.io/client-go@v0.34.5/tools/cache/reflector.go:290" type="*v1alpha2.LLMInferenceServiceConfig"
{"level":"info","ts":"2026-07-10T09:42:03Z","msg":"Stopping and waiting for webhooks"}
{"level":"info","ts":"2026-07-10T09:42:03Z","logger":"controller-runtime.webhook","msg":"Shutting down webhook server with timeout of 1 minute"}
{"level":"info","ts":"2026-07-10T09:42:03Z","msg":"Stopping and waiting for HTTP servers"}
{"level":"info","ts":"2026-07-10T09:42:03Z","logger":"controller-runtime.metrics","msg":"Shutting down metrics server with timeout of 1 minute"}
{"level":"info","ts":"2026-07-10T09:42:03Z","msg":"shutting down server","name":"health probe","addr":"[::]:8081"}
{"level":"info","ts":"2026-07-10T09:42:03Z","msg":"Wait completed, proceeding to shutdown the manager"}
{"level":"error","ts":"2026-07-10T09:42:03Z","logger":"setup","msg":"unable to run the manager","error":"storage version migration for llminferenceservices.serving.kserve.io failed: unable to fetch crd llminferenceservices.serving.kserve.io - customresourcedefinitions.apiextensions.k8s.io \"llminferenceservices.serving.kserve.io\" is forbidden: User \"system:serviceaccount:kubeflow:llmisvc-controller-manager\" cannot get resource \"customresourcedefinitions\" in API group \"apiextensions.k8s.io\" at the cluster scope","stacktrace":"main.main\n\t/go/src/github.com/kserve/kserve/cmd/llmisvc/main.go:320\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:285"}

For now, I continue having deployed Kubeflow in version 26.03, but I thought I should report this one.

Steps to Reproduce

1. Install Kubeflow 26.03.1, or upgrade from 26.03, to 26.03.1

Screenshots or Videos (Optional)

No response