Kubeflow 26.03 istio-ingressgateway accessible through port-forward, not Ingress

[gioargyr]

Validation Checklist

• I confirm that this is a Kubeflow-related issue.
• I am reporting this in the appropriate repository.
• I have followed the Kubeflow installation guidelines.
• The issue report is detailed and includes version numbers where applicable.
• I have considered adding my company to the adopters page to support Kubeflow and help the community, since I expect help from the community for my issue (see 1. and 2.).
• This issue pertains to Kubeflow development.
• I am available to work on this issue.
• You can join the CNCF Slack and access our meetings at the Kubeflow Community website. Our channel on the CNCF Slack is here #kubeflow-platform.

Version

master

Detailed Description

I upgraded from v1.11.0 to 26.03

When everything was stable I created an Ingress that points to service/istio-ingressgateway in istio-system as always.
This time, does not work at all and I cannot have access to Kubeflow. (it's the same Ingress that worked for v1.8.0, v1.9.1, v1.10.2, and v1.11.0)
However, if port-forward the service I have access to it from localhost!
Do you have any ideas on that one?
If I delete any leftovers from previous Kubeflow versions (e.g. all services in istio-system) so they are re-deployed, will it fix any problem?

For what it's worth, I want to mention the following:

• The while command for installing Kubeflow runs repeatedly, even when everything seem to be deployed without any errors. Probably because of these errors:

Error from server (Invalid): CustomResourceDefinition.apiextensions.k8s.io "domainmappings.serving.knative.dev" is invalid: status.storedVersions[0]: Invalid value: "v1alpha1": missing from spec.versions; v1alpha1 was previously a storage version, and must remain in spec.versions until a storage migration ensures no data remains persisted in v1alpha1 and removes v1alpha1 from status.storedVersions
Error from server (Invalid): CustomResourceDefinition.apiextensions.k8s.io "inferenceservices.serving.kserve.io" is invalid: [spec.conversion.strategy: Required value, spec.conversion.webhookClientConfig: Forbidden: should not be set when strategy is not set to Webhook]
Error from server (Invalid): ClusterRoleBinding.rbac.authorization.k8s.io "meta-controller-cluster-role-binding" is invalid: roleRef: Invalid value: {"APIGroup":"rbac.authorization.k8s.io","Kind":"ClusterRole","Name":"kubeflow-metacontroller"}: cannot change roleRef

• When I run this command: kubectl port-forward svc/istio-ingressgateway -n istio-system 3000:80 for port-forwarding istio-ingressgateway to my machine, the logs seem to forward port 8080, and not 80:

Forwarding from 127.0.0.1:3000 -> 8080
Forwarding from [::1]:3000 -> 8080
Handling connection for 3000

Is this normal?

Steps to Reproduce

1. Upgraded from v1.11.0 to 26.03

Screenshots or Videos (Optional)

No response

[juliusvonkohout]

Hello, did you read https://github.com/kubeflow/manifests?tab=readme-ov-file#upgrading-and-extending ? it should also be kubectl port-forward svc/istio-ingressgateway -n istio-system 8080:80. I do offer commercial consulting if you need help with your specific deployment.