CI Signal Hardening: Make Helm vs Kustomize parity check required

[Ashish-Kumar-Dash]

Validation Checklist

• I confirm that this is a Kubeflow-related issue.
• I am reporting this in the appropriate repository.
• I have followed the Kubeflow installation guidelines.
• The issue report is detailed and includes version numbers where applicable.
• I have considered adding my company to the adopters page to support Kubeflow and help the community, since I expect help from the community for my issue (see 1. and 2.).
• This issue pertains to Kubeflow development.
• I am available to work on this issue.
• You can join the CNCF Slack and access our meetings at the Kubeflow Community website. Our channel on the CNCF Slack is here #kubeflow-platform.

Version

master

Detailed Description

The Helm parity CI currently allows soft-fail behavior in the parity job through continue-on-error.
This causes pull requests to appear healthy even when Helm vs Kustomize parity checks fail.

Impact:

Maintainers can miss real parity regressions.
Contributors lose trust in CI signal.
Review latency increases because parity failures are discovered later.
Proposed scope:

Remove soft-fail behavior from the Helm parity workflow job.
Make parity check failures required and blocking for pull requests.
Keep this change workflow-only, without redesigning normalization or comparison algorithms.

Steps to Reproduce

Open a test branch and introduce a known mismatch between Helm and Kustomize manifests for one tested scenario.
Run the Helm parity workflow in CI and observe that the parity command fails internally.
Observe that the workflow job can still report success because soft-fail is enabled.
Expected result:

The workflow fails when parity fails.
Actual result:

The workflow can pass despite parity mismatch, masking regressions.

Screenshots or Videos (Optional)

No response

[github-actions]

Welcome to the Kubeflow Manifests Repository

Thanks for opening your first issue. We are trying to deal with your issue on a best-effort basis. without guarantees.
If you need commercial support instead please reach out to us.
Please make sure to follow our default issue template so maintainers can quickly understand and reproduce the problem instead of just closing the issue.
The template asks for:

• Environment details (Kubernetes version, OS, platform)
• Components involved
• Steps to reproduce
• Current versus expected behavior
• Properly formatted logs, screenshots or error messages

Community Resources:

• Join us on the CNCF Slack: https://www.kubeflow.org/docs/about/community/#kubeflow-slack-channels
• Attend our community meetings: https://www.kubeflow.org/docs/about/community/

Feel free to ask questions — we are here to help.

[Ashish-Kumar-Dash]

This change restores CI signal integrity: Helm and Kustomize parity mismatches will now fail pull requests instead of being silently tolerated. It is intentionally low risk because it only changes failure policy in workflow configuration, not chart templates or comparison logic. Do let me know your thoughts since it's a small but useful change

[christian-heusel]

Why is this a good idea? It sounds like as a consequence everyone that touches a kustomize manifest also has to change the equivalent helm files 🤔

[Ashish-Kumar-Dash]

That’s exactly the point of parity: if a Kustomize default changes, the Helm default should change too, or the chart silently stops representing the supported install path. The CI isn’t adding extra work arbitrarily, it makes divergence visible at PR time instead of letting drift accumulate for months and breaking users later. If a change is intentionally Kustomize-only, that should be an explicit decision with documented scope, not an accidental mismatch hidden by missing checks. These are my notes so far...

[christian-heusel]

Well yeah, but to my understanding the helm manifests are an experiment so far and for changes that are already a lot of work (i.e. see my last few merge requests) I don't want to 2x by also having to take care of the helm files.

If this is something desired then it would be good to get help from the people that are interested in the helm files 🤔

[Ashish-Kumar-Dash]

That makes sense, tho what I’m proposing is to make parity visible, then maybe let contributors who are specifically interested in Helm pick up the corresponding chart work instead of upstream changes silently drifting away from the charts. I’m happy to help on that side myself, since Helm manifest parity was a core part of my GSoC proposal and it’s an area I’m actively interested in contributing to, which is why I opened this issue :) Also, A dedicated parity pipeline can run separately on Helm/Kustomize-relevant paths, report drift per component, and initially be informational before becoming required once the normalization is trustworthy. That way upstream sync contributors get visibility, while Helm contributors can own the follow-up work

[christian-heusel]

But wouldn't it be a good idea to then have this running as a GHA cronjob or similar that opens an issue once there are differences? 🤔

[Ashish-Kumar-Dash]

Yup, I agree, also an even better long term setup can be to pair that with path scoped PR checks for Helm related changes only. That way we get visibility without immediately turning every upstream manifest change into a Helm maintenance requirement @juliusvonkohout thoughts

[juliusvonkohout]

@christian-heusel I will merge your PR also with the experimental helm failing and maybe even remove the old notebooks V1 Helm in the same PR. For notebooks V2 helm charts would be very useful and many other components already have them or plan to have them.

[Ashish-Kumar-Dash]

Hey, I understand and agree with keeping upstream synchronization work unblocked for now.
Could you advise the preferred next step, should I start with a scheduled non blocking Helm/Kustomize drift report workflow, then follow with path scoped informational checks?
I can implement whichever sequence you prefer and own the Helm-side follow-up fixes.

[juliusvonkohout]

The first thing would be helm charts for istio, Dex, oauth2-proxy, namespaces etc.