traefik-crds helm chart 1.12.0 fails to upgrade with --set gatewayAPIExperimental="true"

[FrederikChristiani]

Welcome!

• Yes, I've searched similar issues on GitHub and didn't find any.
• Yes, I've searched similar issues on the Traefik community forum and didn't find any.

What version of the Traefik's Helm Chart are you using?

traefik-crds: 1.12.0 (traefik: 37.2.0)

What version of Traefik are you using?

3.5.3

What did you expect to happen ?

I expected the upgrade to complete when following the upgrade procedure here: https://github.com/traefik/traefik-helm-chart?tab=readme-ov-file#upgrade-when-using-both-traefik-and-opt-in-crds-chart

What did you notice instead ?

The upgrade failed.

$ helm search repo traefik/traefik
NAME                    CHART VERSION   APP VERSION     DESCRIPTION                                       
traefik/traefik         37.3.0          v3.6.0          A Traefik based Kubernetes ingress controller     
traefik/traefik-crds    1.12.0                          A Traefik based Kubernetes ingress controller     
traefik/traefik-hub     4.2.0           v2.11.0         Traefik Hub Ingress Controller                    
traefik/traefik-mesh    4.1.1           v1.4.8          Traefik Mesh - Simpler Service Mesh               
traefik/traefikee       4.2.5           v2.12.5         Traefik Enterprise is a unified cloud-native ne...

$ helm --namespace traefik-crds ls
NAME            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                   APP VERSION
traefik-crds    traefik-crds    8               2025-11-07 10:41:52.315257361 +0100 CET deployed        traefik-crds-1.11.1

$helm upgrade --namespace traefik-crds traefik-crds --set gatewayAPIExperimental="true" traefik/traefik-crds
Error: UPGRADE FAILED: create: failed to create: Secret "sh.helm.release.v1.traefik-crds.v9" is invalid: data: Too long: may not be more than 1048576 bytes

What are your values ?

gatewayAPIExperimental="true"

Additional Information

I assume the upgrade failed because helm tries to save the entire chart in a k8s secret.
helm/helm#9788

[mloiseleur]

@FrederikChristiani wow. We are kind of blocked, here. I'm unsure what we can do on the Chart level about this 🤔.

Gateway API CRDs are already above 1MB:

$ du -hs traefik-crds/crds-files/*
1,9M	traefik-crds/crds-files/gatewayAPI
244K	traefik-crds/crds-files/hub
416K	traefik-crds/crds-files/knative
300K	traefik-crds/crds-files/traefik

Did you try to delete the secret "sh.helm.release.v1.traefik-crds.v9" and try again ? Does this kind of workaround works ?

[FrederikChristiani]

The secret was never created.
Version 1.11.1 was already large (but working), so I guess helm isn't saving both standard and experimental Gateway API CRDs.
But the recent changes to experimental must have brought it over the edge :(

$ git checkout crds_v1.11.1 
HEAD is now at 75bdb79 chore(release): :rocket: publish traefik 37.1.2 and crds 1.11.1

$ du -hs traefik-crds/crds-files/*
1.6M    traefik-crds/crds-files/gatewayAPI
128K    traefik-crds/crds-files/hub
200K    traefik-crds/crds-files/traefik

$ du -hs traefik-crds/crds-files/gatewayAPI/*
976K    traefik-crds/crds-files/gatewayAPI/gateway-experimental-install.yaml
604K    traefik-crds/crds-files/gatewayAPI/gateway-standard-install.yaml

$ git checkout crds_v1.12.0
HEAD is now at 972666c chore(release): :rocket: publish traefik 37.3.0 and 1.12.0

$ du -hs traefik-crds/crds-files/gatewayAPI/*
1.2M    traefik-crds/crds-files/gatewayAPI/gateway-experimental-install.yaml
680K    traefik-crds/crds-files/gatewayAPI/gateway-standard-install.yaml

[mloiseleur]

It's very sad, but apart from stopping this CRD chart, I fail to see what else we can do :/.

Related issue:

• Feature: Helm release data larger than 1MB - proposal to split across multiple Secrets helm/helm#10986

It does not make sense to provide a chart that cannot work with helm.

cc @jnoordsij @bpsoraggi @darkweaver87

[mloiseleur]

@FrederikChristiani Since Helm 4 is able to use ServerSide Apply, and the rc1 is available, do you think you can test helm v4 rc1 and see if it's better with SSA ?

[FrederikChristiani]

That doesn't seem to make a difference (if I'm doing this correctly).

$ ./linux-arm64/helm version
version.BuildInfo{Version:"v4.0.0-rc.1", GitCommit:"99cd1964357c793351be481d55abbe21c6b2f4ec", GitTreeState:"clean", GoVersion:"go1.25.3", KubeClientVersion:"v1.34"}

$ ./linux-arm64/helm upgrade --namespace traefik-crds traefik-crds --set gatewayAPIExperimental="true" --version 1.12.0 --server-side true traefik/traefik-crds
Error: UPGRADE FAILED: create: failed to create: Secret "sh.helm.release.v1.traefik-crds.v11" is invalid: data: Too long: may not be more than 1048576 bytes