proxy-cookie-path annotation validation doesn't seemingly support regex anymore -> v1.12.1

[johnsonr1994]

What happened:

After upgrading from v1.8.0 to v1.12.1 (via helm chart v4.12.1) our application which used regex pathing for the proxy-cookie-path annotation started erroring out in the ingress-nginx pod logs due to an "invalid character" being present -> Error encountered in pod logs:
validators.go:243] validation error on ingress $OurApplicationIngress: annotation proxy-cookie-path contains invalid value ~*^\/(?!PathSubstring) /PathSubstring

We think this issue ultimately stems from issue #9673 where annotation validation was added

NGINX Ingress controller version (exec into the pod and run /nginx-ingress-controller --version):

Kubernetes version (use kubectl version): 1.30.9

Environment:

• Cloud provider or hardware configuration: azure kubernetes service

• OS (e.g. from /etc/os-release): Linux

• Kernel (e.g. uname -a): Ubuntu - AKSUbuntu-2204gen2containerd-202503.02.0

• How was the ingress-nginx-controller installed:

  • If helm was used then please show output of helm ls -A | grep -i ingress

ingress-nginx-internal                  ingress-nginx-internal          9               2025-03-27 13:02:54.9988113 +0000 UTC   deployed        ingress-nginx-4.12.1                                                      1.12.1
ingress-nginx-public                    ingress-nginx-public            7               2025-03-27 13:03:18.3167122 +0000 UTC   deployed        ingress-nginx-4.12.1                                                      1.12.1

• If helm was used then please show output of helm -n <ingresscontrollernamespace> get values <helmreleasename>

USER-SUPPLIED VALUES:
controller:
admissionWebhooks:
  patch:
    image:
      digest: ""
      image: ingress-nginx/kube-webhook-certgen
      registry: $ACRNAME.azurecr.io
      tag: v20230407
    nodeSelector:
      kubernetes.io/os: linux
allowSnippetAnnotations: true
config:
  annotations-risk-level: Critical
extraArgs:
  ingress-class: nginx-internal
image:
  digest: ""
  image: ingress-nginx/controller
  registry: $ACRNAME.azurecr.io
  tag: v1.12.1
ingressClassResource:
  controllerValue: k8s.io/ingress-nginx-internal
  default: false
  enabled: true
  name: nginx-internal
nodeSelector:
  kubernetes.io/os: linux
replicaCount: 2
service:
  annotations:
    service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path: /healthz
    service.beta.kubernetes.io/azure-load-balancer-internal: "true"
  loadBalancerIP: $IP_ADDRESS
defaultBackend:
image:
  image: defaultbackend-amd64
  registry: $ACRNAME.azurecr.io
  tag: "1.5"
nodeSelector:
  kubernetes.io/os: linux

USER-SUPPLIED VALUES:                                                                                                                                                                                             
controller:                                                                                                                                                                                                       
  admissionWebhooks:                                                                                                                                                                                              
    patch:                                                                                                                                                                                                        
      image:                                                                                                                                                                                                      
        digest: ""                                                                                                                                                                                                
        image: ingress-nginx/kube-webhook-certgen
        registry: $ACRNAME.azurecr.io
        tag: v20230407
      nodeSelector:
        kubernetes.io/os: linux
  allowSnippetAnnotations: true
  config:
    annotations-risk-level: Critical
  extraArgs:
    ingress-class: nginx-public
  image:
    digest: ""
    image: ingress-nginx/controller
    registry: $ACRNAME.azurecr.io
    tag: v1.12.1
  ingressClassResource:
    controllerValue: k8s.io/ingress-nginx-public
    default: false
    enabled: true
    name: nginx-public
  nodeSelector:
    kubernetes.io/os: linux
  replicaCount: 2
  service:
    annotations:
      service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path: /healthz
      service.beta.kubernetes.io/azure-load-balancer-internal: "true"
    loadBalancerIP: $IPADDRESS
defaultBackend:
  image:
    image: defaultbackend-amd64
    registry: $ACRNAME.azurecr.io
    tag: "1.5"
  nodeSelector:
    kubernetes.io/os: linux

• Current State of the controller:
  • kubectl describe ingressclasses

Name:         nginx-internal
Labels:       app.kubernetes.io/component=controller
              app.kubernetes.io/instance=ingress-nginx-internal
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=ingress-nginx
              app.kubernetes.io/part-of=ingress-nginx
              app.kubernetes.io/version=1.12.1
              helm.sh/chart=ingress-nginx-4.12.1
Annotations:  meta.helm.sh/release-name: ingress-nginx-internal
              meta.helm.sh/release-namespace: ingress-nginx-internal
Controller:   k8s.io/ingress-nginx-internal
Events:       <none>


Name:         nginx-public
Labels:       app.kubernetes.io/component=controller
              app.kubernetes.io/instance=ingress-nginx-public
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=ingress-nginx
              app.kubernetes.io/part-of=ingress-nginx
              app.kubernetes.io/version=1.12.1
              helm.sh/chart=ingress-nginx-4.12.1
Annotations:  meta.helm.sh/release-name: ingress-nginx-public
              meta.helm.sh/release-namespace: ingress-nginx-public
Controller:   k8s.io/ingress-nginx-public
Events:       <none>

kubectl -n <ingresscontrollernamespace> describe po <ingresscontrollerpodname>

Name:             $POD_NAME
Namespace:        ingress-nginx-public
Priority:         0
Service Account:  ingress-nginx-public
Node:             $NODE_NAME
Start Time:       Thu, 27 Mar 2025 13:03:24 +0000
Labels:           app.kubernetes.io/component=controller
                app.kubernetes.io/instance=ingress-nginx-public
                app.kubernetes.io/managed-by=Helm
                app.kubernetes.io/name=ingress-nginx
                app.kubernetes.io/part-of=ingress-nginx
                app.kubernetes.io/version=1.12.1
                helm.sh/chart=ingress-nginx-4.12.1
                pod-template-hash=76dcfb5bc6
Annotations:      <none>
Status:           Running
IP:               $IPADDRESS
IPs:
IP:           $IPADDRESS
Controlled By:  ReplicaSet/$controller
Containers:
controller:
  Container ID:    containerd://$container
  Image:           $ACRNAME.azurecr.io/ingress-nginx/controller:v1.12.1
  Image ID:        $ACRNAME.azurecr.io/ingress-nginx/controller@sha256:$sha
  Ports:           80/TCP, 443/TCP, 8443/TCP
  Host Ports:      0/TCP, 0/TCP, 0/TCP
  SeccompProfile:  RuntimeDefault
  Args:
    /nginx-ingress-controller
    --enable-annotation-validation=false
    --publish-service=$(POD_NAMESPACE)/ingress-nginx-public-controller
    --election-id=ingress-nginx-public-leader
    --controller-class=k8s.io/ingress-nginx-public
    --ingress-class=nginx
    --configmap=$(POD_NAMESPACE)/ingress-nginx-public-controller
    --validating-webhook=:8443
    --validating-webhook-certificate=$cert
    --validating-webhook-key=$key
    --ingress-class=nginx-public
  State:          Running
    Started:      Thu, 27 Mar 2025 13:03:24 +0000
  Ready:          True
  Restart Count:  0
  Requests:
    cpu:      100m
    memory:   90Mi
  Liveness:   http-get http://:$PORT/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
  Readiness:  http-get http://:$PORT/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
  Environment:
    POD_NAME:         $POD_NAME (v1:metadata.name)
    POD_NAMESPACE:  ingress-nginx-public (v1:metadata.namespace)
    LD_PRELOAD:     /usr/local/lib/libmimalloc.so
  Mounts: $MOUNTS
Conditions:
Type                        Status
PodReadyToStartContainers   True
Initialized                 True
Ready                       True
ContainersReady             True
PodScheduled                True
Volumes:
webhook-cert:
  Type:        Secret (a volume populated by a Secret)
  SecretName:  ingress-nginx-public-admission
  Optional:    false
kube-api-access-$key:
  Type:                    Projected (a volume that contains injected data from multiple sources)
  TokenExpirationSeconds:  3607
  ConfigMapName:           kube-root-ca.crt
  ConfigMapOptional:       <nil>
  DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                           node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                           node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type    Reason     Age                From                      Message
----    ------     ----               ----                      -------
Normal  Scheduled  33m                default-scheduler         Successfully assigned ingress-nginx-public/ $POD_NAME
Normal  Pulled     33m                kubelet                   Container image "$ACRNAME.azurecr.io/ingress-nginx/controller:v1.12.1" already present on machine
Normal  Created    33m                kubelet                   Created container controller
Normal  Started    33m                kubelet                   Started container controller
Normal  RELOAD     29m (x3 over 33m)  nginx-ingress-controller  NGINX reload triggered due to a change in configuration

• kubectl -n <ingresscontrollernamespace> describe svc <ingresscontrollerservicename>

Name:                     ingress-nginx-public-controller
Namespace:                ingress-nginx-public
Labels:                   app.kubernetes.io/component=controller
                          app.kubernetes.io/instance=ingress-nginx-public
                          app.kubernetes.io/managed-by=Helm
                          app.kubernetes.io/name=ingress-nginx
                          app.kubernetes.io/part-of=ingress-nginx
                          app.kubernetes.io/version=1.12.1
                          helm.sh/chart=ingress-nginx-4.12.1
Annotations:              meta.helm.sh/release-name: ingress-nginx-public
                          meta.helm.sh/release-namespace: ingress-nginx-public
                          service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path: /healthz
                          service.beta.kubernetes.io/azure-load-balancer-internal: true
Selector:                 app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx-public,app.kubernetes.io/name=ingress-nginx
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       $IP
IPs:                      $IP
IP:                       $IP
LoadBalancer Ingress:     $IP
Port:                     http  80/TCP
TargetPort:               http/TCP
NodePort:                 http  $PORT/TCP
Endpoints:                $IPWithPort,$IPWithPort
Port:                     https  443/TCP
TargetPort:               https/TCP
NodePort:                 https  PORT/TCP
Endpoints:                $IPWithPort,$IPWithPort
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

• Current state of ingress object, if applicable:
  • kubectl -n <appnamespace> get all,ing -o wide
    Ingress in question:

NAME                        CLASS          HOSTS               ADDRESS      PORTS     AGE
$INGRESSNAME   nginx-public   $HOSTNAME       $IP                80, 443   7d22h

• kubectl -n <appnamespace> describe ing <ingressname>
  Ingress in question:

Labels:           app.kubernetes.io/managed-by=Helm
Namespace:        $namespace
Address:          $ip
Ingress Class:    nginx-public
Default backend:  <default>
TLS:
  $certname terminates $hostname
Rules:
  Host               Path  Backends
  ----               ----  --------
 $hostname
                     /$PathSubstring(/|$)(.*)   $servicename:80 ($IP:$Port)
Annotations:         meta.helm.sh/release-name: $servicename
                     meta.helm.sh/release-namespace: $servicename
                     nginx.ingress.kubernetes.io/configuration-snippet: proxy_set_header X-Original-Path $request_uri;
                     nginx.ingress.kubernetes.io/proxy-buffer-size: 128k
                     nginx.ingress.kubernetes.io/proxy-buffering: on
                     nginx.ingress.kubernetes.io/proxy-buffers-number: 4
                     nginx.ingress.kubernetes.io/proxy-cookie-path: ~*^/(?!$PathSubstring) /$PathSubstring
                     nginx.ingress.kubernetes.io/rewrite-target: /$2
                     nginx.ingress.kubernetes.io/use-regex: true
Events:
  Type    Reason  Age                From                      Message
  ----    ------  ----               ----                      -------
  Normal  Sync    40m (x2 over 43m)  nginx-ingress-controller  Scheduled for sync
  Normal  Sync    40m (x2 over 43m)  nginx-ingress-controller  Scheduled for sync

Repro:
Install specified 4.12.1 helm release with controller tag 1.12.1 and deploy an ingress resource with proxy-cookie-path annotation containing regex pathing. Described in the following docs https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_cookie_path

Observe pod logs and you should see the error message described at the beginning of this issue. The workaround we found is setting controller.enableAnnotationValidations = false on ingress-nginx deploy which does work but is unfortunate that we lose validation on all annotations now.

[k8s-ci-robot]

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[Monpoke]

Hello,
We had the same issue while upgrading to 1.12.1. I tried to downgrade to 1.11.5 and it's working.

The following annotation doesn't work anymore:
nginx.ingress.kubernetes.io/proxy-cookie-domain: ~^ ''

[Gacko]

Well, actually annotation validation was around in v1.11 already, just not enabled by default. So coming from v1.8.0, having it disabled would at least not change the behavior.

I'm totally aware that we might be missing some characters in our regular expression, but on the other hand it's also pretty hard to implement a regular expression to validate a regular expression without opening doors for exploiting this annotation or making the whole point of the validation senseless for users with plain values.