Skip to content

ListenerSet is never reconciled when its referenced TLS Secret is created after the ListenerSet #9614

Description

@dklesev

(secret watch predicate gap, exposed in v1.8.3 by #9486)

A ListenerSet whose certificateRefs Secret does not exist yet is correctly marked
Accepted=True / Programmed=False with ResolvedRefs=False (InvalidCertificateRef)
but when the Secret is created afterwards, Envoy Gateway never re-reconciles. The
ListenerSet stays unprogrammed forever (until a controller restart or an unrelated
watched event forces a full reconcile).

This breaks the standard cert-manager flow: ListenerSet is created first, cert-manager
issues the TLS Secret ~1–2 minutes later (ACME), and the listener then never comes up.
Secret updates are equally dropped, so a renewed certificate is not loaded either —
Envoy keeps serving the pre-renewal cert.

Root cause: validateSecretForReconcile in internal/provider/kubernetes/predicates.go
checks whether a Secret is referenced by Gateways, SecurityPolicies,
ClientTrafficPolicies, HTTPRouteFilters and BackendTLSPolicies — but has no check for
ListenerSet certificateRefs. Up to v1.8.2 this was masked by the
isHTTPRouteFilterReferencingSecret fallthrough bug (return true for every Secret),
which made every Secret event in the cluster trigger a reconcile. #9486 (shipped in
v1.8.3) fixed that fallthrough to return false, removing the accidental safety net and
exposing the missing ListenerSet branch. main has the same gap as of 2026-07-30.

Expected: a create/update of a Secret referenced by a ListenerSet's
listeners[].tls.certificateRefs triggers reconciliation, same as for a Secret
referenced by a Gateway listener.

Supporting evidence for the predicate theory: annotating any Gateway-referenced
Secret (which passes the predicate) immediately triggers a full reconcile that also
resolves the stuck ListenerSet. Annotating the ListenerSet itself does nothing
(generation-filtered), and touching the ListenerSet-referenced Secret does nothing
(dropped by the predicate).

Suggested fix: add an isListenerSetReferencingSecret check (backed by a
ListenerSet-by-certificateRef-Secret index, mirroring the Gateway secret index) to
validateSecretForReconcile.

Repro steps:

  1. Install Envoy Gateway v1.8.3, create a GatewayClass and a Gateway that allows
    ListenerSets:

    apiVersion: gateway.networking.k8s.io/v1
    kind: Gateway
    metadata:
      name: shared
      namespace: envoy-gateway-system
    spec:
      gatewayClassName: eg
      listeners:
        - name: http
          port: 80
          protocol: HTTP
      allowedListeners:
        namespaces:
          from: All
  2. Create a ListenerSet referencing a Secret that does not exist yet:

    apiVersion: gateway.networking.k8s.io/v1
    kind: ListenerSet
    metadata:
      name: app
      namespace: tenant-a
    spec:
      parentRef:
        group: gateway.networking.k8s.io
        kind: Gateway
        name: shared
        namespace: envoy-gateway-system
      listeners:
        - name: https
          hostname: app.example.com
          port: 443
          protocol: HTTPS
          tls:
            mode: Terminate
            certificateRefs:
              - kind: Secret
                name: app-example-com-tls
          allowedRoutes:
            namespaces:
              from: Same

    Status (correct at this point): Accepted=True, Programmed=False, listener
    condition ResolvedRefs=False / InvalidCertificateRef ("Secret
    tenant-a/app-example-com-tls does not exist").

  3. Create the Secret afterwards (what cert-manager does ~1–2 min after issuance):

    openssl req -x509 -newkey rsa:2048 -keyout tls.key -out tls.crt -days 1 \
      -nodes -subj "/CN=app.example.com"
    kubectl -n tenant-a create secret tls app-example-com-tls --cert=tls.crt --key=tls.key
  4. Observe: the ListenerSet status never changes; the listener is never programmed.
    Waiting minutes/hours changes nothing as long as no other watched resource emits an
    event. kubectl annotate on the ListenerSet (metadata only) does not help.

  5. Restart the controller (kubectl -n envoy-gateway-system rollout restart deploy/envoy-gateway) — the ListenerSet becomes Programmed=True within seconds,
    confirming only the event trigger is missing, not the translation.

Also reproducible for renewals: update the Secret's key material — Envoy keeps serving
the old certificate until an unrelated reconcile happens.

Environment:

Logs:

Controller log when the ListenerSet is reconciled before the Secret exists (namespace
and names sanitized) — after the Secret is created, no further log line ever appears
for this ListenerSet:

2026-07-30T12:15:04.369Z  error  provider  kubernetes/controller.go:2086  failed to process TLS SecretRef for ListenerSet
  {"runner": "provider", "listenerSet": {"kind":"ListenerSet","apiVersion":"gateway.networking.k8s.io/v1",
   "metadata":{"name":"app","namespace":"tenant-a", ...},
   "spec":{"parentRef":{"group":"gateway.networking.k8s.io","kind":"Gateway","name":"shared","namespace":"envoy-gateway-system"},
   "listeners":[{"name":"https","hostname":"app.example.com","port":443,"protocol":"HTTPS",
   "tls":{"mode":"Terminate","certificateRefs":[{"group":"","kind":"Secret","name":"app-example-com-tls"}]}, ...}]}, ...},
   "secretRef": {"group":"","kind":"Secret","name":"app-example-com-tls"},
   "error": "unable to find the Secret tenant-a/app-example-com-tls: Secret \"app-example-com-tls\" not found"}

ListenerSet status stuck indefinitely:

status:
  conditions:
  - type: Accepted
    status: "True"
    reason: Accepted
  - type: Programmed
    status: "False"
    reason: ListenersNotValid
    message: No listeners are programmed
  listeners:
  - name: https
    conditions:
    - type: ResolvedRefs
      status: "False"
      reason: InvalidCertificateRef
      message: 'No valid secrets exist: certificate refs 0: Secret tenant-a/app-example-com-tls does not exist.'

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions