(secret watch predicate gap, exposed in v1.8.3 by #9486)
A ListenerSet whose certificateRefs Secret does not exist yet is correctly marked
Accepted=True / Programmed=False with ResolvedRefs=False (InvalidCertificateRef) —
but when the Secret is created afterwards, Envoy Gateway never re-reconciles. The
ListenerSet stays unprogrammed forever (until a controller restart or an unrelated
watched event forces a full reconcile).
This breaks the standard cert-manager flow: ListenerSet is created first, cert-manager
issues the TLS Secret ~1–2 minutes later (ACME), and the listener then never comes up.
Secret updates are equally dropped, so a renewed certificate is not loaded either —
Envoy keeps serving the pre-renewal cert.
Root cause: validateSecretForReconcile in internal/provider/kubernetes/predicates.go
checks whether a Secret is referenced by Gateways, SecurityPolicies,
ClientTrafficPolicies, HTTPRouteFilters and BackendTLSPolicies — but has no check for
ListenerSet certificateRefs. Up to v1.8.2 this was masked by the
isHTTPRouteFilterReferencingSecret fallthrough bug (return true for every Secret),
which made every Secret event in the cluster trigger a reconcile. #9486 (shipped in
v1.8.3) fixed that fallthrough to return false, removing the accidental safety net and
exposing the missing ListenerSet branch. main has the same gap as of 2026-07-30.
Expected: a create/update of a Secret referenced by a ListenerSet's
listeners[].tls.certificateRefs triggers reconciliation, same as for a Secret
referenced by a Gateway listener.
Supporting evidence for the predicate theory: annotating any Gateway-referenced
Secret (which passes the predicate) immediately triggers a full reconcile that also
resolves the stuck ListenerSet. Annotating the ListenerSet itself does nothing
(generation-filtered), and touching the ListenerSet-referenced Secret does nothing
(dropped by the predicate).
Suggested fix: add an isListenerSetReferencingSecret check (backed by a
ListenerSet-by-certificateRef-Secret index, mirroring the Gateway secret index) to
validateSecretForReconcile.
Repro steps:
-
Install Envoy Gateway v1.8.3, create a GatewayClass and a Gateway that allows
ListenerSets:
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: shared
namespace: envoy-gateway-system
spec:
gatewayClassName: eg
listeners:
- name: http
port: 80
protocol: HTTP
allowedListeners:
namespaces:
from: All
-
Create a ListenerSet referencing a Secret that does not exist yet:
apiVersion: gateway.networking.k8s.io/v1
kind: ListenerSet
metadata:
name: app
namespace: tenant-a
spec:
parentRef:
group: gateway.networking.k8s.io
kind: Gateway
name: shared
namespace: envoy-gateway-system
listeners:
- name: https
hostname: app.example.com
port: 443
protocol: HTTPS
tls:
mode: Terminate
certificateRefs:
- kind: Secret
name: app-example-com-tls
allowedRoutes:
namespaces:
from: Same
Status (correct at this point): Accepted=True, Programmed=False, listener
condition ResolvedRefs=False / InvalidCertificateRef ("Secret
tenant-a/app-example-com-tls does not exist").
-
Create the Secret afterwards (what cert-manager does ~1–2 min after issuance):
openssl req -x509 -newkey rsa:2048 -keyout tls.key -out tls.crt -days 1 \
-nodes -subj "/CN=app.example.com"
kubectl -n tenant-a create secret tls app-example-com-tls --cert=tls.crt --key=tls.key
-
Observe: the ListenerSet status never changes; the listener is never programmed.
Waiting minutes/hours changes nothing as long as no other watched resource emits an
event. kubectl annotate on the ListenerSet (metadata only) does not help.
-
Restart the controller (kubectl -n envoy-gateway-system rollout restart deploy/envoy-gateway) — the ListenerSet becomes Programmed=True within seconds,
confirming only the event trigger is missing, not the translation.
Also reproducible for renewals: update the Secret's key material — Envoy keeps serving
the old certificate until an unrelated reconcile happens.
Environment:
Logs:
Controller log when the ListenerSet is reconciled before the Secret exists (namespace
and names sanitized) — after the Secret is created, no further log line ever appears
for this ListenerSet:
2026-07-30T12:15:04.369Z error provider kubernetes/controller.go:2086 failed to process TLS SecretRef for ListenerSet
{"runner": "provider", "listenerSet": {"kind":"ListenerSet","apiVersion":"gateway.networking.k8s.io/v1",
"metadata":{"name":"app","namespace":"tenant-a", ...},
"spec":{"parentRef":{"group":"gateway.networking.k8s.io","kind":"Gateway","name":"shared","namespace":"envoy-gateway-system"},
"listeners":[{"name":"https","hostname":"app.example.com","port":443,"protocol":"HTTPS",
"tls":{"mode":"Terminate","certificateRefs":[{"group":"","kind":"Secret","name":"app-example-com-tls"}]}, ...}]}, ...},
"secretRef": {"group":"","kind":"Secret","name":"app-example-com-tls"},
"error": "unable to find the Secret tenant-a/app-example-com-tls: Secret \"app-example-com-tls\" not found"}
ListenerSet status stuck indefinitely:
status:
conditions:
- type: Accepted
status: "True"
reason: Accepted
- type: Programmed
status: "False"
reason: ListenersNotValid
message: No listeners are programmed
listeners:
- name: https
conditions:
- type: ResolvedRefs
status: "False"
reason: InvalidCertificateRef
message: 'No valid secrets exist: certificate refs 0: Secret tenant-a/app-example-com-tls does not exist.'
(secret watch predicate gap, exposed in v1.8.3 by #9486)
A
ListenerSetwhosecertificateRefsSecret does not exist yet is correctly markedAccepted=True / Programmed=FalsewithResolvedRefs=False (InvalidCertificateRef)—but when the Secret is created afterwards, Envoy Gateway never re-reconciles. The
ListenerSet stays unprogrammed forever (until a controller restart or an unrelated
watched event forces a full reconcile).
This breaks the standard cert-manager flow: ListenerSet is created first, cert-manager
issues the TLS Secret ~1–2 minutes later (ACME), and the listener then never comes up.
Secret updates are equally dropped, so a renewed certificate is not loaded either —
Envoy keeps serving the pre-renewal cert.
Root cause:
validateSecretForReconcileininternal/provider/kubernetes/predicates.gochecks whether a Secret is referenced by Gateways, SecurityPolicies,
ClientTrafficPolicies, HTTPRouteFilters and BackendTLSPolicies — but has no check for
ListenerSet
certificateRefs. Up to v1.8.2 this was masked by theisHTTPRouteFilterReferencingSecretfallthrough bug (return truefor every Secret),which made every Secret event in the cluster trigger a reconcile. #9486 (shipped in
v1.8.3) fixed that fallthrough to
return false, removing the accidental safety net andexposing the missing ListenerSet branch.
mainhas the same gap as of 2026-07-30.Expected: a create/update of a Secret referenced by a ListenerSet's
listeners[].tls.certificateRefstriggers reconciliation, same as for a Secretreferenced by a Gateway listener.
Supporting evidence for the predicate theory: annotating any Gateway-referenced
Secret (which passes the predicate) immediately triggers a full reconcile that also
resolves the stuck ListenerSet. Annotating the ListenerSet itself does nothing
(generation-filtered), and touching the ListenerSet-referenced Secret does nothing
(dropped by the predicate).
Suggested fix: add an
isListenerSetReferencingSecretcheck (backed by aListenerSet-by-certificateRef-Secret index, mirroring the Gateway secret index) to
validateSecretForReconcile.Repro steps:
Install Envoy Gateway v1.8.3, create a GatewayClass and a Gateway that allows
ListenerSets:
Create a ListenerSet referencing a Secret that does not exist yet:
Status (correct at this point):
Accepted=True,Programmed=False, listenercondition
ResolvedRefs=False / InvalidCertificateRef("Secrettenant-a/app-example-com-tls does not exist").
Create the Secret afterwards (what cert-manager does ~1–2 min after issuance):
openssl req -x509 -newkey rsa:2048 -keyout tls.key -out tls.crt -days 1 \ -nodes -subj "/CN=app.example.com" kubectl -n tenant-a create secret tls app-example-com-tls --cert=tls.crt --key=tls.keyObserve: the ListenerSet status never changes; the listener is never programmed.
Waiting minutes/hours changes nothing as long as no other watched resource emits an
event.
kubectl annotateon the ListenerSet (metadata only) does not help.Restart the controller (
kubectl -n envoy-gateway-system rollout restart deploy/envoy-gateway) — the ListenerSet becomesProgrammed=Truewithin seconds,confirming only the event trigger is missing, not the translation.
Also reproducible for renewals: update the Secret's key material — Envoy keeps serving
the old certificate until an unrelated reconcile happens.
Environment:
isHTTPRouteFilterReferencingSecretunconditionally returns true — every Secret write in the cluster triggers a full reconcile #9486; v1.8.2 behaved correctly byaccident — its
isHTTPRouteFilterReferencingSecretreturned true for every Secret).(ListenerSet standard channel).
arrives after the ListenerSet.
Logs:
Controller log when the ListenerSet is reconciled before the Secret exists (namespace
and names sanitized) — after the Secret is created, no further log line ever appears
for this ListenerSet:
ListenerSet status stuck indefinitely: