Support for configurable tls in remote jwks store

[dmitri-d]

Description

This PR depends on changes in #13011. The PR introduces support for setting of tls options for connections to remote jwks sources in jwks store. This is accomplished by setting a backendRef on a AgentgatewayPolicy, finding a matching BackendTLSPolicy, and using it to configure tls options.

If no BackendTLSPolicy is present, an http client with default tls options is used (including setting InsecureSkipVerify to false).

Implemenation Notes

• Only k8s service and static agent backends can be pointed to in backendRefs. We should probably validate that jwksUri and the service/backend are consistent.
• Limited set of tls options can be configured:

• system or custom CA certs to use
• enabling/disabling InsecureSkipVerify. I don't think we support this anywhere else, could be confusing.

Change Type

/kind feature

Changelog

Support setting of tls options in connections to remote jwks sources.

Additional Notes

BackendRef isn't the only possible way to configure tls options for a policy; we have AgentgatewayPolicyBackendSimple (https://github.com/kgateway-dev/kgateway/blob/main/api/v1alpha1/agentgateway_policy_types.go#L120), which is used to configure agent policy backends, and allows for setting of tls, tcp, and http options on connections to backends. A jwks backend could be introduced to support setting of jwks url and connection options; tls for now, but possibly tcp in the future (retries comes first to mind). It might look something like:

apiVersion: gateway.kgateway.dev/v1alpha1
kind: AgentgatewayPolicy
metadata:
  name: gw-policy
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: super-gateway
  traffic:
    jwtAuthentication:
      mode: Strict
      providers:
      - issuer: https://kgateway.dev
        jwks:
          jwksBackend: # this would be a new agent policy backend
            jwksUri: https://dummy-idp.default:8443/org-one/keys
            tls:
              caCertificateRefs:
              - group: ""
                kind: ConfigMap
                name: ca
              insecureSkipVerify: true

Same policy, but tls options are configured using BackendTLSPolicy (this is implemented in this PR):

apiVersion: gateway.kgateway.dev/v1alpha1
kind: AgentgatewayPolicy
metadata:
  name: gw-policy
  namespace: default
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: super-gateway
  traffic:
    jwtAuthentication:
      mode: Strict
      providers:
      - issuer: https://kgateway.dev
        jwks:
          remote:
            cacheDuration: 5m
            jwksUri: https://dummy-idp.default:8443/org-one/keys
      - issuer: https://kgateway.dev
        jwks:
          remote:
            cacheDuration: 5m
            jwksUri: https://dummy-idp.default:8443/org-two/keys
            backendRef:
              group: ""
              kind: Service
              name: dummy-idp
              port: 8443
---
apiVersion: gateway.networking.k8s.io/v1
kind: BackendTLSPolicy
metadata:
  name: tls-policy
  namespace: default
spec:
  targetRefs:
  - group: ""
    kind: Service
    name: dummy-idp
  validation:
    hostname: "dummy-idp.default"
    caCertificateRefs:
    - group: ""
      kind: ConfigMap
      name: ca

I think the former is a better UX (simpler and more consistent between use-cases). There also might be an issue (not sure how realistic) when the same tls policy is applied to both jwks store traffic (traffic from the control plane) and to data-plane traffic.

Looking at MCP auth work (https://github.com/kgateway-dev/kgateway/pull/12966/files#diff-fb00835f76278e918399f81876d65569108fbd35b466698137ebae0c1c985ffdR44), another approach could be to use
the backend field in AgentgatewayPolicy (otherwise this is similar to the jwks backend example above).

It would also mean that MCP auth config and remote jwks configs would be in-line with each other.

[Copilot]

Pull request overview

This PR introduces support for configurable TLS options when connecting to remote JWKS sources in the JWKS store. It allows users to specify custom TLS configurations via BackendTLSPolicy by setting a backendRef on an AgentgatewayPolicy.

Key changes include:

• Refactored JWKS store architecture to use event-driven updates via channels instead of queue-based polling
• Added support for custom TLS configurations per JWKS source through BackendTLSPolicy integration
• Separated concerns into dedicated controllers: policy controller for JWKS source changes and ConfigMap controller for persistence
• Improved security by removing default InsecureSkipVerify: true from HTTP clients

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 13 comments.

Show a summary per file

File	Description
internal/kgateway/jwks/jwks_store.go	Refactored to use channel-based updates; added mapping between ConfigMap names and JWKS URIs
internal/kgateway/jwks/jwks_fetcher.go	Added TLS configuration support per keyset; simplified keyset management API; removed default insecure TLS
internal/kgateway/jwks/jwks_cache.go	Added thread-safe methods with mutex locks; removed comparison logic from add operation
internal/kgateway/jwks/config_map_syncer.go	Simplified by removing write logic and KRT collection; now only handles loading ConfigMaps
internal/kgateway/agentjwksstore/policy_controller.go	New controller that watches policies and creates JWKS sources with TLS configs from BackendTLSPolicy
internal/kgateway/agentjwksstore/cm_controller.go	New controller that persists JWKS to ConfigMaps using event queue pattern
internal/kgateway/controller/start.go	Updated to initialize new controllers in correct order
internal/kgateway/jwks/jwks_fetcher_test.go	Updated tests to use new defaultJwksClient field name
api/v1alpha1/agentgateway_policy_types.go	Made BackendRef a pointer and JwksUri required; removed mutual exclusivity validation
install/helm/kgateway-crds/templates/gateway.kgateway.dev_agentgatewaypolicies.yaml	Updated CRD to reflect API changes
api/v1alpha1/zz_generated.deepcopy.go	Generated code for pointer-based BackendRef deep copy

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review for a chance to win a $100 gift card. Take the survey.