v2.1.0-rc.1
Pre-release
Pre-release
🎉 Welcome to the v2.1.0-rc.1 release of the kgateway project!
Installation
The kgateway project is available as a Helm chart and docker images.
Helm Charts
The Helm chart is available at cr.kgateway.dev/kgateway-dev/charts/kgateway.
Docker Images
The docker images are available at:
- cr.kgateway.dev/kgateway-dev/kgateway:v2.1.0-rc.1
- cr.kgateway.dev/kgateway-dev/sds:v2.1.0-rc.1
- cr.kgateway.dev/kgateway-dev/envoy-wrapper:v2.1.0-rc.1
Quickstart
Try installing this release:
helm install kgateway-crds oci://cr.kgateway.dev/kgateway-dev/charts/kgateway-crds --version v2.1.0-rc.1 --namespace kgateway-system --create-namespace
helm install kgateway oci://cr.kgateway.dev/kgateway-dev/charts/kgateway --version v2.1.0-rc.1 --namespace kgateway-system --create-namespace
For detailed installation instructions and next steps, please visit our quickstart guide.
Changelog
Breaking Changes
- Updates the status API for TrafficPolicy and HTTPListenerPolicy to use Gateway API v1alpha2.PolicyStatus API. (#11141)
- Switching to Envoy's
/stats/prometheus?usedonlyendpoint to only get statistics that Envoy has updated (counters incremented at least once, gauges changed at least once, and histograms added to at least once). (#11358) - Use
kgateway.dev/inherited-policy-priority: ShallowMergePreferParentinstead ofdelegation.kgateway.dev/inherited-policy-priority: PreferParentandkgateway.dev/inherited-policy-priority: ShallowMergePreferChildinstead ofdelegation.kgateway.dev/inherited-policy-priority: PreferChild, as annotations to define inherited policy priority for delegated routes. By default, child HTTPRoute policies take precedence over parent HTTPRoute policies for delegated routes. (#11675) - remove insecureSkipVerify field from Backend and AI ssl validation (#11819)
- Adds disable field to extAuth, extProc, cors, buffer policies to allow disabling the policies per-route. Breaking change: extAuth.enablement has been removed in favor of extAuth.disable. (#11893)
- Inference: Replaces InferencePool v1alpha2 with v1 (#11965)
- Add generic gRPC request timeout to GatewayExtension gRPC services. Add failOpen support to all GatewayExtension external providers. Change ExtProc GatewayExtension provider to failOpen by default. (#12239)
- Rename agentGateway to agentgateway for consistency in helm values. Rename GatewayParameters agentGateway field to agentgateway. (#12293)
- As waypoint functionality is alpha, disable it by default. It can be enabled by setting the
waypoint.enabledhelm value to true (#12385)
New Features
- Enables kgateway to act as the control plane for agentgateway. (#11151)
- Enables policy attachment using labels using the
targetSelectorsAPI for kgateway policy APIs. (#11163) - Introduce BYO global rate limiting so operators can expose an external rate-limit service through a
GatewayExtensionresource and reference that extension from aTrafficPolicy. This enables users to configure both local and cluster-wide quotas within the same API surface. (#11169) - Add a setting to toggle the listener bind address to either ipv4 or ipv6 (#11196)
- Add support for dynamic forward proxy. (#11197)
- Introduce BackendConfigPolicy api to allow configuring envoy clusters. (#11214)
- Enables setting annotations on
Deploymentgenerated bykgatewayHelm chart. (#11224) - Adds InferencePool status management to Inference Extension endpointpicker (EPP) Plugin. (#11230)
- Enables multiple kgateway installs in separate namespaces, and implements
discoveryNamespaceSelectorsto control the namespaces that are considered for config discovery by a kgateway instance based on label selectors. (#11238) - Respect DestinationRule TCP keepalive settings (#11246)
- CORS support has been added and can be configured in the
TrafficPolicyor inHTTPRoute, depending on the desired policy. (#11252) - Allows a Kubernetes gateway to have more than 64 listeners by implementing ListenerSets defined in https://gateway-api.sigs.k8s.io/geps/gep-1713. Listener Sets can define their own listeners and be mapped to a parent gateway via their parentRef. The Kubernetes gateway will have the merged list of all listeners from itself and attached ListenerSets. This experimental feature requires the
xlistenersets.gateway.networking.x-k8s.ioCRD to be present. (#11255) - Invalid durations in our CRDs will now be rejected using CEL, before the CR is admitted. (#11266)
- Allow TrafficPolicy to targetRef using section name. (#11272)
- Add
PathOverrideandAuthHeaderOverridefields for custom LLM provider endpoints (#11282) - add TargetSelectors field in BackendConfigPolicySpec to enable selection of resources with matchLabels. (#11289)
- Support for CSRF policy has been added to the TrafficPolicy. (#11302)
- backendconfigpolicy: add ssl config (#11308)
- Support sessionPersistence on HTTPRoute (#11320)
- Add control plane metrics support for observability of controller, collections, and translation operations. (#11342)
- Adds initial InferencePool e2e tests (#11344)
- added support for extended gateway parameters (#11346)
- Support Service appProtocols
http2,grpc, andgrpc-web. (#11352) - backendconfigpolicy: add load balancer configuration (#11365)
- Enables configuring the payload transformation mode for AWS Lambda backends. (#11381)
- Allow configuring app protocol on Static Backends. (#11384)
- add health check config to backendconfigpolicy (#11393)
- For kubernetes services, set IgnoreHealthOnHostRemoval to true on the cluster. (#11395)
- Adds support for OpenTelemetry Tracing & Access Log Support. This can be configured via the HTTPListenerPolicy (#11396)
- add http2 protocol options to backendconfigpolicy (#11455)
- Add useRemoteAddress, xffNumTrustedHops, serverHeaderTransformation, and streamIdleTimeout to HTTPListenerPolicy #11231 (#11462)
- Users can now define custom environment variables for the envoy proxy container via the gateway parameters. It can be specified as a list via
GatewayParameters.spec.kube.envoyContainer.env. (#11463) - Added image, security context and resource configuration on GatewayParameters for agentgateway. (#11464)
- Enables sorting of HTTPRoutes using weights assigned with the kgateway.dev/route-weight annotation when KGW_WEIGHTED_ROUTE_PRECEDENCE=true. (#11470)
- Added CEL validation to enforce proper attachment semantics for policy APIs. This ensures that policies can only be attached to valid Gateway API resources. (#11499)
- Allow setting listener-level perConnectionBufferLimitBytes by setting the
kgateway.dev/per-connection-buffer-limitannotation on the gateway. (#11505) - Privileged ports used (< 1024) on a listener are no longer mapped to a higher port. If listeners are using privileged ports before upgrading, there may be down time when the port mapping is updated. (#11508)
- Add support for setting request buffer limit using TrafficPolicy (#11523)
- The TrafficPolicy plugin now respects the route replacement mode setting (
KGW_ROUTE_REPLACEMENT_MODE). When in strict mode, the plugin performs additional validation to catch invalid configurations before they reach Envoy. Invalid policies that would cause Envoy to NACK at runtime (e.g. malformed templates) will now be replaced with a direct response (HTTP 500) and report clear status conditions. This prevents fail-open scenarios where invalid policies could allow unintended traffic. (#11553) - CORS's
allowOriginsnow fully supports the format defined by the gateway api v1.3.0 spec, including wildcards. (#11581) - add hash policy to TrafficPolicy for configuring hashing loadbalancers (#11583)
- Adds the ability for resources to attach to policies defined in the global policy namespace when using targetSelectors. (#11585)
- Added comprehensive KGateway load testing framework implementing gateway-api-bench methodology with VCluster simulation for fake cluster resources, baseline (1000 routes) and production (5000 routes) performance tests measuring Gateway API control plane performance through incremental route testing with real traffic validation, event-driven monitoring for precise timing measurements, GitHub Actions integration for CI/CD workflows with optional release validation and nightly testing across multiple Kubernetes versions, Makefile targets for easy execution, VS Code debug configurations for development, and complete documentation. (#11598)
- Adds support for Envoy HealthCheck filter policy to HTTPListenerPolicy (#11629)
- Add OTel instrumentation for AI non-streaming requests following Gen AI semantic conventions (#11670)
- CI: Adds support for running Gateway API Inference Extension conformance tests. (#11679)
- Add InsecureSkipVerify option to backendconfigpolicy. This allows for TLS without verifying server certificates. (#11743)
- Support traffic distribution modes to prefer endpoints close to the kgateway / waypoint with failover to other priorities. (#11793)
- Updated kgateway agentgateway integration to support the latest agentgateway. (#11816)
- Add option for preserving http1 header case to httplistenerpolicy (#11829)
- Add option to preserve http1 header casing in BackendConfigPolicy (#11836)
- Enable the IngressUseWaypoint feature by default. Users can still opt-out by setting an environment variable
KGW_INGRESS_USE_WAYPOINTSto false. (#11857) - Added DirectResponse Support in agentgateway (#11859)
- Add support for leader election. This is enabled by default and can be disabled by setting the
disableLeaderElectionsetting (#11890) - Adds
topologySpreadConstraintsto the Pod struct used in GatewayParameters in order to set the corresponding topologySpreadConstrains field in the gateway-proxy pod. (#11913) - Added AWS Bedrock support for agentgateway. Bumped agentgateway to v0.7.3. (#11933)
- Extend the route replacement functionality so that when kgateway runs in STRICT mode it prevents invalid Envoy route configuration from ever reaching the proxies. It primarily covers HTTPRoute rules that either 1.) define invalid matchers (e.g. bad regular expressions) or 2.) use built-in Gateway API filters that translate into invalid Envoy xDS. (#11939)
- TrafficPolicy supports configuring timeouts at the route level, and retries at the route and gateway listener level. (#11970)
- Add header modifiers, using the API from HTTPHeaderFilter, to TrafficPolicy resources. (#11985)
- Added support for extauth in agentgateway TrafficPolicies. (#11993)
- Expose
acceptHttp10anddefaultHostForHttp10options via httplistenerpolicy to accept incomingHTTP 1.0andHTTP 0.9requests. (#12009) - Added custom configmap support for agentgatway. (#12013)
- Add support for resource attributes in OTel access logs (#12019)
- Expose
envoy'sidle_timeoutviaHTTPListenerPolicy. (#12020) - Supports passive health checking (outlier detection). (#12025)
- Support applying HPA for a gateway by setting a flag in the GatewayParameters (#12045)
- Added CEL-based rbac support. (#12054)
- Added agentgateway rbac support. (#12066)
- Adds support for a TrafficPolicy referencing a GatewayExtension in a different namespace than the policy using a new
namespacefield. (#12067) - Added externaltrafficpolicy support. (#12089)
- Enables optional deep merging of extAuth, extProc, transformation policies in TrafficPolicy for policies attached to the same resource. Enables the ability to prioritize policies and GatewayExtensions using the kgateway.dev/policy-weight annotation. (#12111)
- Added the ability to configure additional resources to agentgateway syncer. Added the ability to configure ExtraVolumes and ExtraVolumeMount via GatewayParameters. (#12117)
- backendconfigpolicy: option to use system CA certs for TLS (#12149)
- Added support for specifying a backend annotation that will disable Istio auto-mtls for that backend when Istio is enabled (#12176)
- Added Transformation support in agentgateway (#12202)
- Add default support for graceful shutdown and zero-downtime rollout of gateway proxies. (#12242)
- Helm charts allow for specifying a rollout strategy for the controller; GatewayParameters allows for specifying a rollout strategy for Envoy. (#12247)
- Add various configuration options to the various external service providers in
GatewayExtensions(#12252) - Added statuses to TrafficPolicies in agentgateway. (#12256)
- Add "Accepted" column to TrafficPolicy and BackendConfigPolicy kubectl output for easier status monitoring (#12303)
- Rename agentgateway controller to
kgateway.dev/agentgateway. Added separate xds port (agw-grpc-xds) for agentgateway. (#12323) - Enable per-provider BackendTLSPolicy attachment for AI backends. (#12369)
- Allow downstreams to provide extra AgentgatewayPolicyStatusSyncHandler for policy status reporting with agentgateway (#12377)
- Added CEL validation to TrafficPolicy transformation policy when using agentgateway. (#12404)
- Deprecates GatewayParameters.FloatingUserId in favor of OmitDefaultSecurityContext (#12418)
Bug Fixes
- Backend targeted policy such as BackendTLSPolicy can now target ServiceEntry either via the
networking.istio.io/ServiceEntrygroup/kind, or the syntheticnetworking.istio.io/Hostnamegroup/kind. (#11212) - Turn off ambient DNS capture by default for kgateway-waypoint, fixing traffic loops in ServiceEntry with DNS resolution. (#11216)
- Gateways' Status.Addresses will now include Spec.Addresses. This allows other controllers that rely on Status.Addresses to read what is specified for self-managed gateways. An example of this is Istio reading the address for a self-managed Waypoint. (#11311)
- Fix a bug where the
AttachedListenerSetscondition is sometimes incorrectly set. (#11321) - Header modification now works on backendRefs inside http route rules. (#11336)
- bugfix: only Apply deployed objects if they are different. (#11338)
- Fix
BackendTLSPolicyto use multiple targetRefs if provided (#11364) - fix an issue with BackendConfigPolicy where Envoy would reject configuration if only CommonHttpProtocolOptions were specified (#11420)
- Fixed an issue where HTTPRoute status updates could fail when reporting translation errors. Previously, status updates would fail schema validation when routes had invalid configurations (like invalid path prefix matchers). Status conditions are now properly set with valid reason fields and detailed error messages. (#11427)
- Makes HTTPBackendRef.Port optional when referencing an InferencePool, ignoring the port if specified. (#11448)
- prevent BackendConfigPolicy with http1protocoloptions set from overwriting an http2 backend (#11449)
- Fixes deprecation warning by replacing deprecated MatchSubjectAltNames field with MatchTypedSubjectAltNames in TLS validation context for BackendConfigPolicy. (#11520)
- enforce max limit on HTTP/2 initial stream and connection window sizes, reporting an error on the policy if max is exceeded. (#11525)
- add validation to ensure maxRequestSize is greater than 0 (#11564)
- validate that HTTPRoute backendRef weights add to greater than zero (#11567)
- Parent HTTPRoutes with unresolved child routes will return a 500 direct response. (#11604)
- Fix HTTPRoute session persistence: marshal
StatefulSessionPerRouteper‐route config instead ofStatefulSession(#11618) - fix: handle changing the value of the PerConnectionBufferLimitBytes annotation on the gateway (#11626)
- fix: correctly set Gateway listener status when protocol is unsupported (#11631)
- Support having
ingress-use-waypointon backend service alias namespace. (#11680) - use lower case for log level and use JSON format for krt logs to be consistent (#11688)
- Adds retries to gateway controller and proxy syncer when updating Gateway status. (#11697)
- Correctly report status for attached policies and gatewayExtensions when only errors change (#11721)
- Validation improvements for TrafficPolicy rate limit descriptors (#11803)
- Fixed the agentgateway TCPRoutes. Fixed the label selector MCP route translation. (#11854)
- Fix consistent CI failures in GIE conformance tests (#11858)
- HTTPRoute status now correctly reflects error when referencing a missing extensionRef (#11883)
- Fixed an issue when dynamically modifying the traffic distribution won't change the distribution. (#11953)
- Add error messages to ListenerSet status conditions (#12044)
- The data-plane backend builder now resolves endpoints from all pods that match the InferencePool’s selector. (#12050)
- Implement listener precedence with listenersets, fix listenersets not to inherit gateway routes, and update status reporting on listenersets. (#12091)
- Added support for listener and route rule policy attachment, bumped agentgateway to 0.7.8. (#12136)
- gateway translator: don't add listener with no filter chains (#12165)
- Use DelayedInformer for BackendTLSPolicy to handle missing CRDs (#12178)
- Descriptive ResolvedRefs condition error message about specifying port in the backendRef (#12190)
- FIX CEL rule evaluation in Backend API for k8s <= 1.31. (#12194)
- Change ExtAuth ext_authz filter to the earlier AuthN stage in the chain (#12300)
- Validate the CA certificate referenced in BackendTLSPolicy (#12402)
- Fixed BackendConfigPolicy outlier detection interval validation to prevent "0s" values that cause Envoy NACKs. (#12403)
- Fixed BackendConfigPolicy TLS parsing to prevent nil pointer panic when minVersion or maxVersion is omitted. (#12409)
Documentation
Cleanup
- Add
kgatewaylabel to data plane pods (#11241) - moved reusable parts of Deployer into pkg/deployer; default GatewayParameter values can be reused in downstream projects (#11377)
- rename ssl config to tls (#11401)
- remove HeadersWithUnderscoresAction from BackendConfigPolicy (#11440)
- Refactored settings test suites to use consistent standard library testing patterns (#11518)
- Bumps the inference extension CRDs to sync with upstream commit 842603b. (#11539)
- Update policy APIs to use pointer types for optional fields, and value types within slices. (#11606)
- Endpoint-picker: Rebuild
InferencePool.status.parentsdeterministically, eliminating flapping. Reconcile InferencePool status based on HTTPRoute, Service, and InferencePool events. HonorHTTPBackendRef.namespacewhen matching pools. Avoid data races in Service reconciliation by using a mutex to lock inferencePool IR errors. (#11621) - Inference: Bumps Kgateway to support release v0.5.0-rc.2 (#11715)
- Inference: Replaces deprecated AddToScheme() with Install() to install the Inference Extension v1alpha2 scheme. (#11729)
- API validation tests now log the output from
kubectl applywhen an unexpected error occurs, making it easier to debug failing test cases. (#11772) - Inference: Added EPP fail-open support by refactoring the inference plugin to use static and subset load-balancing clusters. (#11810)
- Tooling: Adds make targets for managing Gateway API and Gateway API Inference Extension dependencies. (#11830)
- Removed knative dependency for calculating the service FQDN. (#11861)
- The global ratelimit filter's defaults have changed to failOpen=true and timeout=100ms. In addition, removed the statPrefix and turned off enableXRatelimitHeaders. (#12035)
- Bumped agentgateway to v0.7.5. (#12062)
- Bumps InferencePool to v1.0.0-rc.2.
inferencePool.spec.endpointPickerRef.portNumberfield has been replaced withinferencePool.spec.endpointPickerRef.port.number. (#12166) - Updated envoy to v1.35.2 (#12209)
- Inference: Bumps the Gateway API Inference Extension (GIE) dependency from v1.0.0-rc.2 to v1.0.0. (#12241)
- Inference: Updates the agentgateway e2e test Helm values to include inference extension. (#12268)
- Report accurate Gateway Status conditions for types
AcceptedandProgrammedbased on Listener status. If a Gateway has a any Listener with a ListenerConditionProgrammed condition with a Status of false, the GatewayConditionAccepted on the Gateway will be reported with a "Status: true" and "Reason: ListenersNotValid". (#12290) - AI Backend API has been updated to simplify the API. Use ai.llm. instead of
ai.llm.provider.<provider>. Use ai.priorityGroups[].providers[] instead of ai.multipool.priorities[].pool[].provider. (#12327) - If an XListenerSet has any Listener with a ListenerConditionProgrammed condition with a Status of false, the GatewayConditionAccepted on the XListenerSet will be reported with a "Status: true" and "Reason: ListenersNotValid" (#12342)
- Added gosec to the linter workflow. Fixed int types to follow Kubernetes Gateway API standards for primitive types. (#12358)
Dependency Updates
- sigs.k8s.io/gateway-api bumped from v1.2.1 to 1.3.0 (#11263)