Skip to content

Commit 636f2ec

Browse files
sheidkampsheidkamp
authored
Frontend TLS (#13064)
Signed-off-by: sheidkamp <[email protected]>
1 parent b869d5d commit 636f2ec

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

48 files changed

+2898
-53
lines changed

‎api/annotations/gateway.go‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -56,4 +56,10 @@ const (
5656
// Use in the TLS options field of a TLS listener.
5757
// TODO: implement https://github.com/kgateway-dev/kgateway/issues/12955
5858
VerifySubjectAltNames gwv1.AnnotationKey = "kgateway.dev/verify-subject-alt-names"
59+
60+
// VerifyCertificateHash is the annotation key used to set the verify certificate hash used by the client.
61+
// The value is a comma or "-" separated list of certificate hashes which may be whitespace padded for readability.
62+
// Valid values are sha256 hashes in hex format, e.g "7D86C6654C8229364ECFE4D4964C69410090AE09E9B4D0C9B2AD7854175AD51D" or "7D:86:C6:65:4C:82:29:36:4E:CF:E4:D4:96:4C:69:41:00:90:AE:09:E9:B4:D0:C9:B2:AD:78:54:17:5A:D5:1D".
63+
// All characters, including formatting, are limited to 4096 characters by the annotation value specification https://gateway-api.sigs.k8s.io/reference/1.4/spec/#annotationvalue
64+
VerifyCertificateHash gwv1.AnnotationKey = "kgateway.dev/verify-certificate-hash"
5965
)

‎design/12721-e2e-testing-with-gateway-api-versions.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,9 +20,11 @@ In addition to different semver designated versions of the API, there are two ch
2020
* v1.3.0
2121
* XListenerSets added to experimental (not available in standard as of v1.4.0, planned for v1.5.0)
2222
* CORS filters added to experimental (not available in standard as of v1.4.0)
23+
* spec.defaultScope (type GatewayDefaultScope) added in experimental (not available in standard as of v1.4.0)
2324
* v1.4.0
2425
* BackendTLSPolicy promoted to v1 in standard and experimental. Previous v1alpha3 version is not supported.
2526
* HTTPRoutes.spec.rules[].name added to standard
27+
* spec.tls (GatewayTLSConfig) to Gateway in experimental channel
2628

2729
The are a substantial number of tests that need to be modified to
2830

‎docs/guides/frontend-tls.md‎

Lines changed: 334 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,334 @@
1+
@ -1,431 +0,0 @@
2+
# Manual Testing Guide: FrontendTLSConfig (mTLS)
3+
4+
This guide provides step-by-step instructions for manually testing the `FrontendTLSConfig` feature in a Kubernetes cluster, including certificate generation and configuration examples.
5+
6+
## Overview
7+
8+
`FrontendTLSConfig` enables mutual TLS (mTLS) on Gateway listeners by configuring client certificate validation. This guide demonstrates:
9+
- Default validation configuration (applies to all HTTPS listeners)
10+
- Per-port validation configuration (overrides default for specific ports)
11+
- Multiple CA certificate references support
12+
13+
## Prerequisites
14+
15+
- A Kubernetes cluster with kgateway installed
16+
- `openssl` installed locally
17+
- `kubectl` configured to access your cluster
18+
- `curl` for testing
19+
20+
## Step 1: Generate Certificates
21+
22+
### 1.1 Generate CA Certificate (for validating client certificates)
23+
24+
```bash
25+
# Create CA private key
26+
openssl genrsa -out ca-key.pem 2048
27+
28+
# Create CA certificate (valid for 1 year)
29+
openssl req -new -x509 -days 365 -key ca-key.pem -out ca-cert.pem \
30+
-subj "/CN=Test CA/O=Test Org"
31+
32+
# Optional: Generate a second CA certificate to test multiple CA refs
33+
openssl genrsa -out ca2-key.pem 2048
34+
openssl req -new -x509 -days 365 -key ca2-key.pem -out ca2-cert.pem \
35+
-subj "/CN=Test CA 2/O=Test Org"
36+
```
37+
38+
### 1.2 Generate Server Certificate (for TLS termination)
39+
40+
```bash
41+
# Create server private key
42+
openssl genrsa -out server-key.pem 2048
43+
44+
# Create server certificate signing request
45+
openssl req -new -key server-key.pem -out server.csr \
46+
-subj "/CN=example.com/O=Test Org"
47+
48+
# Create server certificate signed by CA (valid for 1 year)
49+
openssl x509 -req -days 365 -in server.csr -CA ca-cert.pem -CAkey ca-key.pem \
50+
-CAcreateserial -out server-cert.pem \
51+
-extensions v3_req -extfile <(echo "[v3_req]"; echo "subjectAltName=DNS:example.com,DNS:*.example.com")
52+
```
53+
54+
### 1.3 Generate Client Certificate (for mTLS client authentication)
55+
56+
```bash
57+
# Create client private key
58+
openssl genrsa -out client-key.pem 2048
59+
60+
# Create client certificate signing request
61+
openssl req -new -key client-key.pem -out client.csr \
62+
-subj "/CN=client.example.com/O=Test Org"
63+
64+
# Create client certificate signed by CA (valid for 1 year)
65+
openssl x509 -req -days 365 -in client.csr -CA ca-cert.pem -CAkey ca-key.pem \
66+
-CAcreateserial -out client-cert.pem
67+
```
68+
69+
## Step 2: Create Kubernetes Resources
70+
71+
### 2.1 Create Namespace
72+
73+
```bash
74+
kubectl create namespace test-mtls
75+
```
76+
77+
### 2.2 Create Server TLS Secret
78+
79+
```bash
80+
# Base64 encode server certificate and key
81+
SERVER_CERT=$(cat server-cert.pem | base64 -w 0)
82+
SERVER_KEY=$(cat server-key.pem | base64 -w 0)
83+
84+
# Create the secret
85+
kubectl create secret tls https-cert \
86+
--cert=server-cert.pem \
87+
--key=server-key.pem \
88+
-n test-mtls
89+
```
90+
91+
### 2.3 Create CA Certificate ConfigMaps
92+
93+
```bash
94+
# Create ConfigMap with CA certificate (for default validation)
95+
kubectl create configmap ca-cert-default \
96+
--from-file=ca.crt=ca-cert.pem \
97+
-n test-mtls
98+
99+
# Create ConfigMap with second CA certificate (for testing multiple refs)
100+
kubectl create configmap ca-cert-default-2 \
101+
--from-file=ca.crt=ca2-cert.pem \
102+
-n test-mtls
103+
104+
# Create ConfigMap with CA certificate (for per-port validation)
105+
kubectl create configmap ca-cert-per-port \
106+
--from-file=ca.crt=ca-cert.pem \
107+
-n test-mtls
108+
```
109+
110+
### 2.4 Create Backend Service
111+
112+
```bash
113+
# Create a simple httpbin service for testing
114+
kubectl create deployment httpbin --image=kennethreitz/httpbin -n test-mtls
115+
kubectl expose deployment httpbin --port=8000 --target-port=80 -n test-mtls
116+
```
117+
118+
## Step 3: Configure Gateway with FrontendTLSConfig
119+
120+
### 3.1 Gateway with Per-Port Override
121+
122+
This configuration shows per-port validation overriding the default:
123+
124+
```bask
125+
kubectl apply -f - <<EOF
126+
apiVersion: gateway.networking.k8s.io/v1
127+
kind: Gateway
128+
metadata:
129+
name: mtls-gateway-per-port
130+
namespace: test-mtls
131+
spec:
132+
gatewayClassName: kgateway
133+
tls:
134+
frontend:
135+
default:
136+
validation:
137+
mode: AllowValidOnly
138+
caCertificateRefs:
139+
- name: ca-cert-default
140+
kind: ConfigMap
141+
group: ""
142+
perPort:
143+
- port: 8444
144+
tls:
145+
validation:
146+
mode: AllowInsecureFallback
147+
caCertificateRefs:
148+
- name: ca-cert-per-port
149+
kind: ConfigMap
150+
group: ""
151+
listeners:
152+
- name: https-default
153+
protocol: HTTPS
154+
port: 8443
155+
tls:
156+
mode: Terminate
157+
certificateRefs:
158+
- name: https-cert
159+
kind: Secret
160+
- name: https-per-port
161+
protocol: HTTPS
162+
port: 8444
163+
tls:
164+
mode: Terminate
165+
certificateRefs:
166+
- name: https-cert
167+
kind: Secret
168+
EOF
169+
```
170+
171+
### 3.2 Create HTTPRoute
172+
173+
```bash
174+
# Apply HTTPRoute
175+
kubectl apply -f - <<EOF
176+
apiVersion: gateway.networking.k8s.io/v1
177+
kind: HTTPRoute
178+
metadata:
179+
name: mtls-route
180+
namespace: test-mtls
181+
spec:
182+
parentRefs:
183+
- name: mtls-gateway-per-port
184+
hostnames:
185+
- "example.com"
186+
rules:
187+
- matches:
188+
- path:
189+
type: PathPrefix
190+
value: /
191+
backendRefs:
192+
- name: httpbin
193+
port: 8000
194+
EOF
195+
```
196+
197+
## Step 4: Get Gateway External IP/Port or Port Forward
198+
199+
```bash
200+
# Get the Gateway service external IP
201+
kubectl get svc -n kgateway-system | grep kgateway
202+
203+
# Or if using port-forward for testing:
204+
kubectl port-forward -n test-mtls deploy/mtls-gateway-per-port 8443:8443 8444:8444 &
205+
```
206+
207+
## Step 5: Test the Configuration
208+
209+
### 5.1 Test Without Client Certificate (Should Fail with AllowValidOnly)
210+
211+
```bash
212+
# This should fail with SSL handshake error or 400 Bad Request
213+
curl -v -k https://localhost:8443/get \
214+
--resolve example.com:8443:127.0.0.1 \
215+
-H "Host: example.com"
216+
```
217+
218+
Expected: Connection failure or SSL handshake error
219+
220+
### 5.2 Test With Valid Client Certificate (Should Succeed)
221+
222+
```bash
223+
# This should succeed
224+
curl -v -k https://localhost:8443/get \
225+
--resolve example.com:8443:127.0.0.1 \
226+
-H "Host: example.com" \
227+
--cert client-cert.pem \
228+
--key client-key.pem \
229+
--cacert ca-cert.pem
230+
```
231+
232+
Expected: HTTP 200 OK with httpbin response
233+
234+
### 5.3 Test Per-Port Override (Port 8444 with AllowInsecureFallback)
235+
236+
```bash
237+
# Without client cert - should work with AllowInsecureFallback
238+
curl -v -k https://localhost:8444/get \
239+
--resolve example.com:8444:127.0.0.1 \
240+
-H "Host: example.com"
241+
242+
# With client cert - should also work
243+
curl -v -k https://localhost:8444/get \
244+
--resolve example.com:8444:127.0.0.1 \
245+
-H "Host: example.com" \
246+
--cert client-cert.pem \
247+
--key client-key.pem \
248+
--cacert ca-cert.pem
249+
```
250+
251+
Expected: Both requests should succeed (HTTP 200 OK)
252+
253+
254+
## Step 6: Verify Configuration
255+
256+
### 6.1 Check Gateway Status
257+
258+
```bash
259+
kubectl get gateway mtls-gateway-per-port -n test-mtls -o yaml
260+
```
261+
262+
Look for:
263+
- `status.listeners[].conditions` - should show `Accepted: True`
264+
- Any error conditions related to certificate references
265+
266+
### 6.2 Check Envoy Configuration (if accessible)
267+
268+
If you have access to Envoy admin interface or xDS dump:
269+
270+
```bash
271+
# Port-forward to Envoy admin
272+
kubectl port-forward -n test-mtls deployment/mtls-gateway-per-port 19000:19000
273+
274+
# Check listener configuration
275+
curl http://localhost:19000/config_dump | jq '.configs[2].dynamic_listeners[] | select(.name | contains("8443"))'
276+
```
277+
278+
Look for:
279+
- `downstream_tls_context.require_client_certificate: true` (for AllowValidOnly)
280+
- `downstream_tls_context.common_tls_context.validation_context.trusted_ca` (should contain CA cert)
281+
282+
### 6.3 Check Logs
283+
284+
```bash
285+
# Check kgateway controller logs
286+
kubectl logs -n kgateway-system -l app.kubernetes.io/name=kgateway --tail=100 | grep -i "frontend\|mtls\|client.*cert"
287+
288+
# Check Envoy logs
289+
kubectl logs -n test-mtls -l app.kubernetes.io/name=mtls-gateway-per-port --tail=100 | grep -i "ssl\|tls\|certificate"
290+
```
291+
292+
## Troubleshooting
293+
294+
### Issue: Connection refused
295+
296+
- Verify Gateway is listening on the expected port
297+
- Check Gateway status for listener conditions
298+
- Verify the Gateway service is exposing the correct ports
299+
300+
### Issue: Certificate validation errors
301+
302+
- Verify CA certificate ConfigMap contains valid PEM data under `ca.crt` key
303+
- Check that client certificate is signed by one of the configured CA certificates
304+
- Verify certificate hasn't expired
305+
306+
### Issue: Client certificate not required
307+
308+
- Verify `mode: AllowValidOnly` is set (not `AllowInsecureFallback`)
309+
- Check that `caCertificateRefs` is properly configured
310+
- Verify the Gateway resource was applied correctly
311+
312+
313+
## Cleanup
314+
315+
```bash
316+
# Delete resources
317+
kubectl delete gateway mtls-gateway-per-port -n test-mtls
318+
kubectl delete httproute mtls-route -n test-mtls
319+
kubectl delete configmap ca-cert-default ca-cert-default-2 ca-cert-per-port -n test-mtls
320+
kubectl delete secret https-cert -n test-mtls
321+
kubectl delete svc httpbin -n test-mtls
322+
kubectl delete deployment httpbin -n test-mtls
323+
kubectl delete namespace test-mtls
324+
325+
# Clean up local certificate files
326+
rm -f *.pem *.csr *.srl
327+
```
328+
329+
## Additional Notes
330+
331+
- **AllowValidOnly**: Requires valid client certificate, rejects connections without one
332+
- **AllowInsecureFallback**: Accepts connections with or without client certificates
333+
- **Multiple CA refs**: All configured CA certificates are trusted, client certs signed by any of them are accepted
334+
- **Per-port override**: Per-port configuration takes precedence over default configuration

‎pkg/kgateway/extensions2/plugins/backendtlspolicy/plugin.go‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -75,7 +75,7 @@ func NewPlugin(ctx context.Context, commoncol *collections.CommonCollections) sd
7575
)
7676
col := krt.WrapClient(cli, commoncol.KrtOpts.ToOptions("BackendTLSPolicy")...)
7777

78-
translate := buildTranslateFunc(commoncol.ConfigMaps)
78+
translate := buildTranslateFunc(commoncol.ConfigMaps.Collection())
7979

8080
policyStatusMarker, tlsPolicyCol := krt.NewStatusCollection(col, func(krtctx krt.HandlerContext, i *gwv1.BackendTLSPolicy) (*krtcollections.StatusMarker, *ir.PolicyWrapper) {
8181
tlsPolicyIR, err := translate(krtctx, i)

‎pkg/kgateway/extensions2/plugins/trafficpolicy/gateway_extension.go‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -202,7 +202,7 @@ func TranslateGatewayExtensionBuilder(
202202
case gExt.JWT != nil:
203203
jwtConfig, err := resolveJwtProviders(
204204
krtctx,
205-
commoncol.ConfigMaps,
205+
commoncol.ConfigMaps.Collection(),
206206
commoncol.BackendIndex,
207207
gExt.ObjectSource,
208208
gExt.Name,

0 commit comments

Comments
 (0)