Adding mTLS kuadrant installation mode #1170

laurafitzgerald · 2025-02-20T13:10:54Z

What

Changes

Verification steps

The storyboard is:

Run kuadrant with nTLS mode OFF. Then apply some rate limit and auth policies and run traffic.
Switch mTLS mode to ON. Verify traffic gets handled as expected and required configuration is in place.
Switch mTLS mode back to OFF. Verify traffic gets handled as expected and required mTLS configuration is no longer in the resources. The purpose of this step is to show reconciliation capabilities of the involved controllers.

Run the following on this branch.

make local-setup GATEWAYAPI_PROVIDER=istio

Apply the Kuadrant custom resource with mtls disabled

kubectl -n kuadrant-system apply -f - <<EOF
---
apiVersion: kuadrant.io/v1beta1
kind: Kuadrant
metadata:
  name: kuadrant-sample
spec: 
  mtls:
    enable: false
EOF

Wait for kuadrant instance to be ready

kubectl wait --timeout=300s --for=condition=Ready kuadrant kuadrant-sample -n kuadrant-system

Verify mTLS is disabled

kubectl get kuadrant kuadrant-sample -n kuadrant-system

Result should show false

NAME              MTLS    AGE
kuadrant-sample   false   3m42s

Follow steps in Secure, protect, and connect APIs with Kuadrant

Skip DNS sections as not relevant for these verification steps
As there is no real DNS resolution, testing the policies should be done providing a custom IP address for a name to curl tool. As follows:

export INGRESS_HOST=$(kubectl get gtw ${KUADRANT_GATEWAY_NAME} -n ${KUADRANT_GATEWAY_NS} -o jsonpath='{.status.addresses[0].value}')
export INGRESS_PORT=$(kubectl get gtw ${KUADRANT_GATEWAY_NAME} -n ${KUADRANT_GATEWAY_NS} -o jsonpath='{.spec.listeners[?(@.name=="api")].port}')

while :; do 
  curl -k --write-out '%{http_code}\n' --silent --output /dev/null --resolve api.$KUADRANT_ZONE_ROOT_DOMAIN:$INGRESS_PORT:$INGRESS_HOST "https://api.$KUADRANT_ZONE_ROOT_DOMAIN:$INGRESS_PORT/cars" | grep -E --color "\b(429)\b|$"
  sleep 1
done

Turn mtls ON

kubectl patch kuadrant kuadrant-sample --type=merge --patch '{"spec": {"mtls": {"enable": true}}}' -n kuadrant-system

Verify mTLS is enabled

kubectl get kuadrant kuadrant-sample -n kuadrant-system

Result should show true

NAME              MTLS   AGE
kuadrant-sample   true   15m

Verify traffic still gets rate limited and auth policy being applied

while :; do 
  curl -k --write-out '%{http_code}\n' --silent --output /dev/null --resolve api.$KUADRANT_ZONE_ROOT_DOMAIN:$INGRESS_PORT:$INGRESS_HOST "https://api.$KUADRANT_ZONE_ROOT_DOMAIN:$INGRESS_PORT/cars" | grep -E --color "\b(429)\b|$"
  sleep 1
done

It should return 403 and 429s.

Verify that a Peer Authentication exists

kubectl get peerauthentication default -n kuadrant-system -o yaml

which should have the following spec

spec:
  mtls:
    mode: STRICT
  selector:
    matchLabels:
      kuadrant.io/managed: "true"

Verify that the ratelimit cluster envoyfilter has a patch with a key "transport_socket" in spec.configPatches

kubectl get envoyfilter kuadrant-ratelimiting-external -n api-gateway -o jsonpath='{.spec.configPatches[0].patch.value.transport_socket}' | yq e -P

which should have the following spec

name: envoy.transport_sockets.tls
typed_config:
  '@type': type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
  common_tls_context:
    tls_certificate_sds_secret_configs:
      - name: default
        sds_config:
          api_config_source:
            api_type: GRPC
            grpc_services:
              - envoy_grpc:
                  cluster_name: sds-grpc
    validation_context_sds_secret_config:
      name: ROOTCA
      sds_config:
        api_config_source:
          api_type: GRPC
          grpc_services:
            - envoy_grpc:
                cluster_name: sds-grpc

Verify that the auth cluster envoyfilter has a patch with a key "transport_socket" in spec.configPatches

kubectl get envoyfilter kuadrant-auth-external -n api-gateway -o jsonpath='{.spec.configPatches[0].patch.value.transport_socket}' | yq e -P

which should have the following spec

name: envoy.transport_sockets.tls
typed_config:
  '@type': type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
  common_tls_context:
    tls_certificate_sds_secret_configs:
      - name: default
        sds_config:
          api_config_source:
            api_type: GRPC
            grpc_services:
              - envoy_grpc:
                  cluster_name: sds-grpc
    validation_context_sds_secret_config:
      name: ROOTCA
      sds_config:
        api_config_source:
          api_type: GRPC
          grpc_services:
            - envoy_grpc:
                cluster_name: sds-grp

Verify that the limitador deployment has the expected labels

kubectl get deployment limitador-limitador -n kuadrant-system -o yaml | yq e '.spec.template.metadata.labels'

Which should return

app: limitador
kuadrant.io/managed: "true"
limitador-resource: limitador
sidecar.istio.io/inject: "true"

Verify that the authorino deployment has the expected labels

kubectl get deployment authorino -n kuadrant-system -o yaml | yq e '.spec.template.metadata.labels'

Which should return

authorino-resource: authorino
control-plane: controller-manager
kuadrant.io/managed: "true"
sidecar.istio.io/inject: "true"

Turn mtls OFF

kubectl patch kuadrant kuadrant-sample --type=merge --patch '{"spec": {"mtls": null}}' -n kuadrant-system

Verify mTLS is disabled

kubectl get kuadrant kuadrant-sample -n kuadrant-system

Result should show false

NAME              MTLS    AGE
kuadrant-sample   false   62m

Verify traffic still gets rate limited and auth policy being applied

while :; do 
  curl -k --write-out '%{http_code}\n' --silent --output /dev/null --resolve api.$KUADRANT_ZONE_ROOT_DOMAIN:$INGRESS_PORT:$INGRESS_HOST "https://api.$KUADRANT_ZONE_ROOT_DOMAIN:$INGRESS_PORT/cars" | grep -E --color "\b(429)\b|$"
  sleep 1
done

It should return 403 (forbidden) and 429s (too many requests).

Verify that a Peer Authentication does not exist.

kubectl get peerauthentication default -n kuadrant-system -o yaml

which should return

Error from server (NotFound): peerauthentications.security.istio.io "default" not found

Verify that the ratelimit cluster envoyfilter does not have a patch with a key "transport_socket" in spec.configPatches

kubectl get envoyfilter kuadrant-ratelimiting-external -n api-gateway -o jsonpath='{.spec.configPatches[0].patch.value.transport_socket}' | yq e -P

which should return empty.

Verify that the auth cluster envoyfilter does not have a patch with a key "transport_socket" in spec.configPatches

kubectl get envoyfilter kuadrant-auth-external -n api-gateway -o jsonpath='{.spec.configPatches[0].patch.value.transport_socket}' | yq e -P

which should return empty.

Verify that the limitador deployment has the istio integration label set to false

kubectl get deployment limitador-limitador -n kuadrant-system -o yaml | yq e '.spec.template.metadata.labels."sidecar.istio.io/inject"'

Which should return false.

Verify that the authorino deployment has the istio integration label set to false

kubectl get deployment authorino -n kuadrant-system -o yaml | yq e '.spec.template.metadata.labels."sidecar.istio.io/inject"'

Which should return false.

Boomatang

There is a few comments that I have. I didn't try running this locally.

I would like to point out with the state of the world and the topology any time you find your self request data from the cluster you are doing something wrong

controllers/mtls_reconciler.go

controllers/state_of_the_world.go

pkg/limitador/conditions.go

pkg/reconcilers/deployment.go

controllers/mtls_reconciler.go

pkg/limitador/conditions.go

Boomatang

I think this is now at the stage where we need to be able to compile and run the operator to do a good review. It is getting close.

api/v1beta1/kuadrant_types.go

controllers/mtls_reconciler.go

eguzki

Very good progress.

Few questions:

If the gateway provider is not Istio and a user enables mtls, should we report in kuadrant status?
~~When a user deletes a gateway, the peerAuthentications objects gets deleted? (not very important now and can be tackled in follow up PR's)~~ (I learned that one peerAuthentications is created at the kuadrant namespace)

api/v1beta1/topology.go

config/samples/kuadrant_v1beta1_kuadrant.yaml

controllers/auth_workflow_helpers.go

controllers/istio_auth_cluster_reconciler.go

controllers/mtls_reconciler.go

pkg/istio/utils.go

pkg/limitador/conditions.go

pkg/reconcilers/deployment.go

controllers/istio_ratelimit_cluster_reconciler.go

eguzki · 2025-03-17T14:56:10Z

Trying to run the verification steps: it fails with

{"level":"info","ts":"2025-03-17T14:54:07Z","logger":"kuadrant-operator","msg":"pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:243: failed to list *v1.PeerAuthentication: peerauthentications.security.istio.io is forbidden: User \"system:serviceaccount:kuadrant-system:kuadrant-operator-controller-manager\" cannot list resource \"peerauthentications\" in API group \"security.istio.io\" at the cluster scope"}

Consider granting required permissions to the operator

controllers/mtls_reconciler.go

laurafitzgerald · 2025-03-20T11:40:22Z

@eguzki thanks for you reviews. I've been caught up with other work but starting to address those now. Need to rebase first. 🤕 😂

Signed-off-by: Laura Fitzgerald <[email protected]>

Cannot add the deployment of Authorino to the topology without adding all the cluster deployment to the topology. Signed-off-by: Eguzki Astiz Lezaun <[email protected]>

Signed-off-by: Eguzki Astiz Lezaun <[email protected]>

eguzki · 2025-04-08T16:48:53Z

@david-martin done with this. Ready for another review. I just addressed your comment and added the link from kuadrant CR to the PeerAuthentication on the topology.

david-martin · 2025-04-09T10:11:39Z

tests/commons.go

@@ -691,3 +692,14 @@ func IsRLPAcceptedAndEnforced(g Gomega, ctx context.Context, cl client.Client, p
 	g.Expect(enforcedCond.Status).To(Equal(metav1.ConditionTrue))
 	g.Expect(enforcedCond.Reason).To(Equal(string(kuadrant.PolicyReasonEnforced)))
 }
+
+func AuthorionIsReady(cl client.Client, key client.ObjectKey) func(g Gomega, ctx context.Context) {


Suggested change

func AuthorionIsReady(cl client.Client, key client.ObjectKey) func(g Gomega, ctx context.Context) {

func AuthorinoIsReady(cl client.Client, key client.ObjectKey) func(g Gomega, ctx context.Context) {

sorry :)

Signed-off-by: Eguzki Astiz Lezaun <[email protected]>

david-martin · 2025-04-09T14:47:28Z

Nice one @laurafitzgerald @eguzki getting this change over the line.
Quite the epic.

laurafitzgerald requested a review from a team as a code owner February 20, 2025 13:10

laurafitzgerald requested a review from Boomatang February 20, 2025 13:11

laurafitzgerald marked this pull request as draft February 20, 2025 13:12

Boomatang reviewed Feb 20, 2025

View reviewed changes

laurafitzgerald force-pushed the mtls branch from 7b55e2a to 83a6d54 Compare February 26, 2025 15:16

Boomatang reviewed Mar 4, 2025

View reviewed changes

pkg/limitador/conditions.go Outdated Show resolved Hide resolved

Boomatang reviewed Mar 7, 2025

View reviewed changes

api/v1beta1/kuadrant_types.go Show resolved Hide resolved

controllers/mtls_reconciler.go Outdated Show resolved Hide resolved

KevFan reviewed Mar 10, 2025

View reviewed changes