envoy-spire-opa-service-mesh

Demo to build Service Mesh on Kubernetese using Envoy as data plane and SPIRE and OPA as control plane. This demo is zlabjp/spiffejp-demo with OPA added.

• Use Kubernetes 1.17.0
• Use Envoy 1.12.2, SPIRE 0.9.0 and OPA 0.15.1
• Envoy uses SPIRE as SDS Server to obtain TLS certificate
• Envoy uses OPA as External Authorization Server to check if the incoming request is authorized or not

Blog is here. (Japanese)

Policy Overview

Four services are running in Kubernetes Cluster.

• ec-web（ spiffe://example.org/ec-web ）
  • ec-web ONLY accept requests with custom header "X-Opa-Secret" containing "ec-web-secret" value
• ec-backend（ spiffe://example.org/ec-backend ）
  • ec-backend can ONLY be accessed from ec-web
• news-web（ spiffe://example.org/news-web ）
  • news-web ONLY accept requests with custom header "X-Opa-Secret" containing "news-web-secret" value
• news-backend（ spiffe://example.org/news-backend ）
  • news-backend can ONLY be accessed from news-web

Architecture

• Authentication using SPIRE
• Authorization using OPA

1. Create Kubernetes Cluster

Create a Kubernetes cluster with the required flags to run NodeAttestor "k8s_psat" in SPIRE.

minikube start \
    --kubernetes-version v1.17.0 \
    --extra-config=apiserver.service-account-signing-key-file=/var/lib/minikube/certs/sa.key \
    --extra-config=apiserver.service-account-key-file=/var/lib/minikube/certs/sa.pub \
    --extra-config=apiserver.service-account-issuer=api \
    --extra-config=apiserver.service-account-api-audiences=api,spire-server

2. Deploy SPIRE Server

Deploy SPIRE Server as StatefulSet.

kubectl apply -f spire-server.yaml

3. Deploy SPIRE Agent

Deploy SPIRE Agent as DaemonSet.

kubectl apply -f spire-agent.yaml

4. Create Registration Entries in SPIRE Server

Create node entry.

kubectl exec -n spire spire-server-0 -- /opt/spire/bin/spire-server entry create \
    -spiffeID spiffe://example.org/node \
    -selector k8s_psat:cluster:demo-cluster \
    -node

Create workload entries.

kubectl exec -n spire spire-server-0 -- /opt/spire/bin/spire-server entry create \
    -parentID spiffe://example.org/node \
    -spiffeID spiffe://example.org/ec-web \
    -selector k8s:pod-label:app:ec-web

kubectl exec -n spire spire-server-0 -- /opt/spire/bin/spire-server entry create \
    -parentID spiffe://example.org/node \
    -spiffeID spiffe://example.org/ec-backend \
    -selector k8s:pod-label:app:ec-backend

kubectl exec -n spire spire-server-0 -- /opt/spire/bin/spire-server entry create \
    -parentID spiffe://example.org/node \
    -spiffeID spiffe://example.org/news-web \
    -selector k8s:pod-label:app:news-web

kubectl exec -n spire spire-server-0 -- /opt/spire/bin/spire-server entry create \
    -parentID spiffe://example.org/node \
    -spiffeID spiffe://example.org/news-backend \
    -selector k8s:pod-label:app:news-backend

Confirm the created registration entries.

kubectl exec -n spire spire-server-0 -- /opt/spire/bin/spire-server entry show

5. Deploy four services

Deploy four services. SPIRE Agent (Unix Domain Socket) mounts on Envoy container and run Envoy and OPA as sidecar.

kubectl apply -f ec-backend.yaml
kubectl apply -f ec-web.yaml
kubectl apply -f news-backend.yaml
kubectl apply -f news-web.yaml

6. Check policy

✅ Request to ec-web and news-web Envoy with custom header "X-Opa-Secret" via Ingress

Expect 200 OK response.

curl -i -H 'Host: ec-web.example.org' -H 'X-Opa-Secret: ec-web-secret' http://$(minikube ip)/noproxy
curl -i -H 'Host: news-web.example.org' -H 'X-Opa-Secret: news-web-secret' http://$(minikube ip)/noproxy

❌ Request to ec-web and news-web Envoy without custom header "X-Opa-Secret" via Ingress

Expect 403 Forbidden response.

curl -i -H 'Host: ec-web.example.org' http://$(minikube ip)/noproxy
curl -i -H 'Host: news-web.example.org' http://$(minikube ip)/noproxy

✅ Request to /data of (ec|news)-backend from (ec|news)-web

Expect 200 OK response.

curl -i -H 'Host: ec-web.example.org' -H 'X-Opa-Secret: ec-web-secret' http://$(minikube ip)/data
curl -i -H 'Host: news-web.example.org' -H 'X-Opa-Secret: news-web-secret' http://$(minikube ip)/data

❌ Request to /admin of (ec|news)-backend from (ec|news)-web

Expect 403 Forbidden response.

curl -i -H 'Host: ec-web.example.org' -H 'X-Opa-Secret: ec-web-secret' http://$(minikube ip)/admin
curl -i -H 'Host: news-web.example.org' -H 'X-Opa-Secret: news-web-secret' http://$(minikube ip)/admin

❌ Request to /data of ec-backend from news-web

Expect 403 Forbidden response.

curl -i -H 'Host: news-web.example.org' -H 'X-Opa-Secret: news-web-secret' http://$(minikube ip)/ec-backend-data

7. Cleaning Up

kubectl delete -f spire-server.yaml
kubectl delete -f spire-agent.yaml
kubectl delete -f ec-backend.yaml
kubectl delete -f ec-web.yaml
kubectl delete -f news-backend.yaml
kubectl delete -f news-web.yaml

References

• Envoy External Authorization with OPA

License

MIT