Q: Does the operator perform state management?

[mkulke]

The upstream kbs deployment recipes lack explicit state management and store resources like secrets, policies on a local container instance. There are APIs that allow altering of that pod-local state, which leads to unpredictable results. If the operator would provide facilities and defaults for explicit state (StatefulSet with Volume?) that would be good additional value, and we could point users to the operator as a default means for deploying KBS.

[bpradipt]

Currently the operator takes most of the config via configmaps. Would that suffice ?

[mkulke]

I was more thinking about making the kbs a StatefulSet deployment and volume provisioning. But if the operator would provide options to mount secrets and policies via k8s read-only secrets/configmaps that would be also a good-enough solution IMO.

[bpradipt]

I was more thinking about making the kbs a StatefulSet deployment and volume provisioning. But if the operator would provide options to mount secrets and policies via k8s read-only secrets/configmaps that would be also a good-enough solution IMO.

We can aim to keep the core KBS as stateless and provide all config and secrets via K8s configMap and Secrets respectively. I think this will be easier to maintain, help with scaling of KBS/AS as needed and integrate with 3rd party vault storage.

[mkulke]

We can aim to keep the core KBS as stateless and provide all config and secrets via K8s configMap and Secrets respectively. I think this will be easier to maintain, help with scaling of KBS/AS as needed and integrate with 3rd party vault storage.

in general that makes sense, I think @kartikjoshi21 is trialing out using configmaps for policies, maybe we can implement it in the scope of this repo.

it would make sense to have a config option "policies/resources.readonly" on kbs so that we disable the respective endpoints and give sensible error messages.

with regard to full statelessness: the kbs protocol is stateful, so a kbs pod has state in memory

solutions that come to mind:

• sticky sessions (potentially litte/no changes to kbs required, but maybe elaborate k8s machinery, not sure)
• implement a redis (or similar) backend for sessions (should be trivial, but we defer the problem of state to either a managed redis or a vanilla k8s stateful redis, the latter is probaby pretty mature, so that's still a win over everything we build ourselves)

[bpradipt]

We can aim to keep the core KBS as stateless and provide all config and secrets via K8s configMap and Secrets respectively. I think this will be easier to maintain, help with scaling of KBS/AS as needed and integrate with 3rd party vault storage.

in general that makes sense, I think @kartikjoshi21 is trialing out using configmaps for policies, maybe we can implement it in the scope of this repo.

Sounds good

it would make sense to have a config option "policies/resources.readonly" on kbs so that we disable the respective endpoints and give sensible error messages.

Can you please point me to the kbs endpoints that will require disabling

with regard to full statelessness: the kbs protocol is stateful, so a kbs pod has state in memory

solutions that come to mind:

• sticky sessions (potentially litte/no changes to kbs required, but maybe elaborate k8s machinery, not sure)

I was thinking about sticky sessions/session affinity when having multiple kbs pod replicas running. Example - https://kubernetes.io/docs/reference/networking/virtual-ips/#session-affinity
https://kubernetes.github.io/ingress-nginx/examples/affinity/cookie/

[mkulke]

Can you please point me to the kbs endpoints that will require disabling

I think it's those endpoints: https://github.com/confidential-containers/trustee/blob/main/kbs/src/api/src/http/config.rs

I was thinking about sticky sessions/session affinity when having multiple kbs pod replicas running. Example - https://kubernetes.io/docs/reference/networking/virtual-ips/#session-affinity https://kubernetes.github.io/ingress-nginx/examples/affinity/cookie/

those would be the stateful RCAR protocol's endpoints: https://github.com/confidential-containers/trustee/blob/main/kbs/src/api/src/http/attest.rs

we might have to make to set some headers to allow sticky sessions, not sure.

[bpradipt]

@mkulke @kartikjoshi21 opa policy via configmaps is now supported with the operator - #33

I'm experimenting with session stickiness using ingress capabilities and will update soon. Here is what I have till now

• Expose ingress route (enable session stickiness as part of the config)
• Increase the replica count of the kbs deployment to 2
• Open two console to look at the logs of the two kbs deployments
• Use curl from two different pods and send request to the ingress

Verify from the logs how the requests are handled by kbs - Typically it'll be handled in a round-robin fashion (by default).

• Add a cookie to the curl (header) request

Verify from the logs how the requests are handled - the requests should only be served by a specific kbs deployment

I'll repeat this with some different configs just to be sure and based on that we can decide how to implement it on the client side.

[lmilleri]

Re sticky sessions, I run some more experiments with ingress & vanilla k8s. Here are the steps:

• Create kind cluster
• Install and configure trustee-operator (with trustee deployment replicas=2)
• Expose kbs-service (by default a ClusterIp service) with ingress route (with session affinity/cookies enabled)
• Open two console to look at the logs of the two trustee pods
• Use kbs-client for sending secret-retrieval requests

I can see the requests are served by both trustee instances in round robin fashion.
(of course auth/attest/get-resource sequence is sent to the same trustee)

Here an example of ingress configuration:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example-ingress
  namespace: trustee-operator-system
  annotations:
    nginx.ingress.kubernetes.io/affinity: "cookie"
    nginx.ingress.kubernetes.io/session-cookie-name: "route"
    nginx.ingress.kubernetes.io/session-cookie-expires: "172800"
    nginx.ingress.kubernetes.io/session-cookie-max-age: "172800"
spec:
  rules:
  - http:
      paths:
      - pathType: Prefix
        path: /
        backend:
          service:
            name: kbs-service
            port:
              number: 8080

I got more or less the same results downstream with OpenShift routes.

[lmilleri]

I put together some instructions at https://gist.github.com/lmilleri/2b99d039b66348e0742ea70f9d860db5

[bpradipt]

Thanks @lmilleri. Would it make sense to add some pointers in the doc mentioning about the annotations when using vanilla K8s ?

[lmilleri]

Please review #80