Feature request: Support for annotation-driven object mutation

[MacroPower]

Thank you for your work on this, and for making it a component decoupled from kluctl proper, I've found it quite useful on several occasions.

I've also recently been using Kyverno quite a bit. Kyverno is super powerful and most of the time I enjoy using it, and for cluster-wide mutations it's perfect for me. But, when I find myself wanting to make small server-side changes to one or a couple objects, I always use template-controller, mainly because it's so much simpler to define an ObjectTemplate when compared to writing a Policy, and also the ObjectTemplates are much easier to read (in the same way a helm template is frequently easier to read than a JSON patch).

But, many times I find myself forced to use Kyverno when I would have rather just written an ObjectTemplate, because I don't have the ability to wrap whatever object was being created in the ObjectTemplate CR. For example, when the object is created with Helm or by another controller/operator.

This led me to an interesting idea that I wanted to get your thoughts on. It might be completely insane / way out of scope but I think it should be possible.

Here's one of the current ObjectTemplate examples:

apiVersion: templates.kluctl.io/v1alpha1
kind: ObjectTemplate
metadata:
  name: postgres-secret-transformer
  namespace: default
spec:
  serviceAccountName: postgres-secret-transformer
  prune: true
  matrix:
    - name: secret
      object:
        ref:
          apiVersion: v1
          kind: Secret
          name: zalando.acid-minimal-cluster.credentials.postgresql.acid.zalan.do
  templates:
  - object:
      apiVersion: v1
      kind: Secret
      metadata:
        name: "transformed-postgres-secret"
      stringData:
        jdbc_url: "jdbc:postgresql://acid-minimal-cluster/zalando?user={{ matrix.secret.data.username | b64decode }}&password={{ matrix.secret.data.password | b64decode }}"
        # sometimes the key names inside a secret are not what another component requires, so we can simply use different names if we want
        username_with_different_key: "{{ matrix.secret.data.username | b64decode }}"
        password_with_different_key: "{{ matrix.secret.data.password | b64decode }}"

But, I think an alternative way of doing this same thing, would be something like this:

apiVersion: v1
kind: Secret
metadata:
  name: "transformed-postgres-secret"
  labels:
    kluctl.io/template: "true"
  annotations:
    kluctl.io/service-account: postgres-secret-transformer
    object.matrix.kluctl.io/secret: |
      ref:
        apiVersion: v1
        kind: Secret
        name: zalando.acid-minimal-cluster.credentials.postgresql.acid.zalan.do
stringData:
  jdbc_url: "jdbc:postgresql://acid-minimal-cluster/zalando?user={{ matrix.secret.data.username | b64decode }}&password={{ matrix.secret.data.password | b64decode }}"
  # sometimes the key names inside a secret are not what another component requires, so we can simply use different names if we want
  username_with_different_key: "{{ matrix.secret.data.username | b64decode }}"
  password_with_different_key: "{{ matrix.secret.data.password | b64decode }}"

In this case, template-controller would have a Mutating Webhook matching objects with kluctl.io/template="true".

This seems like it would, in theory, remove the need for wrapping objects with ObjectTemplate.

Even if you think this has no place as part of template-controller, I would love to get your thoughts. Thank you!

[codablock]

@MacroPower Thanks for the feature suggestion and it at first sounds like a good thing to have. There are however multiple challenges which lead me to abandon other mutating webhook based ideas in the past already.

1. The main challenge is that MWHs are nothing one can rely on. They might be missing on the cluster for several reasons, and deploying the templated object at that time would be a serious issue because whoever is deploying will not get what is expected.
2. Objects might be picked up by other controllers or even edited manually by the administrator. In that case, the already mutated object is picked up, modified and then sent back to the apiserver. The MWH could be implemented in a way that it would notice this and ignore the re-apply, but it feels like this could be error prone and I'm also not sure how it would mess with conflict resolution.
3. RBAC. Allowing the matrix vars to access other objects is quite sensitive so we'd need to come up with a proper RBAC integration for objects that were not intended to be integrated in that way.

I assume that all of these can be solved, but the effort would be quite high. Not sure if I ever find time for this unless I stumble across a situation by myself where I desperately need it :) But if someone wants to work on this, go for it, as long as you address the mentioned challenges.