|
| 1 | +# TemporalWorkerOwnedResource |
| 2 | + |
| 3 | +`TemporalWorkerOwnedResource` (TWOR) lets you attach arbitrary Kubernetes resources — HPAs, PodDisruptionBudgets, custom CRDs — to each active versioned Deployment managed by a `TemporalWorkerDeployment`. The controller creates one copy of the resource per active Build ID, automatically wired to the correct versioned Deployment. |
| 4 | + |
| 5 | +## Why you need this |
| 6 | + |
| 7 | +The Temporal Worker Controller creates one Kubernetes `Deployment` per worker version (Build ID). If you attach an HPA directly to a single Deployment, it breaks as versions roll over — the old HPA still targets the old Deployment, the new Deployment has no HPA, and you have to manage cleanup yourself. |
| 8 | + |
| 9 | +`TemporalWorkerOwnedResource` solves this by treating the attached resource as a template. The controller renders one instance per active Build ID, injects the correct versioned Deployment name, and cleans up automatically when a version is deleted (via Kubernetes owner reference garbage collection). |
| 10 | + |
| 11 | +## How it works |
| 12 | + |
| 13 | +1. You create a `TemporalWorkerOwnedResource` that references a `TemporalWorkerDeployment` and contains the resource spec in `spec.object`. |
| 14 | +2. The validating webhook checks that you have permission to manage that resource type yourself (SubjectAccessReview), and that the resource kind isn't on the banned list. |
| 15 | +3. On each reconcile loop, the controller renders one copy of `spec.object` per active Build ID, injects fields (see below), and applies it via Server-Side Apply. |
| 16 | +4. Each copy is owned by the corresponding versioned `Deployment`, so it is garbage-collected automatically when that Deployment is deleted. |
| 17 | +5. `TWOR.status.versions` is updated with the applied/failed status for each Build ID. |
| 18 | + |
| 19 | +## Auto-injection |
| 20 | + |
| 21 | +The controller auto-injects two fields when you set them to `null` in `spec.object`. Setting them to `null` is the explicit signal that you want injection — if you omit the field entirely, nothing is injected; if you set a non-null value, the webhook rejects the object. |
| 22 | + |
| 23 | +| Field | Injected value | |
| 24 | +|-------|---------------| |
| 25 | +| `spec.scaleTargetRef` (HPA) | `{apiVersion: apps/v1, kind: Deployment, name: <versioned-deployment-name>}` | |
| 26 | +| `spec.selector.matchLabels` (any) | `{temporal.io/build-id: <buildID>, temporal.io/deployment-name: <twdName>}` | |
| 27 | + |
| 28 | +## Resource naming |
| 29 | + |
| 30 | +Each per-Build-ID copy is named `<twdName>-<tworName>-<buildID>`, cleaned for DNS and truncated to 253 characters. Use `kubectl get hpa` (or whatever kind you attached) after a reconcile to see the created resources. |
| 31 | + |
| 32 | +## RBAC |
| 33 | + |
| 34 | +### What the webhook checks |
| 35 | + |
| 36 | +When you create or update a TWOR, the webhook performs SubjectAccessReviews to verify: |
| 37 | + |
| 38 | +1. **You** (the requesting user) can create/update the embedded resource type in that namespace. |
| 39 | +2. **The controller's service account** can create/update the embedded resource type in that namespace. |
| 40 | + |
| 41 | +If either check fails, the TWOR is rejected. This prevents privilege escalation — you cannot use TWOR to create resources you don't already have permission to create yourself. |
| 42 | + |
| 43 | +### What to configure in Helm |
| 44 | + |
| 45 | +`ownedResourceConfig.rbac.rules` controls what resource types the controller's ClusterRole permits it to manage. The defaults cover HPAs and PodDisruptionBudgets: |
| 46 | + |
| 47 | +```yaml |
| 48 | +ownedResourceConfig: |
| 49 | + rbac: |
| 50 | + rules: |
| 51 | + - apiGroups: ["autoscaling"] |
| 52 | + resources: ["horizontalpodautoscalers"] |
| 53 | + - apiGroups: ["policy"] |
| 54 | + resources: ["poddisruptionbudgets"] |
| 55 | +``` |
| 56 | +
|
| 57 | +Add entries for any other resource types you want to attach (e.g., KEDA `ScaledObjects`). For development clusters you can set `rbac.wildcard: true` to grant access to all resource types, but this is not recommended for production. |
| 58 | + |
| 59 | +### What to configure for your users |
| 60 | + |
| 61 | +Users who create TWORs also need RBAC permission to manage the embedded resource type directly. For example, to let a team create TWORs that embed HPAs, they need the standard `autoscaling` permissions in their namespace — there is nothing TWOR-specific to configure for this. |
| 62 | + |
| 63 | +## Webhook TLS |
| 64 | + |
| 65 | +The TWOR validating webhook requires TLS. The recommended approach is to install [cert-manager](https://cert-manager.io/docs/installation/) before deploying the controller — the Helm chart handles everything else automatically (`certmanager.enabled: true` is the default). |
| 66 | + |
| 67 | +If cert-manager is not available in your cluster, set `certmanager.enabled: false` and provide: |
| 68 | +1. A `kubernetes.io/tls` Secret named `webhook-server-cert` in the controller namespace, containing `tls.crt` and `tls.key` for the webhook server. The certificate must have DNS SANs: |
| 69 | + - `<release-name>-webhook-service.<namespace>.svc` |
| 70 | + - `<release-name>-webhook-service.<namespace>.svc.cluster.local` |
| 71 | +2. The base64-encoded CA certificate that signed `tls.crt`, passed as `certmanager.caBundle` in Helm values. |
| 72 | + |
| 73 | +```bash |
| 74 | +helm install temporal-worker-controller oci://docker.io/temporalio/temporal-worker-controller \ |
| 75 | + --namespace temporal-system \ |
| 76 | + --set certmanager.enabled=false \ |
| 77 | + --set certmanager.caBundle="$(base64 -w0 ca.crt)" |
| 78 | +``` |
| 79 | + |
| 80 | +## Example: HPA per Build ID |
| 81 | + |
| 82 | +```yaml |
| 83 | +apiVersion: temporal.io/v1alpha1 |
| 84 | +kind: TemporalWorkerOwnedResource |
| 85 | +metadata: |
| 86 | + name: my-worker-hpa |
| 87 | + namespace: my-namespace |
| 88 | +spec: |
| 89 | + # Reference the TemporalWorkerDeployment to attach to. |
| 90 | + workerRef: |
| 91 | + name: my-worker |
| 92 | +
|
| 93 | + # The resource template. The controller creates one copy per active Build ID. |
| 94 | + object: |
| 95 | + apiVersion: autoscaling/v2 |
| 96 | + kind: HorizontalPodAutoscaler |
| 97 | + spec: |
| 98 | + # null tells the controller to auto-inject the versioned Deployment reference. |
| 99 | + # Do not set this to a real value — the webhook will reject it. |
| 100 | + scaleTargetRef: null |
| 101 | + minReplicas: 2 |
| 102 | + maxReplicas: 10 |
| 103 | + metrics: |
| 104 | + - type: Resource |
| 105 | + resource: |
| 106 | + name: cpu |
| 107 | + target: |
| 108 | + type: Utilization |
| 109 | + averageUtilization: 70 |
| 110 | +``` |
| 111 | + |
| 112 | +See [examples/twor-hpa.yaml](../examples/twor-hpa.yaml) for an example pre-configured for the helloworld demo. |
| 113 | + |
| 114 | +## Example: PodDisruptionBudget per Build ID |
| 115 | + |
| 116 | +```yaml |
| 117 | +apiVersion: temporal.io/v1alpha1 |
| 118 | +kind: TemporalWorkerOwnedResource |
| 119 | +metadata: |
| 120 | + name: my-worker-pdb |
| 121 | + namespace: my-namespace |
| 122 | +spec: |
| 123 | + workerRef: |
| 124 | + name: my-worker |
| 125 | + object: |
| 126 | + apiVersion: policy/v1 |
| 127 | + kind: PodDisruptionBudget |
| 128 | + spec: |
| 129 | + minAvailable: 1 |
| 130 | + # null tells the controller to auto-inject {temporal.io/build-id, temporal.io/deployment-name}. |
| 131 | + selector: |
| 132 | + matchLabels: null |
| 133 | +``` |
| 134 | + |
| 135 | +## Checking status |
| 136 | + |
| 137 | +```bash |
| 138 | +# See all TWORs and which TWD they reference |
| 139 | +kubectl get twor -n my-namespace |
| 140 | +
|
| 141 | +# See per-Build-ID apply status |
| 142 | +kubectl get twor my-worker-hpa -n my-namespace -o jsonpath='{.status.versions}' | jq . |
| 143 | +
|
| 144 | +# See the created HPAs |
| 145 | +kubectl get hpa -n my-namespace |
| 146 | +``` |
0 commit comments