You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Address code review feedback on docs and resource naming
- Truncate owned resource names to 47 chars (safe for Deployments; avoids
per-resource-type special cases if Deployment is ever un-banned)
- Fix docs: replace "active Build ID" with "worker version with running workers"
throughout; "active" is reserved for Ramping/Current versions
- Fix docs: owned-resource deletion is due to versioned Deployment sunset, not
a separate "version delete" operation
- Fix docs: scaleTargetRef injection applies to any resource type with that field,
not just HPA; clarify webhook rejects non-null values because controller owns them
- Fix docs: remove undocumented/untested BYO TLS path; cert-manager is required
- Fix docs: expand TWOR abbreviation to full name throughout; remove ⏳ autoscaling
bullet from README and clarify TWOR is the path for metric/backlog-based autoscaling
- Add note on how to inspect the banned kinds list (BANNED_KINDS env var)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Copy file name to clipboardExpand all lines: README.md
+2-3Lines changed: 2 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -5,7 +5,7 @@
5
5
6
6
> 🚀 **Public Preview**: This project is in [Public Preview](https://docs.temporal.io/evaluate/development-production-features/release-stages) and ready for production use cases*. Core functionality is complete with stable APIs.
7
7
>
8
-
> *Dynamic auto-scaling based on workflow load is not yet implemented. Use cases must work with fixed worker replica counts.
8
+
> *Autoscaling based on Temporal task queue depth is not yet built in. You can attach Horizontal Pod Autoscalers or KEDA ScaledObjects to each versioned Deployment via [`TemporalWorkerOwnedResource`](docs/owned-resources.md).
9
9
10
10
**The Temporal Worker Controller makes it simple and safe to deploy Temporal workers on Kubernetes.**
- ✅ **Deletion of resources** associated with drained Worker Deployment Versions
106
106
- ✅ **Multiple rollout strategies**: `Manual`, `AllAtOnce`, and `Progressive` rollouts
107
107
- ✅ **Gate workflows** - Test new versions with a [pre-deployment test](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning#adding-a-pre-deployment-test) before routing real traffic to them
108
-
- ✅ **Per-version attached resources** - Attach HPAs, PodDisruptionBudgets, or any namespaced Kubernetes resource to each active worker version via [`TemporalWorkerOwnedResource`](docs/owned-resources.md)
109
-
- ⏳ **Temporal-aware auto-scaling** - Scaling based on workflow task queue depth is not yet implemented
108
+
- ✅ **Per-version attached resources** - Attach HPAs, KEDA ScaledObjects, PodDisruptionBudgets, or any namespaced Kubernetes resource to each worker version with running workers via [`TemporalWorkerOwnedResource`](docs/owned-resources.md) — this is also the recommended path for metric-based and backlog-based autoscaling
Copy file name to clipboardExpand all lines: docs/owned-resources.md
+43-31Lines changed: 43 additions & 31 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,44 +1,67 @@
1
1
# TemporalWorkerOwnedResource
2
2
3
-
`TemporalWorkerOwnedResource`(TWOR) lets you attach arbitrary Kubernetes resources — HPAs, PodDisruptionBudgets, custom CRDs — to each active versioned Deployment managed by a `TemporalWorkerDeployment`. The controller creates one copy of the resource per active Build ID, automatically wired to the correct versioned Deployment.
3
+
`TemporalWorkerOwnedResource` lets you attach arbitrary Kubernetes resources — HPAs, PodDisruptionBudgets, KEDA ScaledObjects, custom CRDs — to each worker version that has running workers. The controller creates one copy of the resource per worker version with a running Deployment, automatically wired to the correct versioned Deployment.
4
4
5
5
## Why you need this
6
6
7
7
The Temporal Worker Controller creates one Kubernetes `Deployment` per worker version (Build ID). If you attach an HPA directly to a single Deployment, it breaks as versions roll over — the old HPA still targets the old Deployment, the new Deployment has no HPA, and you have to manage cleanup yourself.
8
8
9
-
`TemporalWorkerOwnedResource` solves this by treating the attached resource as a template. The controller renders one instance per active Build ID, injects the correct versioned Deployment name, and cleans up automatically when a version is deleted (via Kubernetes owner reference garbage collection).
9
+
`TemporalWorkerOwnedResource` solves this by treating the attached resource as a template. The controller renders one instance per worker version with running workers, injects the correct versioned Deployment name, and cleans up automatically when the versioned Deployment is deleted (e.g., during the sunset process after traffic has drained).
10
+
11
+
This is also the recommended mechanism for metric-based or backlog-based autoscaling: attach a KEDA `ScaledObject` (or a standard HPA with custom metrics) to your workers and the controller keeps one per running worker version, each pointing at the right Deployment.
10
12
11
13
## How it works
12
14
13
15
1. You create a `TemporalWorkerOwnedResource` that references a `TemporalWorkerDeployment` and contains the resource spec in `spec.object`.
14
16
2. The validating webhook checks that you have permission to manage that resource type yourself (SubjectAccessReview), and that the resource kind isn't on the banned list.
15
-
3. On each reconcile loop, the controller renders one copy of `spec.object` per active Build ID, injects fields (see below), and applies it via Server-Side Apply.
17
+
3. On each reconcile loop, the controller renders one copy of `spec.object` per worker version with a running Deployment, injects fields (see below), and applies it via Server-Side Apply.
16
18
4. Each copy is owned by the corresponding versioned `Deployment`, so it is garbage-collected automatically when that Deployment is deleted.
17
-
5.`TWOR.status.versions` is updated with the applied/failed status for each Build ID.
19
+
5.`TemporalWorkerOwnedResource.status.versions` is updated with the applied/failed status for each Build ID.
18
20
19
21
## Auto-injection
20
22
21
-
The controller auto-injects two fields when you set them to `null` in `spec.object`. Setting them to `null` is the explicit signal that you want injection — if you omit the field entirely, nothing is injected; if you set a non-null value, the webhook rejects the object.
23
+
The controller auto-injects two fields when you set them to `null` in `spec.object`. Setting them to `null` is the explicit signal that you want injection:
24
+
- If you omit the field entirely, nothing is injected.
25
+
- If you set a non-null value, the webhook rejects the object because the controller owns these fields.
|`spec.scaleTargetRef` (any resource with this field) |`{apiVersion: apps/v1, kind: Deployment, name: <versioned-deployment-name>}`|
30
+
|`spec.selector.matchLabels` (any resource with this field) |`{temporal.io/build-id: <buildID>, temporal.io/deployment-name: <twdName>}`|
31
+
32
+
The `scaleTargetRef` injection applies to any resource type that has a `scaleTargetRef` field — not just HPAs. KEDA `ScaledObjects` and other autoscaler CRDs use the same field and benefit from the same injection.
27
33
28
34
## Resource naming
29
35
30
-
Each per-Build-ID copy is named `<twdName>-<tworName>-<buildID>`, cleaned for DNS and truncated to 253 characters. Use `kubectl get hpa` (or whatever kind you attached) after a reconcile to see the created resources.
36
+
Each per-Build-ID copy is given a unique, DNS-safe name derived from the `(twdName, tworName, buildID)` triple. Names are capped at 47 characters to be safe for all Kubernetes resource types, including Deployment (which has pod-naming constraints that effectively limit deployment names to ~47 characters). The name always ends with an 8-character hash of the full triple, so uniqueness is guaranteed even when the human-readable prefix is truncated.
37
+
38
+
Use `kubectl get <kind>` after a reconcile to see the created resources and their names.
39
+
40
+
## Banned resource kinds
41
+
42
+
Certain resource kinds are blocked by default to prevent misuse (e.g., using `TemporalWorkerOwnedResource` to spin up arbitrary workloads). The default banned list is:
43
+
44
+
```
45
+
Deployment, StatefulSet, Job, Pod, CronJob
46
+
```
47
+
48
+
The banned list is configured via `ownedResourceConfig.bannedKinds` in Helm values and is visible as the `BANNED_KINDS` environment variable on the controller pod:
49
+
50
+
```bash
51
+
kubectl get pod -n <controller-namespace> -l app.kubernetes.io/name=temporal-worker-controller \
When you create or update a TWOR, the webhook performs SubjectAccessReviews to verify:
59
+
When you create or update a `TemporalWorkerOwnedResource`, the webhook performs SubjectAccessReviews to verify:
37
60
38
61
1.**You** (the requesting user) can create/update the embedded resource type in that namespace.
39
62
2.**The controller's service account** can create/update the embedded resource type in that namespace.
40
63
41
-
If either check fails, the TWOR is rejected. This prevents privilege escalation — you cannot use TWOR to create resources you don't already have permission to create yourself.
64
+
If either check fails, the request is rejected. This prevents privilege escalation — you cannot use `TemporalWorkerOwnedResource` to create resources you don't already have permission to create yourself.
42
65
43
66
### What to configure in Helm
44
67
@@ -58,26 +81,13 @@ Add entries for any other resource types you want to attach (e.g., KEDA `ScaledO
58
81
59
82
### What to configure for your users
60
83
61
-
Users who create TWORs also need RBAC permission to manage the embedded resource type directly. For example, to let a team create TWORs that embed HPAs, they need the standard `autoscaling` permissions in their namespace — there is nothing TWOR-specific to configure for this.
84
+
Users who create `TemporalWorkerOwnedResources` also need RBAC permission to manage the embedded resource type directly. For example, to let a team create `TemporalWorkerOwnedResources` that embed HPAs, they need the standard `autoscaling` permissions in their namespace — there is nothing `TemporalWorkerOwnedResource`-specific to configure for this.
62
85
63
86
## Webhook TLS
64
87
65
-
The TWOR validating webhook requires TLS. The recommended approach is to install [cert-manager](https://cert-manager.io/docs/installation/) before deploying the controller — the Helm chart handles everything else automatically (`certmanager.enabled: true` is the default).
66
-
67
-
If cert-manager is not available in your cluster, set `certmanager.enabled: false` and provide:
68
-
1. A `kubernetes.io/tls` Secret named `webhook-server-cert` in the controller namespace, containing `tls.crt` and `tls.key` for the webhook server. The certificate must have DNS SANs:
The `TemporalWorkerOwnedResource` validating webhook requires TLS. Install [cert-manager](https://cert-manager.io/docs/installation/) before deploying the controller — the Helm chart handles everything else automatically (`certmanager.enabled: true` is the default).
79
89
80
-
## Example: HPA per Build ID
90
+
## Example: HPA per worker version
81
91
82
92
```yaml
83
93
apiVersion: temporal.io/v1alpha1
@@ -90,7 +100,8 @@ spec:
90
100
workerRef:
91
101
name: my-worker
92
102
93
-
# The resource template. The controller creates one copy per active Build ID.
103
+
# The resource template. The controller creates one copy per worker version
104
+
# with a running Deployment.
94
105
object:
95
106
apiVersion: autoscaling/v2
96
107
kind: HorizontalPodAutoscaler
@@ -111,7 +122,7 @@ spec:
111
122
112
123
See [examples/twor-hpa.yaml](../examples/twor-hpa.yaml) for an example pre-configured for the helloworld demo.
113
124
114
-
## Example: PodDisruptionBudget per Build ID
125
+
## Example: PodDisruptionBudget per worker version
115
126
116
127
```yaml
117
128
apiVersion: temporal.io/v1alpha1
@@ -135,11 +146,12 @@ spec:
135
146
## Checking status
136
147
137
148
```bash
138
-
# See all TWORs and which TWD they reference
139
-
kubectl get twor -n my-namespace
149
+
# See all TemporalWorkerOwnedResources and which TWD they reference
150
+
kubectl get temporalworkerownedresource -n my-namespace
`TemporalWorkerOwnedResource`(TWOR) lets you attach Kubernetes resources — HPAs, PodDisruptionBudgets, etc. — to each active versioned Deployment. The controller creates one copy per active Build ID and wires it to the correct Deployment automatically.
126
+
`TemporalWorkerOwnedResource` lets you attach Kubernetes resources — HPAs, PodDisruptionBudgets, etc. — to each worker version with running workers. The controller creates one copy per worker version with a running Deployment and wires it to the correct Deployment automatically.
127
127
128
-
The TWOR validating webhook enforces that you have permission to create the embedded resource type yourself, and it requires TLS (provided by cert-manager, installed in step 3 above).
128
+
The `TemporalWorkerOwnedResource` validating webhook enforces that you have permission to create the embedded resource type yourself, and it requires TLS (provided by cert-manager, installed in step 3 above).
129
129
130
130
After deploying the helloworld worker (step 5), apply the example HPA:
131
131
132
132
```bash
133
133
kubectl apply -f examples/twor-hpa.yaml
134
134
```
135
135
136
-
Watch the controller create an HPA for each active Build ID:
136
+
Watch the controller create an HPA for each worker version with running workers:
137
137
138
138
```bash
139
-
# See TWOR status (Applied: true once the controller reconciles)
140
-
kubectl get twor
139
+
# See TemporalWorkerOwnedResource status (Applied: true once the controller reconciles)
140
+
kubectl get temporalworkerownedresource
141
141
142
142
# See the per-Build-ID HPAs
143
143
kubectl get hpa
144
144
```
145
145
146
-
You should see one HPA per active worker version, with `scaleTargetRef` automatically pointing at the correct versioned Deployment.
146
+
You should see one HPA per worker version with running workers, with `scaleTargetRef` automatically pointing at the correct versioned Deployment.
147
147
148
-
When you deploy a new worker version (e.g., step 8), the controller creates a new HPA for the new Build ID and keeps the old one until that version is deleted.
148
+
When you deploy a new worker version (e.g., step 8), the controller creates a new HPA for the new Build ID and keeps the old one until that versioned Deployment is deleted during the sunset process.
149
149
150
150
See [docs/owned-resources.md](../../docs/owned-resources.md) for full documentation.
0 commit comments