fix: clean up Temporal server-side versioning data on TWD deletion (#240)

- Add a finalizer to `TemporalWorkerDeployment` to run Temporal
server-side cleanup before K8s deletion
- Add a finalizer to `TemporalConnection` to prevent it from being
deleted while any TWD still references it
- On TWD deletion, set current version to unversioned, clear ramping
version, and delete registered versions

## Problem

When a `TemporalWorkerDeployment` CRD is deleted (e.g., switching back
to plain Deployments), the Temporal server retains the build ID routing
configuration. The matching service continues routing new tasks to the
deleted build ID's physical queue, while unversioned workers poll a
different physical queue. Tasks sit in `Scheduled` state indefinitely
with no errors.

A secondary race condition exists: Helm deletes both the
`TemporalConnection` and `TWD` in the same upgrade. Without the
connection, the controller cannot talk to Temporal to clean up. This is
solved by adding a finalizer to the `TemporalConnection` that blocks its
deletion until all referencing TWDs are gone.

## Changes

**`internal/controller/worker_controller.go`:**

**TWD finalizer (`temporal.io/worker-deployment-cleanup`):**
- Added to all TWD resources during normal reconciliation
- On deletion, triggers `handleDeletion()` which:
1. Sets the current version to unversioned (`BuildID: ""`) -- the
critical step that unblocks task dispatch
  2. Clears any ramping version
  3. Deletes all registered versions with `SkipDrainage: true`
  4. Attempts to delete the deployment record itself
  5. Removes the connection finalizer if no other TWDs reference it
  6. Removes its own finalizer, allowing K8s to complete deletion

**TemporalConnection finalizer (`temporal.io/connection-in-use`):**
- Added to the `TemporalConnection` during normal TWD reconciliation via
`ensureConnectionFinalizer()`
- Prevents the connection from being deleted while any TWD still
references it
- Removed by `removeConnectionFinalizerIfUnused()` during TWD deletion,
after checking no other TWDs in the same namespace reference the
connection
- Guarantees the connection is always available during TWD cleanup -- no
race condition with Helm deleting both resources simultaneously

**RBAC updates:**
- Added `update;patch` verbs for `temporalconnections` (was
`get;list;watch`)
- Added `update` verb for `temporalconnections/finalizers`

## Deletion flow

```
Helm upgrade (TWD disabled)
  |
  v
Helm deletes TWD CRD + TemporalConnection CRD simultaneously
  |
  +--> TemporalConnection: has finalizer, K8s sets DeletionTimestamp but blocks deletion
  |
  +--> TWD: has finalizer, K8s sets DeletionTimestamp, triggers Reconcile
         |
         v
       handleDeletion() runs:
         1. Fetches TemporalConnection (guaranteed to exist via finalizer)
         2. Connects to Temporal server
         3. Sets current version to unversioned
         4. Deletes versions
         5. Removes connection finalizer (no other TWDs reference it)
         6. Removes TWD finalizer
         |
         v
       TWD deleted by K8s
         |
         v
       TemporalConnection: no more finalizers, deleted by K8s
```

Issue #55 
Closes #166

---------

Signed-off-by: Anuj Agrawal <anujagrawal380@gmail.com>

Author: anujagrawal380

helm/temporal-worker-controller-crds/templates/temporal.io_temporalworkerdeployments.yaml

@@ -3753,6 +3753,10 @@ spec:
                                             type: integer
                                           signerName:
                                             type: string
+                                          userAnnotations:
+                                            additionalProperties:
+                                              type: string
+                                            type: object
                                         required:
                                         - keyType
                                         - signerName
@@ -3947,6 +3951,18 @@ spec:
                         x-kubernetes-list-map-keys:
                         - name
                         x-kubernetes-list-type: map
+                      workloadRef:
+                        properties:
+                          name:
+                            type: string
+                          podGroup:
+                            type: string
+                          podGroupReplicaKey:
+                            type: string
+                        required:
+                        - name
+                        - podGroup
+                        type: object
                     required:
                     - containers
                     type: object

helm/temporal-worker-controller/templates/rbac.yaml

@@ -107,17 +107,8 @@ rules:
       - temporal.io
     resources:
       - temporalconnections
+      - workerresourcetemplates
     verbs:
-      - get
-      - list
-      - watch
-  - apiGroups:
-      - temporal.io
-    resources:
-      - temporalworkerdeployments
-    verbs:
-      - create
-      - delete
       - get
       - list
       - patch
@@ -126,28 +117,31 @@ rules:
   - apiGroups:
       - temporal.io
     resources:
+      - temporalconnections/finalizers
       - temporalworkerdeployments/finalizers
     verbs:
       - update
   - apiGroups:
       - temporal.io
     resources:
-      - temporalworkerdeployments/status
-      - workerresourcetemplates/status
+      - temporalworkerdeployments
     verbs:
+      - create
+      - delete
       - get
+      - list
       - patch
       - update
+      - watch
   - apiGroups:
       - temporal.io
     resources:
-      - workerresourcetemplates
+      - temporalworkerdeployments/status
+      - workerresourcetemplates/status
     verbs:
       - get
-      - list
       - patch
       - update
-      - watch
   # GENERATED RULES END
   # Rules for managing resources attached via WorkerResourceTemplate.
   # Driven entirely by workerResourceTemplate.allowedResources in values.yaml.