Skip to content

Latest commit

 

History

History
53 lines (36 loc) · 2.28 KB

File metadata and controls

53 lines (36 loc) · 2.28 KB

Manager Identity and Ownership Transfer in the Worker Controller

Problem

If the worker controller is managing a Worker Deployment (i.e. updating its routing config), but a user makes a manual change via the CLI, SDK, or gRPC API instead of via the TemporalWorkerDeployment CRD interface, the controller should not clobber the user's change.

Once the user has finished their manual intervention, they need a way to hand ownership back to the controller.

Solution

The controller uses the Temporal server's ManagerIdentity field on Worker Deployments to coordinate exclusive ownership of routing changes.

When ManagerIdentity is set on a Worker Deployment, only clients whose identity matches ManagerIdentity can make routing changes (set current version, set ramping version). The controller's identity is visible in the managerIdentity field of the TemporalWorkerDeployment status.

How the controller claims ownership

The first time the controller plans a routing change for a Worker Deployment (i.e. when ManagerIdentity is empty), it calls SetManagerIdentity to claim ownership before applying the change. Subsequent routing changes succeed because the controller's identity already matches ManagerIdentity.

Taking manual control

To take manual control away from the controller, set ManagerIdentity to your own identity:

temporal worker deployment manager-identity set \
   --deployment-name $MY_DEPLOYMENT \
   --self

The --self flag sets ManagerIdentity to the identity of the caller (auto-generated by the CLI if not explicitly provided via --identity; similarly, the SDK uses its own auto-generated or configured identity). After this, the controller's routing change attempts will fail and it will retry on a backoff until ownership is returned.

You can then make routing changes freely (the server enforces ManagerIdentity for all clients, not just the controller).

Returning ownership to the controller

When you are done with your manual changes and want the controller to resume, clear ManagerIdentity:

temporal worker deployment manager-identity unset \
   --deployment-name $MY_DEPLOYMENT

On the next reconcile, the controller will detect that ManagerIdentity is empty, claim it for itself, and resume managing routing changes.