Description
Due to sticky workflow execution local activities are always retried on the same host. Local activity retry policy allows retries to be performed on the same host until decision timeout, and then it creates a timer decision to backoff on the server to force another decision for further retries. This is a problem if the activity is failing due to some issue on the specific host.
One option is to have a policy for local retries which allows for expiration for retries on the machine and then respond back with DecisionFailed to clear stickiness. This would allow workflow execution to be picked up on a different host and executing local activities on it.
We have to be careful to make sure clear of stickiness happens with a jitter to protect against all caches being cleared simultaneously.