Fix task queue rate-limit for invalid tasks

[carlydf]

What changed?

Implement RecycleToken in MultiRateLimiter and add functional tests.

Why?

ClockedRateLimiter is the base rate limiter for many rate limiters in our system, so I made the mistake of assuming that implementing RecycleToken there and and then calling ClockedRateLimiter.RecycleToken from other wrapper rate limiters (such as MultiRateLimiter and DynamicRateLimiter) would "just work."

However, a limitation of the recycle token implementation is that another process must be waiting on a token (via ClockedRateLimiter.Wait) at the time when RecycleToken is called.

DynamicRateLimiter.Wait calls ClockedRateLimiter.Wait, which means that DynamicRateLimiter.RecycleToken can simply call ClockedRateLimiter.RecycleToken.

However, MultiRateLimiter.Wait does not call the Wait method of its sub-rate-limiters; instead it implements a custom Wait method using the Reserve methods of its sub-rate-limiters. This means the sub-rate-limiters never call Wait and never wait for the recycled token to arrive. Giving MultiRateLimiter a custom RecycleToken implementation that pairs with the custom MultiRateLimiter.Wait solves this problem.

How did you test it?

Tested in a test server using sample workflows, and with functional tests for old and new matcher.
Note that New Matcher did not have this bug, because it does not use MultiRateLimiter for task queue rate limiting.

Potential risks

Documentation

Is hotfix candidate?

It fixes a bug, but the bug only occurs when many workflows that have backlogged tasks become invalid (ie. via termination or deletion).