implement an example rate limiter

Signed-off-by: Hayim.Shaul@ibm.com <hayimsha@fhe03.vpc.cloud9.ibm.com>

Author: Hayim.Shaul@ibm.com

docs/README.md

@@ -16,7 +16,7 @@ Welcome to Panurus documentation.
 ## Security
 
 *   [**HTLC Deadlines and Clock Synchronisation**](security/htlc_deadline_clock_assumptions.md): The clock-synchronisation assumption that the HTLC claim/reclaim deadline rests on, and the deadline margin it requires of a deployment.
-*   [**Selector Resource Limits**](security/selector_resource_limits.md): How to throttle token selection by supplying a custom `Locker`.
+*   [**Selector Resource Limits**](security/selector_resource_limits.md): How to throttle token selection, either with the opt-in built-in per-wallet rate limiter or by supplying your own limiter or `Locker`.
 
 ## Command-Line Tools
 

docs/configuration.md

@@ -39,6 +39,24 @@ token:
     # fetcherCacheMaxQueries is the number of queries after which a soft refresh (non-blocking background update) is triggered.
     # This helps keep the cache fresh without blocking queries. If not specified or set to 0, defaults to 5 queries.
     fetcherCacheMaxQueries: 5
+    # Built-in per-wallet rate limiter for token selection (both drivers).
+    # It is disabled by default: without these keys, selection requests are not metered.
+    # One selection request (a Selector.Select call) costs one unit, no matter how many tokens it
+    # locks or how often it retries internally. Unlocking tokens is never throttled.
+    # A throttled request fails fast with an error wrapping token.SelectorRateLimited.
+    # See docs/security/selector_resource_limits.md.
+    # rateLimitEnabled turns the limiter on with the default rate and burst below.
+    rateLimitEnabled: true
+    # rateLimit is the maximum number of selection requests per second a single wallet may issue.
+    # A positive value implies rateLimitEnabled: true. If not specified or set to 0, defaults to 100.
+    rateLimit: 100
+    # rateLimitBurst is the maximum number of selection requests a single wallet may issue
+    # back-to-back. If not specified or set to 0, defaults to twice rateLimit.
+    rateLimitBurst: 200
+    # rateLimitMaxBuckets caps the number of per-wallet buckets kept in memory. When the cap is
+    # reached, idle buckets are pruned first and, if that is not enough, the least recently used
+    # ones are dropped. If not specified or set to 0, defaults to 4096.
+    rateLimitMaxBuckets: 4096
 
   # When we are interested in knowing when a transaction reaches finality, we subscribe to the Finality Listener Manager for the finality event of that transaction.
   # This configuration specifies the way the manager is instantiated (i.e., how it gets notified about the finality events, how often it checks).
@@ -471,6 +489,15 @@ Default values:
 - numRetries: 3
 - leaseExpiry: 3m
 - leaseCleanupTickPeriod: 90s
+- rateLimitEnabled: false (the built-in per-wallet selection rate limiter is opt-in)
+- rateLimit: 100 requests/s per wallet, when rate limiting is enabled
+- rateLimitBurst: 2 × rateLimit, so 200 requests, when rate limiting is enabled
+- rateLimitMaxBuckets: 4096
+
+Setting a positive `rateLimit` is enough to enable the limiter; `rateLimitEnabled: true` alone
+enables it with the defaults above. See
+[docs/security/selector_resource_limits.md](security/selector_resource_limits.md) for what is
+metered and how to plug in your own limiter instead.
 
 ---
 

docs/security/selector_resource_limits.md

@@ -4,39 +4,150 @@
 
 Token selection acquires a short-lived *lock* on each candidate token so that two
 concurrent transactions do not try to spend the same token. Under load, a single
-wallet can drive a large number of selection/lock requests. Applications that need to
-throttle this — to protect the lock store, to enforce fairness between wallets, or to
-integrate with an existing quota system — can do so by supplying their own `Locker`
-implementation.
+wallet can drive a large number of selection/lock requests. To protect the lock store,
+to enforce fairness between wallets, or to integrate with an existing quota system,
+Panurus offers two ways to throttle this:
+
+1. A **built-in per-wallet rate limiter**, activated purely from configuration or from
+   code. It is a token bucket per wallet, in process, and it is **disabled by default**.
+2. A **fail-fast contract**, `token.SelectorRateLimited`, plus a **wallet-id-aware lock
+   function**. Both selector drivers (simple and sherdlock) pass the wallet id the tokens
+   are being selected for into the `Locker`'s lock function, and abort the selection
+   immediately when a lock is denied with an error wrapping that sentinel. This is the
+   integration point for applications that would rather reuse the rate-limiting
+   infrastructure they already run (for example a Redis-backed limiter shared across
+   processes).
+
+## The built-in limiter
+
+Package `token/services/selector/ratelimit`. One **selection request** — one
+`Selector.Select` call — costs one unit from the bucket of the wallet it selects for,
+regardless of how many tokens the request ends up locking or how many times the selector
+retries internally on contention. Deliberately *not* per token lock attempt: charging
+there would make a large transfer cost more than a small one and would let the selector's
+own contention retries drain a wallet's allowance.
+
+Properties worth knowing:
+
+- **Per wallet, per TMS.** Buckets are keyed by the wallet id *and* the TMS id, so the
+  same wallet id in two networks or namespaces gets two independent allowances.
+- **Bounded memory.** Buckets are created on first use and pruned without any background
+  goroutine: idle ones are swept during ordinary access, and a hard cap
+  (`rateLimitMaxBuckets`) evicts the least recently used ones if the sweep is not enough.
+- **Unlocking is never throttled.** Only `Select` is metered; `Unlock` and `Close` pass
+  through, so a throttled wallet can always clean up after itself.
+- **Empty wallet ids are never throttled.**
+- **Nothing is leaked on a denial.** The request is rejected before the selector runs, so
+  no token is locked and there is nothing to release.
+
+### Enabling it from configuration
+
+Under `token.selector` (see [../configuration.md](../configuration.md)):
+
+```yaml
+token:
+  selector:
+    driver: sherdlock
+    # Enables the limiter with the defaults below.
+    rateLimitEnabled: true
+    # Selection requests per second per wallet. A positive value implies rateLimitEnabled: true.
+    # Defaults to 100.
+    rateLimit: 100
+    # Back-to-back requests allowed to one wallet. Defaults to 2 × rateLimit, so 200.
+    rateLimitBurst: 200
+    # Maximum number of per-wallet buckets held in memory. Defaults to 4096.
+    rateLimitMaxBuckets: 4096
+```
+
+Omitting all four keys, which is the default, leaves selection unmetered.
+
+### Enabling it from code
+
+Both selector services accept `ratelimit.Option` values, which take precedence over the
+configuration:
+
+```go
+import (
+    "github.com/LFDT-Panurus/panurus/token/services/selector/ratelimit"
+    "github.com/LFDT-Panurus/panurus/token/services/selector/sherdlock"
+)
+
+// The built-in limiter with its defaults (100 requests/s per wallet, burst 200).
+svc := sherdlock.NewService(fetcherProvider, lockStoreManager, configProvider, metricsProvider,
+    ratelimit.WithDefaultLimiter())
+
+// Explicit rate and burst.
+svc = sherdlock.NewService(fetcherProvider, lockStoreManager, configProvider, metricsProvider,
+    ratelimit.WithLimiter(ratelimit.New(ratelimit.Config{Rate: 20, Burst: 40})))
+
+// Explicitly off, whatever the configuration says.
+svc = sherdlock.NewService(fetcherProvider, lockStoreManager, configProvider, metricsProvider,
+    ratelimit.WithLimiter(nil))
+```
+
+`simple.NewService(lockerProvider, configProvider, opts ...ratelimit.Option)` takes the
+same options.
+
+A limiter passed with `WithLimiter` belongs to the caller: the service never stops it, not
+even from `Shutdown`. That matters because `Shutdown` also runs on routine public-parameter
+reloads, and resetting every wallet's bucket there would let a throttled client wash out
+its debt. `BucketLimiter.Stop()` exists for callers that own a limiter and want its memory
+back.
 
-Panurus deliberately ships **no built-in rate limiter or quota**. Instead it
-gives you two things:
+### Supplying your own limiter
 
-1. A **wallet-id-aware lock function**. Both selector drivers (simple and sherdlock)
-   pass the wallet id the tokens are being selected for into the `Locker`'s lock
-   function, so a custom `Locker` can apply per-wallet policies.
-2. A **fail-fast contract**, `token.SelectorRateLimited`. When a `Locker` denies a
-   lock by returning an error that wraps this sentinel, the selector aborts the
-   selection immediately and returns the error to the caller instead of retrying.
+`WithLimiter` accepts any implementation of:
 
-This keeps the Panurus minimal and lets applications reuse whatever rate-limiting
-infrastructure they already run (for example a Redis-backed limiter shared across
-processes).
+```go
+// Limiter meters token selection requests per wallet.
+type Limiter interface {
+    // Allow returns nil when a selection request for walletID within scope may proceed,
+    // and an error wrapping token.SelectorRateLimited when it must be denied.
+    // scope is the TMS id. An empty walletID is never throttled.
+    Allow(ctx context.Context, scope string, walletID string) error
+}
+```
+
+This is the simplest way to plug in a shared, cluster-wide limiter (Redis, a quota table,
+a sidecar) while keeping the metering point — one unit per selection request — and the
+fail-fast behaviour that Panurus already implements.
+
+## The fail-fast contract
+
+`token/selector.go` defines:
+
+```go
+// SelectorRateLimited is the contract error returned (directly or wrapped) to deny a
+// selection for policy reasons such as rate limiting or quota.
+var SelectorRateLimited = errors.New("selection rate limit exceeded")
+```
+
+When a `Locker` returns an error `e` with `errors.Is(e, token.SelectorRateLimited)`, the
+selector:
+
+- stops iterating candidate tokens,
+- releases any tokens it already locked for this request, and
+- returns `e` to the caller.
+
+Any *other* error from the lock function keeps the existing semantics: the token is
+treated as unavailable (e.g. already locked by another transaction) and selection
+continues / retries as before.
 
 ## The lock function
 
 Both selector drivers route through a `Locker` whose lock function receives the
-wallet id.
+wallet id, so a custom `Locker` can apply per-wallet policies of its own — per token
+lock rather than per selection request.
 
 **Simple selector** — `token/services/selector/simple/selector.go`:
 
 ```go
 type Locker interface {
-    // Lock locks the token id for the consumer transaction txID on behalf of walletID
+    // Lock locks the token id for the consumer transaction txID on behalf of owner
     // (ownerFilter.ID()). Return an error wrapping token.SelectorRateLimited to deny
     // the lock and make the selection fail fast.
-    Lock(ctx context.Context, id *token.ID, txID string, walletID string, reclaim bool) (string, error)
-    UnlockIDs(ctx context.Context, ids ...*token.ID) []*token.ID
+    Lock(ctx context.Context, owner string, id *token.ID, txID string, reclaim bool) (string, error)
+    UnlockIDs(ctx context.Context, owner string, ids ...*token.ID) []*token.ID
     UnlockByTxID(ctx context.Context, txID string)
     IsLocked(id *token.ID) bool
 }
@@ -60,34 +171,12 @@ type TokenLockStore interface {
 The built-in in-memory locker and the SQL-backed `TokenLockStore` accept `walletID`
 but do not act on it — they apply no rate limiting or quota.
 
-## The fail-fast contract
-
-`token/selector.go` defines:
-
-```go
-// SelectorRateLimited is the contract error a Locker implementation returns (directly
-// or wrapped) to deny a lock for policy reasons such as rate limiting or quota.
-var SelectorRateLimited = errors.New("selection rate limit exceeded")
-```
-
-When your `Locker` returns an error `e` with `errors.Is(e, token.SelectorRateLimited)`,
-the selector:
-
-- stops iterating candidate tokens,
-- releases any tokens it already locked for this request, and
-- returns `e` to the caller.
-
-Any *other* error from the lock function keeps the existing semantics: the token is
-treated as unavailable (e.g. already locked by another transaction) and selection
-continues / retries as before.
-
-## Integrating your own rate limiting
+### Integrating your own rate limiting in a Locker
 
 Provide a `Locker` that wraps the SDK's default locker and enforces your policy before
 delegating. Below, a Redis-backed limiter throttles per wallet; the same shape works
-for an in-process limiter, a quota table, etc.
-
-### Simple selector
+for an in-process limiter, a quota table, etc. Note that this is charged **per token lock
+attempt**, unlike the built-in limiter above.
 
 ```go
 import (
@@ -105,24 +194,21 @@ type rateLimitedLocker struct {
     limiter       RedisLimiter // your existing infrastructure
 }
 
-func (l *rateLimitedLocker) Lock(ctx context.Context, id *tokenapi.ID, txID string, walletID string, reclaim bool) (string, error) {
-    if !l.limiter.Allow(ctx, walletID) {
-        return "", errors.Wrapf(token.SelectorRateLimited, "wallet %s throttled", walletID)
+func (l *rateLimitedLocker) Lock(ctx context.Context, owner string, id *tokenapi.ID, txID string, reclaim bool) (string, error) {
+    if !l.limiter.Allow(ctx, owner) {
+        return "", errors.Wrapf(token.SelectorRateLimited, "wallet %s throttled", owner)
     }
 
-    return l.Locker.Lock(ctx, id, txID, walletID, reclaim)
+    return l.Locker.Lock(ctx, owner, id, txID, reclaim)
 }
 ```
 
 Wire it in by providing a `simple.LockerProvider` whose `New` returns your decorator
 instead of the default `inmemory.NewLocker`.
 
-### Sherdlock selector
-
-Provide a `TokenLockStore` (via the `tokenlockdb.StoreServiceManager` used by
-`sherdlock.NewService`) whose `Lock` enforces the limit before delegating to the
-SQL-backed store, returning an error wrapping `token.SelectorRateLimited` when a wallet
-is throttled.
+For the sherdlock selector, provide a `TokenLockStore` (via the
+`tokenlockdb.StoreServiceManager` used by `sherdlock.NewService`) whose `Lock` enforces the
+limit before delegating to the SQL-backed store.
 
 ## Handling the error
 
@@ -136,9 +222,14 @@ if errors.Is(err, token.SelectorRateLimited) {
 }
 ```
 
+The built-in limiter's error message states how long to wait before the wallet has a
+request available again.
+
 ## Notes
 
 - Passing an empty `walletID` is valid; a `Locker` that keys its policy on wallet id
-  should treat empty as "no throttling" (the default lockers ignore it entirely).
-- Because the policy lives in your `Locker`, its scope (per process vs shared across a
-  cluster), persistence, and lifecycle are entirely under your control.
+  should treat empty as "no throttling" (the default lockers ignore it entirely, and so
+  does the built-in limiter).
+- The built-in limiter is per process. If a wallet's traffic is spread over several nodes,
+  each node enforces its own allowance; supply a shared `Limiter` if you need a
+  cluster-wide budget.

token/selector.go

@@ -25,13 +25,15 @@ var (
 	// SelectorSufficientFundsButConcurrencyIssue is returned when funds are sufficient to cover the request, but
 	// concurrency issues does not make some of the selected tokens available.
 	SelectorSufficientFundsButConcurrencyIssue = errors.New("sufficient funds but concurrency issue")
-	// SelectorRateLimited is the contract error a Locker implementation returns (directly
-	// or wrapped) to deny a lock for policy reasons such as rate limiting or quota.
+	// SelectorRateLimited is the contract error returned (directly or wrapped) to deny a
+	// selection for policy reasons such as rate limiting or quota.
 	// Both the simple and sherdlock selectors detect it via errors.Is and abort the
 	// selection immediately, returning the error to the caller instead of retrying.
-	// Panurus ships no built-in limiter: applications integrate their own
-	// (e.g. a Redis-backed limiter) by providing a Locker implementation that returns
-	// this error when a request must be throttled.
+	// Panurus ships an opt-in per-wallet limiter that returns it, see
+	// token/services/selector/ratelimit and the token.selector.rateLimit* configuration
+	// keys; it is disabled by default. Applications that would rather reuse their own
+	// infrastructure (e.g. a Redis-backed limiter) can either supply a Limiter to that
+	// package or return this error from a custom Locker implementation.
 	SelectorRateLimited = errors.New("selection rate limit exceeded")
 )