fix(recovery): guard claim fan-out send on context cancellation to unblock Stop

[AkramBitar]

Fixes #2038

The recovery manager fanned claims out to its worker pool with an unguarded work <- claim send. On shutdown the workers return from their own select on ctx.Done() without draining work, so that send blocked forever: close(work) was never reached, the recovery loop's deferred wg.Done() never ran, and Stop()'s wg.Wait() hung while holding m.mu — wedging every later Start()/Stop().

Changes

• Extract the fan-out into Manager.fanOut and guard the send on ctx.Done(), mirroring services/cleanup/manager.go.
• Keep close(work) / workerWG.Wait() on the cancellation path so no worker goroutine leaks, and surface the cancellation as errors.Join(fanOutErr, firstErr) instead of swallowing it.
• Undispatched claims stay Pending, so the next sweep re-claims them once their lease expires — no work is lost.
• docs/services/storage/recovery.md: new "Shutdown Behaviour" section documenting the above.

Test

TestManager_StopDuringFanOutDoesNotDeadlock reproduces the issue's numbered failure scenario step by step: 64 claims to a single worker parked inside Recover (fan-out blocked on the unbuffered send), Stop() from a goroutine, worker released only after cancellation so both its select arms are ready, then a guard on Stop() returning and a final Start()/Stop() pair proving the manager is not wedged.

Verified both directions: it fails against the unguarded send (Stop() deadlocked while the sweep was fanning out claims) and passes with the fix under -count=3 -race. gofmt, go vet, and golangci-lint are clean.

[github-actions]

📊 Token Validation Benchmark

Comparison of this PR against the base branch. 🟢 improvement · 🔴 regression · ➖ within ±1.0% noise.

Variant	Benchmark	Params	Workers	TPS (base → PR)	Δ TPS
csp	BenchmarkAPIGRPC	f=1, nc=4, w=token-validation-service	4	147 → 148	➖ +0.7%
csp	BenchmarkLocalTokenValidation	out-tokens=2in-tokens=2	4	115 → 115	➖ +0.1%
ipa	BenchmarkAPIGRPC	f=1, nc=4, w=token-validation-service	4	78 → 78	➖ +0.0%
ipa	BenchmarkLocalTokenValidation	out-tokens=2in-tokens=2	4	68 → 69	➖ +0.4%

[github-actions]

📊 Token Validation Benchmark

Comparison of this PR against the base branch. 🟢 improvement · 🔴 regression · ➖ within ±1.0% noise.

Variant	Benchmark	Params	Workers	TPS (base → PR)	Δ TPS
csp	BenchmarkAPIGRPC	f=1, nc=4, w=token-validation-service	4	118 → 118	➖ -0.1%
csp	BenchmarkLocalTokenValidation	out-tokens=2in-tokens=2	4	104 → 104	➖ -0.1%
ipa	BenchmarkAPIGRPC	f=1, nc=4, w=token-validation-service	4	78 → 78	➖ -0.3%
ipa	BenchmarkLocalTokenValidation	out-tokens=2in-tokens=2	4	89 → 89	➖ +0.1%

[atharrva01]

hi @AkramBitar , took a careful read since I have been working in this file recently. The deadlock traces through exactly as you describe: worker returns from its own select on ctx.Done() without draining work, work is unbuffered and fanOut is the only sender, so the send parks with no receiver and close(work) is never reached. Guarding the send looks right to me, and keeping close(work) and workerWG.Wait() on the cancellation path seems like the important detail.

Ran the test at -count=3 -race and it passes. I liked that it drives the failure through the real Start/Stop path rather than reaching for the unexported fan-out.

Three observations below, none of them blocking, and some possibly things you already weighed up.

[Effi-S]

LGTM

[AkramBitar]

@atharrva01

Thanks a lot for the review. I fixed all the comments. Hope now it is ok.

Regards,
Akram