-
-
Notifications
You must be signed in to change notification settings - Fork 47
Open
Labels
Description
When set_state is called during a Redis outage, the state is written to the in-memory failover store. The impact depends on the caller:
Admin panel: Operation silently has no effect
Admin's FailSafe instance writes to admin's local Memory store. Application instances never see it - they read from Redis (or their own isolated Memory stores):
- Admin calls
light.lock(RED)during Redis outage set_state(LOCKED_RED)-> written to Admin's Memory- Application instances read from Redis -> UNLOCKED
- Admin sees success, but lock had zero effect
Automated process (same application instance): State lost or leaked
When automation runs in-process (deployment hooks, health checks, feature flags), it shares the FailSafe instance with request handling:
State lost after recovery:
- Automation calls
lock(RED)during Redis outage -> Memory - Redis recovers
- Requests read from Redis -> UNLOCKED
Stale state resurfaces on future outages:
- Automation locks during outage -> Memory
- Redis recovers, automation unlocks -> Redis
- Future Redis outage
- Requests read from Memory -> stale
LOCKED_RED