Skip to content

Commit a33e1d0

Browse files
gabrielshanahanGabriel Shanahanclaude
authored
fix: Resilient Redis cache reads for rate limiting (#3455)
## Summary - **Exclude actuator endpoints from rate limit filters** — health probes (`/actuator/health`) no longer go through `GlobalIpRateLimitFilter`/`GlobalUserRateLimitFilter`, preventing crash loops when Redis cache entries are corrupted - **Add `ResilientCacheAccessor` component** — catches `RedisException` on direct `cache.get()` calls (not covered by `TolgeeCacheErrorHandler` which only handles `@Cacheable`), evicts bad entries, returns `null` (cache miss) - **Integrate into `RateLimitService`** — corrupted Bucket entries are treated as cache misses instead of 500 errors ### Root cause (v3.154.0 crash loop) `Bucket` class gained `strikeCount`/`lastStrikeAt` fields → old Kryo-serialized entries in Redis couldn't deserialize → `KryoBufferUnderflowException` → health checks hit rate limit filter → HTTP 500 → K8s restarts pod → crash loop. ### Defense in depth | Layer | What it does | Protects against | |-------|-------------|-----------------| | Actuator bypass | Skips rate limiting for `/actuator/**` | Health probe failures from any cache issue | | ResilientCacheAccessor | Catches `RedisException`, evicts, returns null | 500 errors on API requests from corrupted cache | ## Test plan - [x] `ResilientCacheAccessorTest` — 5 tests (normal get, null return, RedisException caught + eviction, non-Redis propagation) - [x] `RateLimitServiceTest` — 15 tests including new `corrupted cache entry is treated as cache miss` - [ ] Full backend test suite (`server-app:runStandardTests`) - [ ] Manual: start with Redis, corrupt a `rateLimits` entry, verify graceful handling 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Improved cache resilience: corrupted cache entries are evicted and treated as misses to prevent runtime errors. * Rate limiting now skips actuator endpoints so health and metrics remain accessible (including when an application context path is set). * **Tests** * Added tests validating cache resilience, eviction on corruption, null-handling, and exception propagation. * Expanded rate limiting tests to verify actuator endpoint exemptions and context-path handling. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Gabriel Shanahan <gabe@tolgee.io> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
1 parent 320d87a commit a33e1d0

17 files changed

Lines changed: 274 additions & 511 deletions

File tree

backend/data/src/main/kotlin/io/tolgee/component/CacheCleaner.kt

Lines changed: 0 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -4,24 +4,15 @@ import io.tolgee.configuration.tolgee.TolgeeProperties
44
import org.springframework.boot.context.event.ApplicationReadyEvent
55
import org.springframework.cache.CacheManager
66
import org.springframework.context.event.EventListener
7-
import org.springframework.core.annotation.Order
87
import org.springframework.stereotype.Component
98

10-
/**
11-
* Clears all caches on startup when configured via [TolgeeProperties.cache.cleanOnStartup].
12-
*
13-
* This runs with @Order(100) to ensure [CacheSchemaCleaner] runs first (@Order(50)).
14-
* The schema cleaner handles revision-based cache clearing, while this component
15-
* provides a way to force-clear all caches if needed.
16-
*/
179
@Component
1810
class CacheCleaner(
1911
private val allCachesProvider: AllCachesProvider,
2012
private val cacheManager: CacheManager,
2113
private val tolgeeProperties: TolgeeProperties,
2214
) {
2315
@EventListener
24-
@Order(100)
2516
fun onAppStartup(event: ApplicationReadyEvent) {
2617
if (tolgeeProperties.cache.cleanOnStartup) {
2718
cleanCaches()
Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
/**
2+
* Copyright (C) 2023 Tolgee s.r.o. and contributors
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
17+
package io.tolgee.component
18+
19+
import org.redisson.client.RedisException
20+
import org.slf4j.LoggerFactory
21+
import org.springframework.cache.Cache
22+
import org.springframework.stereotype.Component
23+
24+
/**
25+
* Provides resilient cache access for direct cache.get() calls.
26+
*
27+
* TolgeeCacheErrorHandler only covers @Cacheable annotations (Spring AOP).
28+
* Code that calls cache.get() directly (e.g. RateLimitService) needs this
29+
* component to handle deserialization errors gracefully.
30+
*
31+
* On RedisException (e.g. KryoBufferUnderflowException from schema changes),
32+
* logs a warning, evicts the bad entry, and returns null (cache miss).
33+
*/
34+
@Component
35+
class ResilientCacheAccessor {
36+
private val logger = LoggerFactory.getLogger(ResilientCacheAccessor::class.java)
37+
38+
fun <T> get(
39+
cache: Cache,
40+
key: Any,
41+
type: Class<T>,
42+
): T? {
43+
return try {
44+
cache.get(key, type)
45+
} catch (e: RedisException) {
46+
logger.warn(
47+
"Suppressing RedisException for cache {} on key {}. " +
48+
"This is likely due to outdated cache data, therefore this cache entry has been removed. " +
49+
"If this re-occurs for the same cache and key, it is likely from a bug that should be reported.",
50+
cache.name,
51+
key,
52+
)
53+
logger.warn("The following error occurred while fetching the key", e)
54+
cache.evictIfPresent(key)
55+
null
56+
}
57+
}
58+
}

backend/data/src/main/kotlin/io/tolgee/component/cache/CacheSchemaCleaner.kt

Lines changed: 0 additions & 124 deletions
This file was deleted.

backend/data/src/main/kotlin/io/tolgee/component/cache/CacheSchemaRegistry.kt

Lines changed: 0 additions & 84 deletions
This file was deleted.

backend/data/src/main/kotlin/io/tolgee/component/cache/LocalSchemaRevisionStore.kt

Lines changed: 0 additions & 59 deletions
This file was deleted.

0 commit comments

Comments
 (0)