Limit Redis memory usage of system RAM

[kixorz]

Limit Redis memory usage to 25% of system RAM and enable eviction policy to prevent OOM.

Summary

This PR configures Redis in our Docker Compose deployment to use a bounded amount of memory (25% of the container’s available RAM) and enables an eviction policy. This reduces the risk of Redis consuming all available memory, triggering out-of-memory (OOM) conditions, and forcing costly instance size increases in cloud environments.

What changed

• Updated the Redis service command to compute a maxmemory value at container start and pass it to redis-server.
• Set the eviction policy to volatile-lru so Redis evicts the least recently used keys that have an expiration set when under memory pressure.

Why this is needed

• Unlimited Redis memory leads to unbounded growth: Redis holds data in memory; without a cap, it can grow until the host/container runs out of memory.
• OOM crashes and instability: When Redis (or the container) exhausts memory, the kernel may OOM-kill processes, causing downtime and data loss (in-memory datasets) or cascading failures in dependent services.
• Cloud cost pitfalls: The usual band-aid for OOM is to permanently bump instance sizes (more RAM). That’s expensive and scales poorly as workload grows. Setting a sane cap plus an eviction policy keeps memory predictable and avoids unnecessary instance class upgrades.

How it works

• On container startup, we read total memory from /proc/meminfo and compute 25% for --maxmemory.
• We set --maxmemory-policy volatile-lru to evict the least recently used keys among those with TTLs when approaching the memory cap.

Impact

• Predictable memory footprint for Redis within the container.
• Reduced risk of host-level OOM and improved overall stability.
• Some non-expiring keys won’t be evicted by volatile-lru. If the dataset is dominated by non-expiring keys, you may want a different policy (see below).

Configuration and overrides

• Default behavior: 25% of container RAM, eviction policy volatile-lru.
• To change the allocation fraction: edit the compose command expression (e.g., use /3 or /2 instead of /4).
• To change eviction policy: replace --maxmemory-policy volatile-lru with one of Redis’s supported policies (e.g., allkeys-lru, volatile-ttl, allkeys-random, noeviction, etc.).
• If you prefer an explicit fixed cap, replace the computed value with a static byte value, for example: --maxmemory 2gb.

Risks and trade-offs

• Evictions under memory pressure: If Redis reaches the cap, keys will be evicted per the chosen policy. Applications relying on retained cache entries should tolerate misses.

Legal Boilerplate

Look, I get it. The entity doing business as "Sentry" was incorporated in the State of Delaware in 2015 as Functional Software, Inc. and is gonna need some rights from me in order to utilize my contributions in this here PR. So here's the deal: I retain all rights, title and interest in and to my contributions, and by keeping this boilerplate intact I confirm that Sentry can use, modify, copy, and redistribute my contributions, under Sentry's choice of terms.

[aminvakil]

Applications relying on retained cache entries should tolerate misses.

Are we sure this is true for self-hosted?

Also MemTotal / 4 in our minimum requirement setup which is 16GB RAM + 16GB Swap would be 4GB. So we have to be sure redis does not use more than that.

I checked two self-hosted instances and one of them is using 20MB and the other is using 40MB :)

[aminvakil]

@kixorz Which version is self-hosted are you on? I remember we had a memory leak problem with redis about two or three years ago, but it got fine after a couple of releases, unfortunately I cannot remember the exact version.

And I was wondering why your redis instance got OOMed?

[kixorz]

Hey, thanks for the question. We ran out of disk space, Sentry stopped working. When we rebooted, redis allocated all available memory on the system and the machine hung. We had to double the RAM to make it work again. After it started to work, we modified the RAM back. Redis exhausted the memory again when it was trying to start back up. This is because its disk database is already as big as the highest RAM value from the previous start.

[aminvakil]

I do not think redis usage is relevant to total memory of server, so there is no need for it to be 1 / 4 of total memory.

On another note some users may have changed their redis.conf locally and this change breaks them.

It'll be nice if we have a estimation of how redis uses memory and set a maxmemory suitable for that usage, for example if self-hosted redis is using 2GB memory at max for example, we can just set maxmemory to 4GB.

[aminvakil]

Also copying this from original PR which created redis.conf:

The default value of the maxmemory setting in Redis depends on the system architecture:

64-bit Systems: By default, there is no limit on memory usage. This allows Redis to utilize as much RAM as the operating system permits until it runs out of available memory.
32-bit Systems: The implicit memory limit is typically set to 3GB, which is a consequence of the limitations inherent in 32-bit addressing.

I believe we can set the default value to unlimited, allowing users to adjust this setting as needed based on their specific requirements.

Originally posted by @Hassanzadeh-sd in #3427 (comment)

[kixorz]

The problem is that with maxmemory 0 there is no cap on redis memory use. There are situations where redis will use all available memory - for example after the system runs out of disk space. This hangs the system.

If the solution is to modify the conf file, feel free to close this.
I think having the two params in the compose file is more elegant.

[aminvakil]

The problem is that with maxmemory 0 there is no cap on redis memory use. There are situations where redis will use all available memory - for example after the system runs out of disk space. This hangs the system.

It's fine to set an appropriate limit on redis maxmemory if we know how it uses over time.

If the solution is to modify the conf file, feel free to close this. I think having the two params in the compose file is more elegant.

We can change this PR to set the limit in redis.conf if that's OK with you. Let's see what maintainers think about this before you put work onto it though.

[BYK]

👎🏻 on this. We should just let people change the redis config.

@kixorz if you feel strongly about the env variable you can also use docker.compose.override.yml file to override the file per your changes and use an env variable in your own installation.

[aminvakil]

👎🏻 on this. We should just let people change the redis config.

@kixorz if you feel strongly about the env variable you can also use docker.compose.override.yml file to override the file per your changes and use an env variable in your own installation.

@BYK How do you feel about a default redis maxmemory instead of 0?

[BYK]

@BYK How do you feel about a default redis maxmemory instead of 0?

Don't have enough ops expertise to make an intelligent comment about this 😅

[kixorz]

Thanks for the feedback. I updated the improvement to limit redis memory to 2GB using the config file.

Please let me know if you have other suggestions.

[aminvakil]

We need to make sure this suffices self-hosted usage for different types of usage.

I tend not to merge this unless we're 100% sure, as it would cost more than it benefits.

Also PR title needs to be changed prior to merging it.

[williamdes]

Thanks for the feedback. I updated the improvement to limit redis memory to 2GB using the config file.

Please let me know if you have other suggestions.

I like this change, else we could also limit the docker container max memory
I have

    mem_reservation: 512M
    mem_limit: 8G

[aminvakil]

We need to make sure this suffices self-hosted usage for different types of usage.

Quoting myself on this, unless we know the maximum redis memory usage of self-hosted in different scenarios, you can change it locally for yourself.

[kixorz]

0 (unlimited) is not the right default. It leads to problems and system crashes.

[kixorz]

@williamdes I updated the docker compose file with the suggested limits.

Please let me know any other feedback.

[kixorz]

I updated the change based on your feedback. Redis is configured to use up to 2GB, the container limit is 2.5GB.

[aminvakil]

I checked two self-hosted instances and one of them is using 20MB and the other is using 40MB :)

Correcting myself, I just saw this in another self-hosted instance:

used_memory_human:22.68G

So I guess this differs from instance to instance?

[aminvakil]

Please remove this, we do not have this for other services as well.

[aminvakil]

This is absolutely crucial for maxmemory-policy to kick in.

https://redis.io/docs/latest/develop/reference/eviction/ :

Use the maxmemory-policy configuration directive to select the eviction policy you want to use when the limit set by maxmemory is reached.

[aldy505]

I would argue that 2gb is not enough. I would prefer 8gb of 12gb.

[kixorz]

8 or 12g requires a large and more expensive instance.

We have a relatively busy instance and 2g works fine for us.

[aldy505]

Redis is just a cache, the system may only be slightly slower in certain situations.

Surprisingly that's not how Sentry uses Redis.

Memcached is being used just for caching. But Redis, it's being used as (1) lock contention store, (2) storing uptime monitor configuration, (3) storing project configs that is used for validating incoming SDK envelopes, and much more that I can't remember. These are not really "caches" for me though.

Although, I understand that this shouldn't how you use Redis, but it's a live software anyway, we can only work around it.

[kixorz]

Are those use cases subject to cache eviction?

[aminvakil]

So I guess this differs from instance to instance?

Yep, I've set two instances with 20MB and 40MB instances redis maxmemory to 4gb manually, and the other one with 22gb redis usage, did not have a maxmemory set.

[jjbayer]

Redis is used for a lot of different use cases across sentry. I am sure that some of those will break when keys disappear sooner than expected. If any of those use cases causes infinite memory growth then that is a bug that we should investigate independently. @kixorz do you have any clue what type of redis key (e.g. what prefix) causes the memory pressure in your case?

[kixorz]

Redis is used for a lot of different use cases across sentry. I am sure that some of those will break when keys disappear sooner than expected. If any of those use cases causes infinite memory growth then that is a bug that we should investigate independently. @kixorz do you have any clue what type of redis key (e.g. what prefix) causes the memory pressure in your case?

Just to repeat - The issue is that uncapped redis will use all available system memory. It becomes a problem when it's restarted and attempts to load the save file, which will cause a crash loop.

I'd say the limit is the first step to a solution.

How can I find out what prefix is causing it?

[jjbayer]

The issue is that uncapped redis will use all available system memory. It becomes a problem when it's restarted and attempts to load the save file, which will cause a crash loop.

I'd say the limit is the first step to a solution.

Limiting redis memory usage for your own sentry instance makes sense, but setting a default now breaks every instance that already uses > 2 GB today.

How can I find out what prefix is causing it?

Not sure, I would start by running something like redis-cli --scan | cut -d: -f1 | sort | uniq -c | sort -r to find the dominant prefixes. That does not necessarily tell you that they use a lot of memory but it might be a place to start.