On caching strategy & backing store

[bdach]

This issue is going to touch on a few disparate topics which to me are interlinked enough to warrant being included in one issue.

Usage of redis

The caching primitive the service is using is IDistributedCache backed by redis:

osu-server-replay-store/osu.Server.ReplayStore/Program.cs

Lines 21 to 24 in f461cdf

	builder.Services.AddStackExchangeRedisCache(options =>
	{
	options.Configuration = AppSettings.RedisHost;
	});

While we do have a production redis instance, there are significant reservations about using it for the purpose of this service. @peppy and @ThePooN can probably elaborate on that topic, but the gist of it from my understanding is that redis is not very reliable and has in fact had several instances of falling over, primarily due to running out of storage.

I understand @peppy has some ideas for alternatives to redis but I'll leave it to him to elaborate on that.

The cache period

Every replay that the service is caching, is cached for a full day:

osu-server-replay-store/osu.Server.ReplayStore/ReplayStoreController.cs

Lines 59 to 62 in f461cdf

	new DistributedCacheEntryOptions
	{
	AbsoluteExpirationRelativeToNow = TimeSpan.FromDays(1),
	});

which seems to be way over the top IMO but I guess it makes some degree of sense given that consumers of the “firehose” score API want to be using this. Will require cache hit rate monitoring. Maybe even a good idea to extract the cache duration to an envvar or something other that can be adjusted without having to apply source changes.

Back-of-napkin estimate of storage required for a day of replays for lazer alone to be in memory would be (if I’m not misreading ddog metrics):

266 800 replays/24hr * 50 KB/replay (semi-educated guess) to GB is 13,34 GB

so my bets are that 1 day is going to be too much, especially if this is to encompass stable as well (extrapolating from user numbers, you could be looking at about 5x as much storage).

Replays only enter cache via the upload operation

The download operation will never put anything in cache itself, it will only query it. If the cache is missed, any replays fetched from S3 will not be stored into the cache, not even for a short time. The only thing that puts a replay in the cache is uploading the replay.

I could see this making sense because I don’t think very many people download replays, but maybe someone with cf analytics access is able to disprove this.

[tsunyoku]

We should be able to swap Redis for anything else pretty easily, so I'll wait for comments on what I should be using instead and then I can proceed.

This should be encompassing stable as well, yes.

I agree about the duration probably being too long, I'll extract it to an envvar and put it down to 12 hours just as a baseline. With metrics, it can probably be adjusted more later down the line.

The download operation not putting anything into the cache was very much intended from initial discussions surrounding this service as it was my goal to reduce overhead for replays on recent scores (as they're more likely to be requested frequently during any time period). If we want to extend that scope we can, but I'd have concerns about increasing the size of the cache considerably for a gain that I'm not sure would justify it.

[peppy]

I'd hope for 24 hours storage to be feasible. We can size up the droplet running this to easily account for that, IMO.

The limitation, if we stick with redis will be RAM. It's a good reason to consider not using redis and instead just using a disk store of some kind. If we want to use redis for the cache / invalidation / LRU handling, that's fine – just store the replays outside of redis and the problem goes away. Handle deletion from disk whenever items are dropped from the cache (i'm not sure if redis has an flow for monitoring this, so that may be a consideration).

[ThePooN]

Redis is a memory-only database. Scaling this with lazer+stable and accounting for headroom is going to be expensive, and we don't currently have an option with our current deployment, nor experience in scaling redis without downtime.

Our best bet instead is disk-backed storage for cache. My concern if we do this is that this service would not be horizontally-scalable and become a single point-of-failure.
osu-server-spectator would need some retry mechanism with enough headroom to store transient replays in case of downtime.
README doesn't specify which services is intended to consume with, but consider downtime-handling there as well.

[bdach]

osu-server-spectator would need some retry mechanism with enough headroom to store transient replays in case of downtime.

See #2 for semi-related discussion.

README doesn't specify which services is intended to consume with, but consider downtime-handling there as well.

osu-server-spectator and osu-web-10 probably.

[ThePooN]

If we're looking for this to be highly-available (as #2 suggests), a somewhat far-fetched alternative (maybe not?) is to just build a small S3 cluster, probably backed by minio, and use it as the cache storage.

[peppy]

The hope was that this was going to live on a single droplet and be "high availability" from not being touched, like other components we've been running for years without issue (singular redis instances, rank lookup cache, etc).

[ThePooN]

I have my concerns but this is a valid solution if we deem downtime acceptable during upgrades.

[stanriders]

Our best bet instead is disk-backed storage for cache. My concern if we do this is that this service would not be horizontally-scalable and become a single point-of-failure.

I think having a separate mounted volume for the files would allow for proper scaling? Not necessarily a full minio instance, but just a normal storage volume which should be cheap to both host and mount to the caching service pods. There's no real reason to bring in additional complexity of hosting an S3-compatible storage since replays are completely immutable, and having multiple instances access the same cache storage shouldn't cause any concurrency issues. The only issue I can see is if you'd want to CDN the cache for some reason, but I think this service is not built for that to begin with.

[ThePooN]

DigitalOcean doesn't offer hosted file systems, only block volumes. They cannot be attached to more than one server at a time.

[peppy]

block storage should work fine for this purpose.