fix(events-processor): scope redis ctx per event - #785
Merged
Conversation
## Context The flag and cache stores captured the process-wide cancelable context at construction time. That context is canceled on SIGTERM, so every Redis write still in flight during a rolling restart failed with `context canceled`, even though batch processing itself deliberately runs on a fresh context so that in-flight events survive shutdown. ## Description Redis writes now receive the context of the record being processed instead of reading one off the store, so a shutdown no longer cancels work that the consumer is still allowed to finish. The context field is removed from both stores and the `Flagger` and `Cacher` interfaces carry it as an argument. Connection setup keeps using the process context: aborting a startup ping along with the process is the correct behavior.
toommz
approved these changes
Aug 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
FlagStoreandCacheStorestored the context they were built with, and that context is theprocess-wide one created in
mainand canceled on SIGTERM. Batch processing, on the other hand,deliberately runs on a fresh
context.Background()so a rolling restart lets in-flight recordsfinish instead of dropping them. The two never agreed: the batch kept going while every Redis
write inside it failed with
context canceled.Every restart therefore produced a burst of failures, one per record still being processed. The
events are safe, since the failure is retryable and the offset is not committed, but the failing
flag aborts the rest of
processEvent, so charge usage cache keys are never expired for thatbatch and stay warm until the next event on the same subscription.
Description
The stores no longer hold a context;
Flagger.FlagandCacher.ExpireKeytake one, and theprocessor passes the context of the record it is working on. Suppressing
context.Canceledatcapture time was the cheaper option and was rejected: it hides the log line while still skipping
cache expiration and still bouncing the batch back through Kafka.
Connection setup still uses the process context on purpose — a startup ping should die with the
process. Existing store and service tests only gained a context argument; no expectation changed.