Bug Description
For users with large libraries tracking hundreds of active TV shows and Anime items, loading the /calendar view takes 90+ seconds, resulting in 504 Gateway Timeout or 499 Client Closed Request from the reverse proxy.
Profiling Event.objects.get_user_events revealed two major bottlenecks scaling poorly with library size:
1. TV Drop Filter SQL Generation:
In events/models.py -> _build_tv_query(), iterating over dropped_seasons to build exclude_query with |= Q(...) generates thousands of OR clauses in Python.
Fix: Replaced the Python loop with a correlated Exists subquery. This drops the query time from ~160ms to ~15ms and entirely avoids compiling a massive AST:
from django.db.models import Exists
# Subquery to check if there's any dropped season with a season_number
# less than or equal to the current event's season_number
dropped_season_exists = Exists(
Season.objects.filter(
user=user,
item__media_id=OuterRef("item__media_id"),
status__in=INACTIVE_TRACKING_STATUSES,
item__season_number__lte=OuterRef("item__season_number")
)
)
return (
Q(
item__media_type=MediaTypes.SEASON.value,
item__media_id__in=active_tv_shows,
)
& ~Q(dropped_season_exists)
)2. Anime Mapping Redis Deserialization Loop:
In events/models.py -> _cross_bucket_hidden_anime_item_ids(), the loop iterates over all active anime items and calls resolve_provider_series_id.
Under the hood, find_entries_for_mal_id() calls load_mapping_snapshot(). While the snapshot itself is O(1) indexed, load_mapping_snapshot() retrieves the massive payload from cache.get(cache_key) (Redis) on every single loop iteration. For 500 anime items, this triggers 1000 cache deserializations of a multi-megabyte JSON dictionary, taking ~94 seconds in Python overhead.
Fix: Memoized load_mapping_snapshot() in-memory as a module-level variable (_IN_MEMORY_SNAPSHOT = None) in integrations/anime_mapping.py so the worker only deserializes from Redis once per snapshot revision. This dropped the deduplication loop from 94 seconds to 0.4 seconds.
These optimizations combined dropped calendar load time from ~96s to ~0.5s for large libraries!
Bug Description
For users with large libraries tracking hundreds of active TV shows and Anime items, loading the
/calendarview takes 90+ seconds, resulting in504 Gateway Timeoutor499 Client Closed Requestfrom the reverse proxy.Profiling
Event.objects.get_user_eventsrevealed two major bottlenecks scaling poorly with library size:1. TV Drop Filter SQL Generation:
In
events/models.py->_build_tv_query(), iterating overdropped_seasonsto buildexclude_querywith|= Q(...)generates thousands ofORclauses in Python.Fix: Replaced the Python loop with a correlated
Existssubquery. This drops the query time from ~160ms to ~15ms and entirely avoids compiling a massive AST:2. Anime Mapping Redis Deserialization Loop:
In
events/models.py->_cross_bucket_hidden_anime_item_ids(), the loop iterates over all active anime items and callsresolve_provider_series_id.Under the hood,
find_entries_for_mal_id()callsload_mapping_snapshot(). While the snapshot itself is O(1) indexed,load_mapping_snapshot()retrieves the massive payload fromcache.get(cache_key)(Redis) on every single loop iteration. For 500 anime items, this triggers 1000 cache deserializations of a multi-megabyte JSON dictionary, taking ~94 seconds in Python overhead.Fix: Memoized
load_mapping_snapshot()in-memory as a module-level variable (_IN_MEMORY_SNAPSHOT = None) inintegrations/anime_mapping.pyso the worker only deserializes from Redis once per snapshot revision. This dropped the deduplication loop from 94 seconds to 0.4 seconds.These optimizations combined dropped calendar load time from ~96s to ~0.5s for large libraries!