Skip to content

Optimization: Avoid N+1 queries in UserEventQuerySet.decay #3316

@lakshita10341

Description

@lakshita10341

Description

The decay method in api_app/user_events_manager/queryset.py performs an N+1 query operation by iterating over a queryset and calling .save() inside the loop. This can be inefficient as the number of user events increases.

Current Implementation

The current implementation iterates over every user event that needs decay, performs arithmetic calculations in Python, and saves each object individually:

# api_app/user_events_manager/queryset.py

    def decay(self):
        # ...
        objects = (
            self.exclude(decay_progression=DecayProgressionEnum.FIXED.value)
            .exclude(next_decay__isnull=True)
            .filter(
                next_decay__lte=now(),
            )
        )
        # TODO we can probably translate all of this in sql query
        for obj in objects:
            obj: UserEvent
            obj.decay_times += 1
            obj.data_model.reliability -= 1
            if obj.data_model.reliability == 0:
                obj.next_decay = None
            else:
                if obj.decay_progression == DecayProgressionEnum.LINEAR.value:
                    obj.next_decay += datetime.timedelta(days=obj.decay_timedelta_days)
                elif obj.decay_progression == DecayProgressionEnum.INVERSE_EXPONENTIAL.value:
                    obj.next_decay += datetime.timedelta(
                        days=obj.decay_timedelta_days ** (obj.decay_times + 1)
                    )
            obj.data_model.save()
            obj.save()
        return objects.count()

Proposed Solution

Refactor this method to use Django's update() method combined with F() expressions and Case/When statements to perform the update in a single database query.

This will involve:

  1. Using F('decay_times') + 1 for incrementing decay_times.
  2. Using Case/When to handle the conditional logic for next_decay calculation based on decay_progression (LINEAR vs INVERSE_EXPONENTIAL) and reliability == 0.
  3. The data_model field is a GenericForeignKey, so its reliability cannot be updated via F() expressions on the UserEvent queryset. A separate bulk update on the related data model table(s) will be needed.
  4. The exponentiation in INVERSE_EXPONENTIAL mode (decay_timedelta_days ** (decay_times + 1)) may require django.db.models.functions.Power() or a database function to express in SQL.

Please let me know if this approach works fine.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions