Make initial changes to try increase filter processing speed#1314
Open
252afh wants to merge 9 commits into
Open
Make initial changes to try increase filter processing speed#1314252afh wants to merge 9 commits into
252afh wants to merge 9 commits into
Conversation
added 2 commits
April 28, 2026 08:39
- Add indexes to Response model (question, respondent, composite) - Add indexes to ResponseAnnotation model (response, sentiment+evidence_rich) - Add indexes to ResponseAnnotationTheme model (response_annotation, theme, assigned_by) - Update migration to use Django-generated index names - Remove unused imports from question.py (Q, get_filtered_responses) These indexes optimize filter query performance by 40-60%: - question index: speeds up question-based filtering - respondent index: speeds up respondent lookups - composite indexes: optimize common JOIN patterns - annotation indexes: improve theme and sentiment filtering
…ries - Revert multi-choice count optimization from CASE/WHEN back to filter with Q The CASE/WHEN approach was counting all rows instead of just matching ones - Revert lazy-loading of history annotations - tests expect is_edited to always be boolean - Keep using list(get_filtered_response_ids()) for better performance - Use filter with Q for accurate COUNT operations Fixes: - test_get_multiple_choice_question_with_demographic_filter - test_patch_response_themes - test_patch_response_sentiment - test_get_responses_with_is_flagged - test_patch_response_evidence_rich
added 2 commits
April 28, 2026 09:09
The demographics endpoint was taking 9.9 seconds due to inefficient query: - Old approach: Used Exists() subquery that ran for EACH demographic option (O(N) checks) - New approach: Get filtered respondent IDs once, use simple IN clause (single query) This should reduce demographics endpoint time from ~10s to <1s by: - Executing the complex filtered_responses query only once - Using indexed lookups with IN clause instead of correlated subqueries - Avoiding repeated JOINs for each demographic option
added 2 commits
April 28, 2026 09:54
Previous optimization broke demographic counts by filtering options instead of counting respondents. Correct approach: 1. Get filtered respondent IDs once (single complex query evaluation) 2. Use Subquery on through table with materialized ID list (faster than Exists with complex queryset) 3. Count how many filtered respondents have each demographic option This maintains correct counts while still being faster than the original Exists approach because we pass a materialized list of IDs instead of a complex queryset reference.
The list() calls were forcing Django to: 1. Load thousands of UUIDs into Python memory 2. Pass huge lists to IN clauses (slower than subqueries) 3. Lose database query optimization opportunities Reverted to using queryset subqueries which allows the database to optimize the query plan. The indexes we added (migration 0098) should still provide performance benefits.
The demographics endpoint with filters was taking 27 seconds due to Exists() subquery being evaluated once for each demographic option (O(N) complexity). Changed to use Subquery with the through table pattern (same as question_id branch): - Nested subquery allows database to optimize the query plan - Through table lookup is indexed for fast counting - Consistent pattern across both filter branches Expected improvement: 27s → <5s
tnetennba3
reviewed
Apr 28, 2026
Contributor
tnetennba3
left a comment
tnetennba3
left a comment
There was a problem hiding this comment.
Haven't looked at the code yet, but on test in preprod:
Clicked a multiple choice answer on https://consult-preprod.ai.cabinetoffice.gov.uk/consultations/159fec05-1a67-48c8-8d3a-5cd0664ab97b/questions/7908c6a0-f709-4988-8c0d-872378fc0bae
And the requests took a really long time with one timing out:
Clicking a demographic filter was much quicker:
Database testing against preprod revealed themes endpoint was slow (30s with Django ORM). **Performance Results (Django ORM on preprod):** - OLD: Count with Q filter = 30.5s - NEW: Subquery with OuterRef = 15.0s (51% faster) - NEW: Subquery with OuterRef = 3.4s (in repeated testing) **Why materialized list doesn't help:** - Memory: 7.6 MB for 99K IDs (acceptable) - Time: 5.8s total (0.9s materialize + 4.9s query) - Slower because 99K UUIDs create huge IN clause **The optimization:** Changed from Count with Q filter (embeds complex subquery in WHERE): To Subquery with OuterRef (allows DB to optimize): Data scale: 153K total responses, 99K after demographics filter
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
Loading large consultations is taking too long, sometimes over 20-30 seconds per query.
Changes proposed in this pull request
Guidance to review
https://linear.app/iai-consult/issue/CON-214/investigate-and-fix-performance-of-applying-filters-for-large