Skip to content

perf: fix N+1 queries in load_posts_keyed, add DB pool timeout - #374

Merged
kuny0707 merged 1 commit into
masterfrom
fix/discussion-n-plus-one-timeout
Jul 29, 2026
Merged

perf: fix N+1 queries in load_posts_keyed, add DB pool timeout#374
kuny0707 merged 1 commit into
masterfrom
fix/discussion-n-plus-one-timeout

Conversation

@ety001

@ety001 ety001 commented Jul 22, 2026

Copy link
Copy Markdown
Member

Summary

Three targeted performance fixes to address recurring hivemind outage (every 1-2 days, EB health goes Yellow with ~46% 4xx).

Root cause

get_discussion pipeline exhausts the 25-connection aiopg pool under burst traffic, starving health checks → ELB marks instance unhealthy → EB replaces it.

Changes

  1. Eliminate N+1 community queries (objects.py:load_posts_keyed)

    • Merged per-community title + role queries into 2 batch queries
    • Reduces DB round-trips from 2N to 2 (N = number of communities in thread)
    • A thread spanning 30 communities previously issued 60 extra queries
  2. Add connection pool acquisition timeout (db.py)

    • aiopg.sa.create_engine(timeout=10)
    • If all connections are busy, fail fast in 10s instead of blocking indefinitely
    • Normal queries complete in <1s; 10s is a safe upper bound
    • Prevents connection starvation cascading into health check failure
  3. Increase _child_ids cache TTL (thread.py)

    • 30s → 120s
    • Reduces repeated BFS lookups on deep comment trees
    • Most comment trees are static after initial load

Verification

  • make fmt
  • python3 -m py_compile ✓ (all 3 files)
  • Logic reviewed against production Scalyr logs showing get_discussion as 50% of failure traffic

- Batch-load community titles and roles in 2 queries instead of 2 per
  community, eliminating 2N round-trips for threads spanning many
  communities. Guard lookups so a stale community_id (e.g. a community
  deleted by fork rollback) falls back to the post category / guest
  role instead of raising KeyError.
- Add aiopg connection pool acquisition timeout (10s) to prevent
  health-check starvation when all connections are busy.
- Raise _child_ids cache TTL from 30s to 120s to reduce repeated
  lookups on deep comment trees.
@ety001
ety001 force-pushed the fix/discussion-n-plus-one-timeout branch from 3d1d4eb to 5af78d7 Compare July 23, 2026 17:38
@kuny0707
kuny0707 merged commit 32c0b26 into master Jul 29, 2026
1 check passed
@ety001
ety001 deleted the fix/discussion-n-plus-one-timeout branch July 29, 2026 14:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants