perf: remove statement_timeout, keep health_db isolation + pool timeout - #376
Merged
Merged
Conversation
PR #375 introduced statement_timeout=30s to cap runaway queries, but it caused a query amplification loop in production: PG cancels the query at 30s and releases the connection, but the Python async handler is NOT cancelled — it immediately acquires a new connection and retries, flooding RDS with concurrent queries. Simple queries went from 200ms to 341s, and hivemind memory piled to 99%. This commit removes statement_timeout entirely while keeping the four safe changes from PR #375: - health_db: isolated maxsize=1 engine for /health and /head_age (prevents ELB from killing healthy instances when the main pool is saturated) - MAX_DEPTH=50 / MAX_THREAD_POSTS=500: bounds _load_discussion recursion - hide_id lookups cached (300s TTL): reduces per-request DB connections - pool acquire timeout=10s (from PR #374): bounds waiting for a free connection Inbound load is now controlled at the openresty layer (PR steemit/openresty#18: get_state dual-track rate limiting) rather than killing queries server-side, which avoids the async-handler-not-cancelled trap entirely.
kuny0707
approved these changes
Aug 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Reverts the
statement_timeout=30schange from PR #375 while keeping all four safe changes.Background
PR #375 introduced
statement_timeout=30000(via libpqoptionsstring) to cap runaway queries server-side. In production (2026-08-01) this caused a query amplification loop:Root cause:
statement_timeoutkills the PG query but does NOT cancel the Python async coroutine (aiopg has no mechanism for this). The freed connection is immediately reused for a retried query, creating more concurrent load than before.Changes
Removed:
STATEMENT_TIMEOUT_MS = 30000constantoptionsstring construction ininit()(-c statement_timeout=30000)querydict merge logic (back to**conf.querydirectly)Kept (from PR #375):
health_db: isolatedmaxsize=1engine for/healthand/head_ageMAX_DEPTH=50/MAX_THREAD_POSTS=500: bounds_load_discussionrecursiontimeout=10(from PR perf: fix N+1 queries in load_posts_keyed, add DB pool timeout #374)Why Now
Inbound load is now controlled at the openresty layer (steemit/openresty#18):
get_statedual-track rate limiting (internal aggregate 30r/s + external per-IP 10r/s). This reduces hivemind inbound traffic at the source rather than killing queries server-side, avoiding the async-handler-not-cancelled trap entirely.After openresty rate limiting deployment, hivemind slow queries dropped from 10k+/5min to 0-71/5min, and the system is healthy. The remaining safe changes (health_db isolation, recursion caps, cache) further harden hivemind without the statement_timeout risk.
Verification
db.pysyntax verified (ast.parse)statement_timeout/STATEMENT_TIMEOUT/timeout_optreferences remainhealth_dbandconf.queryusage intacttests/bridge_thread/) unchanged — they test the thread recursion caps, not statement_timeout