Skip to content

perf: remove statement_timeout, keep health_db isolation + pool timeout - #376

Merged
kuny0707 merged 1 commit into
masterfrom
perf/pr375-without-statement-timeout
Aug 3, 2026
Merged

perf: remove statement_timeout, keep health_db isolation + pool timeout#376
kuny0707 merged 1 commit into
masterfrom
perf/pr375-without-statement-timeout

Conversation

@ety001

@ety001 ety001 commented Aug 1, 2026

Copy link
Copy Markdown
Member

Summary

Reverts the statement_timeout=30s change from PR #375 while keeping all four safe changes.

Background

PR #375 introduced statement_timeout=30000 (via libpq options string) to cap runaway queries server-side. In production (2026-08-01) this caused a query amplification loop:

PG cancels query at 30s → releases connection
  → hivemind async handler NOT cancelled → grabs new connection
    → new query also times out → repeat
      → RDS flooded with retried queries → I/O saturates
        → simple queries (200ms normal) take 341s+
          → hivemind memory piles to 99% (8GB c5.xlarge)

Root cause: statement_timeout kills the PG query but does NOT cancel the Python async coroutine (aiopg has no mechanism for this). The freed connection is immediately reused for a retried query, creating more concurrent load than before.

Changes

Removed:

  • STATEMENT_TIMEOUT_MS = 30000 constant
  • options string construction in init() (-c statement_timeout=30000)
  • query dict merge logic (back to **conf.query directly)

Kept (from PR #375):

Why Now

Inbound load is now controlled at the openresty layer (steemit/openresty#18): get_state dual-track rate limiting (internal aggregate 30r/s + external per-IP 10r/s). This reduces hivemind inbound traffic at the source rather than killing queries server-side, avoiding the async-handler-not-cancelled trap entirely.

After openresty rate limiting deployment, hivemind slow queries dropped from 10k+/5min to 0-71/5min, and the system is healthy. The remaining safe changes (health_db isolation, recursion caps, cache) further harden hivemind without the statement_timeout risk.

Verification

PR #375 introduced statement_timeout=30s to cap runaway queries, but it
caused a query amplification loop in production: PG cancels the query at
30s and releases the connection, but the Python async handler is NOT
cancelled — it immediately acquires a new connection and retries, flooding
RDS with concurrent queries. Simple queries went from 200ms to 341s, and
hivemind memory piled to 99%.

This commit removes statement_timeout entirely while keeping the four safe
changes from PR #375:

- health_db: isolated maxsize=1 engine for /health and /head_age (prevents
  ELB from killing healthy instances when the main pool is saturated)
- MAX_DEPTH=50 / MAX_THREAD_POSTS=500: bounds _load_discussion recursion
- hide_id lookups cached (300s TTL): reduces per-request DB connections
- pool acquire timeout=10s (from PR #374): bounds waiting for a free connection

Inbound load is now controlled at the openresty layer (PR steemit/openresty#18:
get_state dual-track rate limiting) rather than killing queries server-side,
which avoids the async-handler-not-cancelled trap entirely.
@kuny0707
kuny0707 merged commit d213ee5 into master Aug 3, 2026
1 check passed
@ety001
ety001 deleted the perf/pr375-without-statement-timeout branch August 9, 2026 23:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants