Skip to content

feat: implement read replicas, queue-based math updates, and fix scal…#393

Merged
nicobao merged 1 commit into
zkorum:mainfrom
nicobao:load-testing-solve-performance-issues
Oct 21, 2025
Merged

feat: implement read replicas, queue-based math updates, and fix scal…#393
nicobao merged 1 commit into
zkorum:mainfrom
nicobao:load-testing-solve-performance-issues

Conversation

@nicobao

@nicobao nicobao commented Oct 21, 2025

Copy link
Copy Markdown
Member

…ability issues

Read Replica Infrastructure:

  • Add read replica support with automatic query routing via withReplicas()
  • Primary DB for writes, replica for SELECTs across API and math-updater
  • AWS RDS configuration with Secrets Manager support

Queue-Based Architecture:

  • Replace needsMathUpdate flag with conversation_update_queue table
  • Rate limiting based on lastMathUpdateAt (actual processing time)
  • Race-condition safe updates with requestedAt timestamp matching

Counter Reconciliation:

  • Self-healing counters reconciled by math-updater every 20s
  • Eliminates expensive COUNT queries from opinion/vote hot path
  • Updates lastReactedAt to track conversation activity

Math-Updater Scalability Fixes:

  • Upgrade pg-boss v10→v11 with 'singleton' queue policy
  • Prevent concurrent job execution for same conversation
  • Fix Drizzle ORM stack overflow with per-batch CASE statements (1000/batch)
  • Handle 113k+ votes, 19k+ opinions without crashes (50-85s processing time for backend math update, frontend runs smoothly all the way)

Load Testing:

  • Add k6 load testing with 2 scenarios (single/multiple conversations)
  • Grafana dashboards + Prometheus monitoring for PostgreSQL
  • UCAN crypto implementation for authenticated k6 tests

Performance Impact:

  • Eliminated database lock contention causing system hangs under load
  • Sequential math processing (no concurrent waste)
  • Self-healing counters accurate within 20 seconds
  • System responsive under 200 (very active) concurrent users per conversation

Documentation:

  • Comprehensive performance analysis with load test results
  • Counter reconciliation strategy documentation
  • AWS RDS deployment guide with read replica architecture

Python-bridge

  • rollback to previous version of red-dwarf that didn't have a bug that prevented math update
  • improve logging

@nicobao
nicobao force-pushed the load-testing-solve-performance-issues branch from 1cf631e to 6bfa0ed Compare October 21, 2025 19:19
…ability issues

Read Replica Infrastructure:
- Add read replica support with automatic query routing via withReplicas()
- Primary DB for writes, replica for SELECTs across API and math-updater
- AWS RDS configuration with Secrets Manager support

Queue-Based Architecture:
- Replace needsMathUpdate flag with conversation_update_queue table
- Rate limiting based on lastMathUpdateAt (actual processing time)
- Race-condition safe updates with requestedAt timestamp matching

Counter Reconciliation:
- Self-healing counters reconciled by math-updater every 20s
- Eliminates expensive COUNT queries from opinion/vote hot path
- Updates lastReactedAt to track conversation activity

Math-Updater Scalability Fixes:
- Upgrade pg-boss v10→v11 with 'singleton' queue policy
- Prevent concurrent job execution for same conversation
- Fix Drizzle ORM stack overflow with per-batch CASE statements (1000/batch)
- Handle 113k+ votes, 19k+ opinions without crashes (50-85s processing time)

Load Testing:
- Add k6 load testing with 2 scenarios (single/multiple conversations)
- Grafana dashboards + Prometheus monitoring for PostgreSQL
- UCAN crypto implementation for authenticated k6 tests

Performance Impact:
- Eliminated database lock contention causing system hangs under load
- Sequential math processing (no concurrent waste)
- Self-healing counters accurate within 20 seconds
- System responsive under 200 (very active) concurrent users per conversation

Documentation:
- Comprehensive performance analysis with load test results
- Counter reconciliation strategy documentation
- AWS RDS deployment guide with read replica architecture
@nicobao
nicobao force-pushed the load-testing-solve-performance-issues branch from 6bfa0ed to fe9e525 Compare October 21, 2025 19:20
@nicobao
nicobao merged commit 2b15251 into zkorum:main Oct 21, 2025
1 check passed
@nicobao
nicobao deleted the load-testing-solve-performance-issues branch October 21, 2025 19:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant