Add batched database writes for offers and impressions via Redis

[ericholscher]

Summary

This PR implements a batched database write system that accumulates offer and impression counter updates in Redis and flushes them to the database in bulk periodically. This significantly reduces per-request database load for high-traffic ad serving scenarios.

Key Changes

• New batch_writer.py module: Core batching logic with functions to:

  • batch_incr_impressions(): Queue impression counter increments in Redis instead of immediate DB writes
  • batch_create_offer(): Queue offer creation in Redis with automatic flush when batch size is reached
  • get_pending_offer(): Look up offers still in the Redis queue before they're flushed to DB
  • flush_offer_queue(): Bulk create all pending offers using bulk_create()
  • flush_impression_counters(): Apply accumulated counter increments to AdImpression records using F() expressions

• Configuration options:

  • Global ADSERVER_BATCH_DB_WRITES setting to enable batching globally
  • Per-publisher batch_db_writes field for fine-grained control
  • ADSERVER_BATCH_SIZE setting to control flush threshold (default 100)
  • ADSERVER_BATCH_FLUSH_SECONDS for periodic flush scheduling

• Integration points:

  • Modified Advertisement.incr() to use batched writes when enabled
  • Modified Advertisement._record_base() to queue Offer creation instead of immediate insert
  • Updated get_offer() view to check Redis queue for pending offers before querying DB
  • Added new Celery task flush_batched_db_writes() for periodic flushing

• Database migration: Added batch_db_writes BooleanField to Publisher model

• Admin interface: Exposed batch_db_writes field in PublisherAdmin

• Comprehensive test suite: 344 lines of tests covering:

  • Configuration checks
  • Impression counter batching and flushing
  • Offer creation batching and flushing
  • Pending offer lookup
  • Deduplication logic
  • Edge cases (null advertisements, existing records, etc.)

Implementation Details

• Uses Redis hash keys to store impression counters with TTL to prevent accumulation
• Stores pending offers in Redis with 1-hour TTL for quick lookup before flush
• Implements deduplication in flush_offer_queue() to handle offers flushed individually via get_pending_offer()
• Uses get_or_create() + F() expressions for atomic counter increments on existing AdImpression records
• Gracefully falls back to immediate DB writes if Redis is unavailable
• Auto-triggers async flush via Celery when batch size threshold is reached
• Includes comprehensive error handling and logging throughout

https://claude.ai/code/session_01VSGX1CJZeq6VZsspxFAUeh