Problem
The observability.invariants.check worker task (polar/observability/invariants/tasks.py) runs every 15 minutes and consistently times out for two rules:
PayoutTransactionsAmountInvariantSubscriptionsCanceledDeletedCustomerInvariant
Logfire traces show the inner SELECT for both queries runs for exactly ~30.03-30.04s before erroring with an empty-message TimeoutError at asyncpg's bind_execute, indicating a fixed ~30s Postgres statement timeout is being hit consistently, not intermittently.
In a 6h production sample: 17 of 185 check task executions failed this way (~8%), concentrated entirely on these two rules (9 and 8 failures respectively); the other 5 invariant rules had zero failures.
Because these tasks are configured with max_retries=0, a timeout doesn't retry, it's silently dropped until the next scheduled cycle (surfaced only as a polar.worker.sqs_retry_exhausted log, no InvariantError/Slack alert since the query never completes).
Example traces
PayoutTransactionsAmountInvariant: trace 019fade880e8ad9f7c930f2682ed2d9e, span ced74fb7cb78a1efSubscriptionsCanceledDeletedCustomerInvariant: trace 019fade8ba029816dffec21748716dac, span 3862eb0d06175ff4
Likely cause
PayoutTransactionsAmountInvariant's query does an anti-join subquery on transactions.payout_id plus a GROUP BY aggregation over all rows with payout_transaction_id set, likely a large scan. SubscriptionsCanceledDeletedCustomerInvariant joins subscriptions to customers filtering on customers.deleted_at IS NOT NULL AND subscriptions.active IS TRUE, which may lack a supporting index.
Related prior art
Similar Postgres statement-timeout issues in worker tasks were previously fixed by:
Suggested fix
Sent by @joebon from Logfire polar-production-worker event investigation.
Problem
The
observability.invariants.checkworker task (polar/observability/invariants/tasks.py) runs every 15 minutes and consistently times out for two rules:PayoutTransactionsAmountInvariantSubscriptionsCanceledDeletedCustomerInvariantLogfire traces show the inner
SELECTfor both queries runs for exactly ~30.03-30.04s before erroring with an empty-messageTimeoutErrorat asyncpg'sbind_execute, indicating a fixed ~30s Postgres statement timeout is being hit consistently, not intermittently.In a 6h production sample: 17 of 185 check task executions failed this way (~8%), concentrated entirely on these two rules (9 and 8 failures respectively); the other 5 invariant rules had zero failures.
Because these tasks are configured with
max_retries=0, a timeout doesn't retry, it's silently dropped until the next scheduled cycle (surfaced only as apolar.worker.sqs_retry_exhaustedlog, noInvariantError/Slack alert since the query never completes).Example traces
PayoutTransactionsAmountInvariant: trace019fade880e8ad9f7c930f2682ed2d9e, spanced74fb7cb78a1efSubscriptionsCanceledDeletedCustomerInvariant: trace019fade8ba029816dffec21748716dac, span3862eb0d06175ff4Likely cause
PayoutTransactionsAmountInvariant's query does an anti-join subquery ontransactions.payout_idplus aGROUP BYaggregation over all rows withpayout_transaction_idset, likely a large scan.SubscriptionsCanceledDeletedCustomerInvariantjoinssubscriptionstocustomersfiltering oncustomers.deleted_at IS NOT NULL AND subscriptions.active IS TRUE, which may lack a supporting index.Related prior art
Similar Postgres statement-timeout issues in worker tasks were previously fixed by:
subscriptions(status, current_period_end))Suggested fix
customers.deleted_at,subscriptions.active,transactions.payout_id/payout_transaction_id).PayoutTransactionsAmountInvariantscan similar to server/meter: chunk billing-entry customer prefetch to fix SQL timeouts #13362/server/meter: bound billing entry backlog processing #13363.max_retries=0is appropriate for these tasks given failures are currently silent until the next 15-min cycle.Sent by @joebon from Logfire polar-production-worker event investigation.