Skip to content

Reconcile automations when Postgres notifications are unavailable#21089

Merged
zzstoatzz merged 3 commits intomainfrom
codex/21083-pgbouncer-automation-reconcile
Mar 11, 2026
Merged

Reconcile automations when Postgres notifications are unavailable#21089
zzstoatzz merged 3 commits intomainfrom
codex/21083-pgbouncer-automation-reconcile

Conversation

@zzstoatzz
Copy link
Collaborator

@zzstoatzz zzstoatzz commented Mar 11, 2026

closes #21083

this PR reconciles automations from the database when the triggers service cannot rely on PostgreSQL notifications alone.

Details
  • add a lightweight automation state snapshot and periodic reconciliation in the triggers service
  • keep the existing NOTIFY/LISTEN path as the fast path, but recover on startup, reconnect, and periodic evaluation
  • add trigger service tests for snapshot-based reconciliation behavior

Manual verification:

  • reproduced the failure locally with the full stack using Postgres + Redis + pgbouncer in transaction pooling mode
  • confirmed backend Postgres had 0 LISTEN "prefect_automation_changes" sessions in that topology
  • verified that, without restarting prefect server services, a newly created automation was picked up by reconciliation and emitted event traffic created a deployment flow run

Automated verification:

  • uv run pytest tests/events/server/triggers/test_service.py tests/events/server/triggers/test_service_changes.py
  • uv run ruff check src/prefect/server/events/triggers.py tests/events/server/triggers/test_service.py

@codspeed-hq
Copy link

codspeed-hq bot commented Mar 11, 2026

Merging this PR will not alter performance

✅ 2 untouched benchmarks


Comparing codex/21083-pgbouncer-automation-reconcile (0705a38) with main (c796a54)1

Open in CodSpeed

Footnotes

  1. No successful run was found on main (0b734db) during the generation of this report, so c796a54 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report.

Copy link
Collaborator

@chrisguidry chrisguidry left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense to me! Would love to understand why the LISTEN/NOTIFY didn't work, is it more that things drift over time?

@zzstoatzz zzstoatzz marked this pull request as ready for review March 11, 2026 17:27
@zzstoatzz
Copy link
Collaborator Author

zzstoatzz commented Mar 11, 2026

Would love to understand why the LISTEN/NOTIFY didn't work, is it more that things drift over time?

since LISTEN/NOTIFY is session-based, and pgbouncer transaction pooling does not preserve a stable pg session, i was able to repro that the listener was never reliably active in setups including pgbouncer

@zzstoatzz zzstoatzz added the fix A fix for a bug in an existing feature label Mar 11, 2026
@zzstoatzz zzstoatzz merged commit 958fbce into main Mar 11, 2026
78 checks passed
@zzstoatzz zzstoatzz deleted the codex/21083-pgbouncer-automation-reconcile branch March 11, 2026 17:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working fix A fix for a bug in an existing feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Automations created after PostgreSQL restart are not triggered until prefect-server-background-services is restarted

3 participants