Working on enabling HA architecture with OSS servers

[zzstoatzz]

this discussion is meant as a place to compile thoughts related to running many instances of prefect server using redis as the message broker

as of 05/22/2025 there are some outstanding issues with this, see:

• missing events with multi-server setup #18049
• Prefect Server Experiencing Timeouts Due to Slow Database Communication #16299 (comment)

as of aug 2025, the above issues are addressed. outstanding known issues:

• some background services are not safe to horizontally scale, see document and improve horizontal scalability of background services #18753

TODO:

• event consumers should have unique names (multi-server setup fixes pt1 #18149)
• allow full configuration of redis consumer and publisher settings (add redis publisher/consumer settings #18160)
• min group id fix (prefect-redis: fix min group id and deprecation warning #18215)
• we need to signal updates to automations (use pg notify/listen for multi-server automation synchronization #18266)
• allow independent scaling of background services (document and improve horizontal scalability of background services #18753)
• long tail of benchmarking, identifying issues, making improvements

---

Documentation on scaling open source installations can now be found here!

[fatih-acar]

Hi @zzstoatzz, thanks for opening up this discussion thread.

I'd like to point you to this extra fix: 3568d26

It was pretty hard to debug because the consumer would not throw any exception (probably because it is caught somewhere), even with debug level for logging. This commit fixes the initial issue I had, but having the exception logged would have helped a lot (and it will probably help catching any other potential issue).

Also we'd like to have the publisher batch size configurable (hardcoded fix was there: f8f2a0b) so that we don't get any "lag" when using triggers/automations (because the more replicas you have, the longer you have to wait when there is little activity).

---

Other than that, we've been testing these changes to achieve better performance regarding task/flow execution throughput and it worked good so far (it scales well for our use case). We don't know if there are other issues around due to in-memory storage (like the one for automations), but you probably do.

[fatih-acar]

#18160 this is great, thanks for the quick turnaround!

We're looking forward to giving a try to the PG NOTIFY/LISTEN solution, this is indeed a more robust one. Thanks for the information, will be waiting for a PR :)

[zzstoatzz]

hi @fatih-acar - draft PR is open here! will likely undergo a round of reviews or two, but feel free to try it out if you have the energy (initial testing lgtm)

[fatih-acar]

Thanks for the heads up! Will give it a try very soon 👍

[zzstoatzz]

this is now on main!

edit: and now released in 3.4.6

[ogenstad]

Nice, thank you!

[fatih-acar]

Also regarding the docker compose example you provided here: #16299 (comment)

The migrate "init" container solves the concurrent DB initialization issue, but we also hit a similar issue when initializing blocks. But unfortunately we do not have a straightforward command to initialize blocks...

To workaround this, we implemented a new "bootstrap" command that is available here: opsmill/infrahub@6a86339

[zzstoatzz]

great! thanks for sharing that

[fatih-acar]

Another issue I forgot to report: the default value of PREFECT_REDIS_MESSAGING_CONSUMER_MIN_IDLE_TIME is 0, which seems invalid. In our testing, we see some messages claimed by multiple consumers because of that (and thus actions may be executed twice or more, when we have multiple replicas with triggers service enabled).

I think we should change the default value, but last time I tried that CI tests wouldn't pass.

[zzstoatzz]

thank you! good call, updated here: #18215

[fatih-acar]

Hi @zzstoatzz,
We're in the process of using the HA setup in production, but it seems we're missing a recent prefect-redis release in PyPI...
Would it be possible for you to release a new prefect-redis package please?

Thanks a lot for the work on this!

Edit: ping @desertaxle as @zzstoatzz seems off

[desertaxle]

@fatih-acar I've published prefect-redis==0.2.3 to PyPI!

[fatih-acar]

Thanks @desertaxle for the quick turnaround! Works like a charm! 👍