Horizontal scallability of backend #2251
-
|
Hi. I'm integrating MedusaJS at work for a project in house, and it started to make my personal interest lately as I become more and more familiar to it. I'm curious as I see the logs of the backend flooding with async jobs triggered, scheduled jobs etc., how it actually can perform in platforms that allow horizontal scaling of the apps (example Heroku, Digital Ocean's platform, Kubernetes), where the backend could have 2,3 or many more replicas running on multiple servers (all connecting to same Postgresql and Redis instances). BTW, are you going to support connecting to Redis in Cluster mode (if not yet)? My work is very oriented around high availability. |
Beta Was this translation helpful? Give feedback.
Replies: 4 comments
-
|
Hi @madalinignisca - thanks for the question. The async jobs are managed with Bull which uses Redis and we have seen examples of installations running on Heroku with multiple Dynos and other horizontal scaling setups; generally, there are no issues although we have experienced a few instances of competing DB queries creating transaction deadlocks custom async job subscribers do heavy DB work - we are currently reevaluating Medusa's transaction management to avoid these issues. To your second question about connecting to Redis in cluster mode: we don't have experience with this - I would guess it isn't possible with the current setup, but curious to hear if you give it a try :D We are planning to make the EventBusService agnostic so that it can use tools other than Redis/Bull in the future (e.g., Kafka), in that process I expect we will have an abstraction that will allow connections to clustered Redis installations too. Work on this will likely not begin before January :) |
Beta Was this translation helpful? Give feedback.
-
|
Hey @srindom, is there any news about the Redis Cluster feature ? |
Beta Was this translation helpful? Give feedback.
-
|
@lperrious check this plugin https://github.com/abdokouta/medusa-event-bus-kafka |
Beta Was this translation helpful? Give feedback.
-
|
How can I integrate my custom Kafka module built with NestJS into Medusa? |
Beta Was this translation helpful? Give feedback.
Hi @madalinignisca - thanks for the question.
The async jobs are managed with Bull which uses Redis and we have seen examples of installations running on Heroku with multiple Dynos and other horizontal scaling setups; generally, there are no issues although we have experienced a few instances of competing DB queries creating transaction deadlocks custom async job subscribers do heavy DB work - we are currently reevaluating Medusa's transaction management to avoid these issues.
To your second question about connecting to Redis in cluster mode: we don't have experience with this - I would guess it isn't possible with the current setup, but curious to hear if you give it a try :D
We are plan…