[Bug]: Events aren't getting consumed by subscribers, causing unprocessed events and memory leaks

[mateomoreno]

Package.json file

{
  "name": "marketplace-backend",
  "version": "0.0.1",
  "description": "A starter for Medusa projects.",
  "author": "Medusa (https://medusajs.com)",
  "license": "MIT",
  "keywords": [
    "sqlite",
    "postgres",
    "typescript",
    "ecommerce",
    "headless",
    "medusa"
  ],
  "scripts": {
    "build": "rm -rf .medusa && medusa build",
    "seed": "medusa exec ./src/scripts/seed.ts",
    "start": "medusa start",
    "dev": "medusa develop",
    "test:integration:http": "TEST_TYPE=integration:http NODE_OPTIONS=--experimental-vm-modules jest --silent=false --runInBand --forceExit",
    "test:integration:modules": "TEST_TYPE=integration:modules NODE_OPTIONS=--experimental-vm-modules jest --silent --runInBand --forceExit",
    "test:unit": "TEST_TYPE=unit NODE_OPTIONS=--experimental-vm-modules jest --silent --runInBand --forceExit --detectOpenHandles",
    "predeploy": "medusa db:migrate",
    "uploadimage:test": "node src/scripts/run-cloudinary-upload.js"
  },
  "dependencies": {
    "@anthropic-ai/sdk": "^0.71.2",
    "@google-shopping/products": "^0.8.0",
    "@medusajs/admin-sdk": "2.11.3",
    "@medusajs/cli": "2.11.3",
    "@medusajs/framework": "2.11.3",
    "@medusajs/medusa": "2.11.3",
    "@opentelemetry/api": "^1.9.0",
    "@opentelemetry/exporter-trace-otlp-grpc": "^0.207.0",
    "@opentelemetry/sdk-node": "^0.207.0",
    "@portabletext/block-tools": "^1.1.35",
    "@sanity/block-tools": "^3.70.0",
    "@sanity/client": "^7.3.0",
    "@sanity/schema": "^4.10.2",
    "@sanity/types": "^3.97.1",
    "@sentry/node": "^10.22.0",
    "@sentry/opentelemetry-node": "^7.114.0",
    "@sentry/profiling-node": "^10.22.0",
    "@shopify/storefront-api-client": "^1.0.6",
    "@types/jsdom": "^21.1.7",
    "algoliasearch": "^5.29.0",
    "cloudinary": "^2.6.0",
    "facebook-nodejs-business-sdk": "^24.0.1",
    "googleapis": "^166.0.0",
    "groq": "^3.92.0",
    "jsdom": "^27.0.0"
  },
  "devDependencies": {
    "@medusajs/test-utils": "2.11.3",
    "@swc/core": "1.5.7",
    "@swc/jest": "^0.2.36",
    "@types/jest": "^29.5.13",
    "@types/node": "^20.0.0",
    "@types/react": "^18.3.2",
    "@types/react-dom": "^18.2.25",
    "jest": "^29.7.0",
    "prop-types": "^15.8.1",
    "react": "^18.2.0",
    "react-dom": "^18.2.0",
    "ts-node": "^10.9.2",
    "typescript": "^5.6.2",
    "vite": "^5.2.11",
    "yalc": "^1.0.0-pre.53"
  },
  "engines": {
    "node": ">=20"
  },
  "packageManager": "yarn@1.22.22+sha512.a6b2f7906b721bba3d67d4aff083df04dad64c399707841b7acf00f6b133b7ac24255f2652fa22ae3534329dc6180534e98d17432037ff6fd140556e2bb3137e"
}

Node.js version

v22.0.0

Database and its version

PostgresSQL 16.11

Operating system name and version

Ubuntu 22.04 LTS or Ubuntu 24.04 LTS

Browser name

Chrome

What happended?

Just to clarify, we've been running into this issue for a couple of months now and there are a couple of owned and peer threads on Discord talking about this issue. Our main Medusa architecture consists of various systems including Shopify stores, Algolia indexes, Anthropic APIs, etc. They all speak to each other through either scheduled jobs, manual workflow triggers and also important events and subscribers that keep everything synced. Over the past couple of months we started running into memory leaks in our Redis instance due to events piling up and not getting processed. We did a pretty good job in tryying to reduce the number of events in ordfer to redu e the load on the Redis instance. While this helped a little bit, events soon started piling up again and in most cases events aren't getting picked up by subscribers so logic never executes. These events are critical events like product.created, product.update and order.created.

Expected behavior

Expected behavior is for events to get added to the event bus/queue and processed with retries if needed and ttls. Mainly like so:

Events emitted → queued in Redis → processed by worker → subscribers triggered
Events processed in order (FIFO)
No events lost
Worker processes queue continuously

Actual behavior

Events are being added to Redis but are never processed, causing them to pile up and crash the Redis instance

Link to reproduction repo

n/a

[scherddel]

Hey @mateomoreno

Can you share a deeper insight on the analysis you have done specifically on your redis instance? How did you find out about these numbers? What debugging steps were performed in the redis-cli?

Thanks

[mateomoreno]

Hey @scherddel, sure. The issue has been very hard to debug because there really aren't errors, just unprocessed events that pile up, resulting in unprocessed events and eventually crashing our redis instance. Please find more information below:

1. We first found out about this when our Redis instance crashed due to a memory leak. We have been growing, which has resulted in more events. When this memory leak occured, we actually found there were millions of unprocessed events. Our observability tools skyrocketed in 1-2 days.
2. We manually purged all of these events, hoping it had been a one off loop that had broken. This seemingly fixed the issue and our Redis instance was back up and running. We added logs to see if there were any logs causing infinite loops or something similar.
3. We soon ran into the same issue again. Our Redis instance skyrocketed and crashed again with millions of unprocessed events. Based on the Medusa documentation, the event bus could handle a high number of events. The event bus seemingly had its own logic to consume events, retry and delete whenever necessary. Since the event bus is configured by Medusa, it was hard to really understand how events were getting handled. Regardless, the issue was not an error on our end. It was just unprocessed events.
4. This is our Redis setup in our medusa.config:

modules: [ { resolve: '@medusajs/medusa/cache-redis', options: { redisUrl: process.env.REDIS_URL, }, }, { resolve: '@medusajs/medusa/event-bus-redis', options: { redisUrl: process.env.REDIS_URL, }, }, { resolve: '@medusajs/medusa/workflow-engine-redis', options: { redis: { url: process.env.REDIS_URL, }, }, },

5. We tried being more particular with events priority and did some cleaning up on our end. We are still running into unprocessed events. Events are simply not getting processed. Seemingly no errors in our logs. Events get created but subscribers/consumers don't seem to catch them
6. We've tried to clean up as much as we can and even added another worker to see if it was a compute issue. The issue persisted.
7. Even today in our server observability tools, our memory keeps increasing and at some point every 2 weeks we have to purge, resulting in a loss of a lot of valuable data
8. We have a lot of systems speaking to each other and product.created, product.updated and order.created events are very valuable. These are getting missed.

Any direction here would be much appreciated. We've been trying with no luck.

[mateomoreno]

UPDATE With no extra configuration other than the one i sent above the Redis-backed Medusa Event Bus (@medusajs/medusa/event-bus-redis) is not being consumed by any worker, causing the events-queue:wait list to grow indefinitely (millions of jobs), eventually exhausting Redis memory.
Despite having multiple Medusa workers running, no process is actively consuming the Event Bus queue, while other Bull/BullMQ queues (workflows) are working correctly.