ReTraced — Distributed Job Queue with retry orchestration and observability (Redis + Node.js)

ReTraced is a Dockerized Distributed Job Queue with retry orchestration and observability built with
Node.js + TypeScript + Redis, focused on reliability, retries, DLQ, and observability.

ReTraced tells the lifecycle story of every job it executes — from execution to retries to DLQ.

---

Demo

---

Highlights (Strengths)

What makes ReTraced strong:

• Redis-backed distributed queueing
• At-least-once delivery using BRPOPLPUSH
• Delayed jobs using Redis ZSET
• Retry strategies: Exponential / Fixed / Linear / Three-tier
• Dead Letter Queue (DLQ) for poison jobs + manual debugging
• Visibility timeout + stuck job recovery watchdog
• Job metadata + history (observability-first approach)
• Dockerized setup (API + Worker + Redis)
• TypeScript type-safety end-to-end
• Unit + Integration tests with Jest
• Rate Limiting using Token Bucket (lua for atomic operations)

---

📚 Documentation Links

• Documentation* : Documentation →
• 🏗️ Architecture: Read Architecture →
• ⚠️ Limitations: Read Limitations →
• 🔁 Retries & Backoff: Retry Design →
• 🧪 Technical Model: Technical Model →
• 🆚 Comparison: ReTraced vs BullMQ / Sidekiq / Others →

---

System Overview

Core Components

• API Service

  • Creates jobs
  • Stores job metadata
  • Pushes job IDs into Redis ready queue

• Worker Service

  • Consumes jobs from Redis
  • Executes handler (webhook / job processor)
  • Retries on transient failures
  • Moves permanent failures to DLQ

• Redis

  • ready → jobs waiting to execute
  • processing → jobs currently being executed
  • delayed (ZSET) → scheduled retries / delayed jobs
  • dlq → permanently failed jobs

---

Retry Strategy (Three-Tier)

ReTraced supports a three-tier retry model:

1. Immediate retry (short transient failures)
2. Backoff retry (exponential / fixed / linear)
3. DLQ fallback after max tries

---

Quick Start (Run Locally)

Clone

git clone https://github.com/Anshikakalpana/job-scheduler

docker compose up --build      ( for one worker )

docker compose up --scale worker=5    ( for multiple workers )


 Enqueue Test Jobs

docker compose exec worker node dist/handlers/email.handler.js

docker compose exec redis redis-cli

GET job:job-0

Roadmap

• Multiple worker replicas (horizontal scaling)
• Job prioritization
• Cron / scheduled jobs
• Job cancellation + pause/resume queues
• Prometheus / Grafana metrics
• Distributed tracing (OpenTelemetry)
• Lua scripts for fully atomic Redis operations
• PostgreSQL persistence for long-term storage

License

MIT License — free for personal & commercial use.

---

Contributing

Contributions, issues, and feature requests are welcome!
Feel free to check the issues page.

Guidelines

• Fork the repository
• Create your feature branch (git checkout -b feature/amazing-feature)
• Commit your changes (git commit -m 'Add some amazing feature')
• Push to the branch (git push origin feature/amazing-feature)
• Open a Pull Request

---

Show Your Support

Give a ⭐️ if this project helped you!

---