Job Queue System

A robust, asynchronous job queue system built with Node.js, Express, BullMQ, and Redis.

Features

• Express API: Exposes endpoints to easily enqueue new jobs.
• BullMQ: Powerful Redis-based queue for reliable background job processing.
• Retry Mechanism: Exponential backoff configured for failed jobs (retries 3 times with a 5-second initial delay).
• TypeScript: Strictly typed backend code.

Prerequisites

Before running the project, make sure you have the following installed:

• Node.js (v18+ recommended)
• Redis server running on localhost:6379 (default BullMQ configuration)

Installation

1. Clone the repository and navigate to the project directory:

   cd job-q-sys

2. Install the dependencies:

   npm install

3. Set up your environment variables by copying .env.example to .env (if applicable), ensuring you configure the appropriate values like PORT=3000.

Running the Application

The application is split into two components: the API server and the background worker. You'll need to run both concurrently in separate terminal windows.

1. Start the API Server

This process handles incoming HTTP requests and pushes jobs to the Redis queue.

npm run dev2

The server will start listening on port 3000 (or the port defined in your .env).

2. Start the Background Worker

This process listens to the Redis queue and executes the job logic.

npm run dev1

API Documentation

The project includes an OpenAPI specification file (openapi.yaml) detailing the available endpoints.

Endpoints

POST /job

Adds a new job to the queue.

Request Body (JSON):

{
  "type": "email",
  "payload": {
    "to": "test@example.com",
    "subject": "Hello!"
  }
}

Response (JSON):

{
  "message": "Job added to queue",
  "jobId": "1"
}

How It Works

1. A client sends a POST /job request to the Express server.
2. The server creates a new job using BullMQ (jobQueue.add()) with the provided type and payload, configured for 3 attempts and an exponential backoff of 5000ms.
3. The BullMQ Worker picks up the job from Redis.
4. The worker processes the job based on its type (e.g., if type === "email", it currently throws an error to demonstrate the failure/retry mechanism, otherwise it completes after a 2-second delay).
5. The worker logs the completion or failure status.