⚙️ Django Celery Async Tasks

A production-ready async task processing system built with Django, Celery, Redis, PostgreSQL, and Docker.

---

📌 Features

• ⚙️ Background Job Processing — Submit long-running jobs (report generation, data export) as async tasks
• 🕐 Scheduled Tasks (Celery Beat) — Cron-style periodic tasks stored in the database
• 📧 Async Email Notifications — Welcome, alert, and digest emails queued via Celery
• 🔁 Webhook Delivery + Retries — Fan-out event delivery with exponential backoff retries
• 📊 Task Results in PostgreSQL — All job results stored via django-celery-results
• 🔀 Multiple Queues — Separate queues for default, email, webhooks, scheduled
• 📖 Swagger UI — Full interactive API documentation

---

🏗️ Architecture

┌─────────────┐     HTTP      ┌──────────────┐
│   Client    │ ────────────▶ │  Django Web  │
└─────────────┘               └──────┬───────┘
                                     │ .delay() / .apply_async()
                              ┌──────▼───────┐
                              │    Redis     │  ◀── Message Broker
                              └──────┬───────┘
                    ┌────────────────┼────────────────┐
             ┌──────▼──────┐  ┌─────▼──────┐  ┌──────▼──────┐
             │   Worker    │  │   Worker   │  │    Beat     │
             │  (default)  │  │  (email/   │  │ (Scheduler) │
             │             │  │  webhooks) │  │             │
             └──────┬──────┘  └─────┬──────┘  └─────────────┘
                    └───────────────┘
                           │
                    ┌──────▼───────┐
                    │  PostgreSQL  │  ◀── Job results, Beat schedules
                    └──────────────┘

---

🏗️ Project Structure

django-celery-tasks/
├── apps/
│   ├── jobs/                    # Background job processing
│   │   ├── models.py            # Job model (status, result, retries)
│   │   ├── tasks.py             # generate_report, export_data, cleanup_old_jobs
│   │   ├── views.py             # Submit, list, cancel jobs
│   │   ├── serializers.py
│   │   ├── urls.py
│   │   └── management/commands/
│   │       └── seed_beat_schedules.py
│   ├── notifications/           # Async email notifications
│   │   ├── models.py            # Notification model
│   │   ├── tasks.py             # send_email, send_welcome, send_alert, daily_digest
│   │   ├── views.py
│   │   ├── serializers.py
│   │   └── urls.py
│   └── webhooks/                # Webhook delivery + retries
│       ├── models.py            # WebhookEndpoint, WebhookDelivery
│       ├── tasks.py             # deliver_webhook, trigger_event, retry_failed
│       ├── views.py
│       ├── serializers.py
│       └── urls.py
├── config/
│   ├── settings.py              # Celery, Redis, Beat, Email config
│   ├── celery.py                # Celery app instance
│   ├── urls.py
│   └── wsgi.py
├── scripts/
│   ├── entrypoint.sh            # Web: migrate → seed → runserver
│   ├── start_worker.sh          # Celery worker startup
│   └── start_beat.sh            # Celery beat startup
├── tests/
│   └── test_all.py
├── Dockerfile
├── docker-compose.yml           # 5 services: db, redis, web, worker, beat
└── .env.example

---

🚀 Quick Start

git clone https://github.com/harshalkumar-ishi/django-celery-tasks.git
cd django-celery-tasks
cp .env.example .env

docker-compose up --build -d

This starts 5 containers:

Container	Role
celery_db	PostgreSQL database
celery_redis	Redis message broker
celery_web	Django API server
celery_worker	Celery worker (all queues)
celery_beat	Celery beat scheduler

---

📖 API Endpoints

Jobs

Method	Endpoint	Description
GET	/api/v1/jobs/	List all jobs
GET	/api/v1/jobs/<id>/	Get job status & result
POST	/api/v1/jobs/submit/report/	Submit report generation job
POST	/api/v1/jobs/submit/export/	Submit data export job
POST	/api/v1/jobs/<id>/cancel/	Cancel a pending job

Notifications

Method	Endpoint	Description
GET	/api/v1/notifications/	List all notifications
POST	/api/v1/notifications/send/	Send custom email async
POST	/api/v1/notifications/welcome/	Queue welcome email
POST	/api/v1/notifications/alert/	Queue alert email

Webhooks

Method	Endpoint	Description
GET/POST	/api/v1/webhooks/endpoints/	List / Register endpoints
GET/PUT/DELETE	/api/v1/webhooks/endpoints/<id>/	Manage endpoint
GET	/api/v1/webhooks/deliveries/	List deliveries
GET	/api/v1/webhooks/deliveries/<id>/	Delivery detail
POST	/api/v1/webhooks/deliveries/<id>/retry/	Retry failed delivery
POST	/api/v1/webhooks/trigger/	Trigger event (fan-out)

---

🕐 Scheduled Tasks (Celery Beat)

Seeded automatically on startup:

Task	Schedule	Description
send_daily_digest	Daily at 08:00 UTC	Sends digest emails to all users
cleanup_old_jobs	Daily at midnight	Deletes jobs older than 30 days
retry_failed_webhooks	Every 15 minutes	Re-queues failed webhook deliveries

Manage schedules via Django Admin → Periodic Tasks.

---

🔀 Queue Architecture

Queue	Purpose
default	Report generation, data export, general tasks
email	All email sending tasks
webhooks	Webhook delivery tasks
scheduled	Beat-triggered periodic tasks

---

🔁 Webhook Retry Strategy (Exponential Backoff)

Attempt 1 — immediate
Attempt 2 — 60 seconds later
Attempt 3 — 300 seconds later
After 3 failures → status = FAILED (manual retry available via API)

Payloads are signed with HMAC-SHA256 when a secret is set on the endpoint:

X-Webhook-Signature: <hmac-sha256-hex>

---

🧪 Running Tests

docker-compose exec web pytest

---

🛠️ Useful Commands

# View all container logs
docker-compose logs -f

# View worker logs only
docker-compose logs -f worker

# View beat scheduler logs
docker-compose logs -f beat

# Check running containers
docker-compose ps

# Open Django shell
docker-compose exec web python manage.py shell

# Manually trigger a task in shell
from apps.jobs.tasks import generate_report
generate_report.delay(job_id='test', report_type='sales')

---

📦 Tech Stack

Layer	Technology
Framework	Django 4.2, Django REST Framework 3.15
Task Queue	Celery 5.3
Broker	Redis 7
Beat Scheduler	django-celery-beat 2.6
Task Results	django-celery-results 2.5
Database	PostgreSQL 15
Containerisation	Docker, Docker Compose
Docs	drf-yasg (Swagger + ReDoc)
Testing	pytest-django

---

Built by Harshalkumar Ishi