A side project demonstrating an end-to-end async image processing pipeline with distributed tracing across web, worker, and object storage. Upload an image → Django persists it to MinIO → Celery generates a thumbnail → status surfaced back to the user — every hop stitched into a single trace in Jaeger.
The point of this repo is not the thumbnail. It is the plumbing around it: clean domain modeling, owner isolation, retry semantics, and observability that actually works across process boundaries.
A single trace_id flows from the HTTP request through apply_async, into the Celery worker, down into S3/MinIO calls — with correct parent/child nesting. Getting this right required fixing four separate root-cause bugs in OpenTelemetry wiring; the post-mortem lives in docs/OBSERVABILITY.md and is the most valuable artifact in this repo.
[app] POST /api/tasks/
[app] └─ apply_async/generate_thumbnail ← traceparent injected here
[worker] └─ run/generate_thumbnail ← extracted, parented
[worker] └─ celery.task.generate_thumbnail
[worker] ├─ minio.download → S3.GetObject
[worker] └─ minio.upload → S3.PutObject
Image and its processing lifecycle are separate aggregates. A retry creates a new ImageTask row — full audit history, the Image record stays clean. Owner isolation enforced at the QuerySet layer (Image.objects.for_user(request.user)).
generate_thumbnail distinguishes between DONE / INVALID (don't retry) / RETRY (exponential backoff) — span attributes carry the outcome so traces are queryable by failure mode.
Thumbnails served via short-lived S3 pre-signed URLs scoped per-user. No anonymous reads against MinIO.
settings/test.py swaps storage to InMemoryStorage and forces CELERY_TASK_ALWAYS_EAGER — tests run without MinIO, Redis, or Jaeger. CI spins up only Postgres as a service container.
| Layer | Tech |
|---|---|
| Web | Django 6, django-bootstrap5 |
| Async | Celery + Redis broker, @shared_task |
| Storage | django-storages[s3] + boto3 → MinIO (S3-compatible) |
| Database | PostgreSQL 17 |
| Tracing | OpenTelemetry SDK → OTLP gRPC → Jaeger all-in-one |
| Tooling | uv, ruff, pytest-django, factory-boy |
| CI | GitHub Actions |
make up # postgres + redis + minio + jaeger
make migrate
make dev # terminal 1 — Django on :8000
make worker # terminal 2 — Celery worker
make smoke # upload → dispatch → poll status
open http://localhost:16686 # Jaeger UI — full nested trace
open http://localhost:8000 # login as test1@example.com / test1Test accounts seeded automatically: test1@example.com / test1, test2@example.com / test2.
django_thumbnail/
├── django_thumbnail/
│ ├── settings/
│ │ ├── base.py env-driven config
│ │ ├── local.py
│ │ └── test.py InMemoryStorage + eager Celery
│ ├── celery.py
│ └── telemetry.py OTel TracerProvider setup
└── images/ the only app
├── models.py Image, ImageTask, ImageStatus
├── tasks.py generate_thumbnail (idempotent, retrying)
├── apps.py Django+Celery+Botocore instrumentor wiring
├── utils.py S3 helper (direct put/get, no s3transfer threads)
├── views.py plain JsonResponse, @login_required
└── tests/ pytest-django + factory-boy
docs/OBSERVABILITY.md ★ the trace-stitching post-mortem
.github/workflows/ci.yml lint + pytest on Postgres service
- No DRF. Plain
JsonResponse+@login_required— small surface, less magic. - Direct boto3
put_object/get_objectinstead ofupload_fileobj— avoidss3transferthread hops that break OTel context propagation. Seedocs/OBSERVABILITY.mdSymptom 2. SimpleSpanProcessorin dev,BatchSpanProcessorin prod — dev workflow needs spans visible immediately even onSIGTERM.- Programmatic OTel init in
apps.py:ready()instead of theopentelemetry-instrumentCLI wrapper — the wrapper accesseddjango.conf.settingsbeforeDJANGO_SETTINGS_MODULEwas applied, breakingrunserver. makeis the only entry point.manage.pyis never invoked directly so env vars (OTEL service name, settings module) are always consistent.
docs/OBSERVABILITY.md— five symptoms, five root causes, one working distributed tracetasks/THUMBNAIL_WORKER.md— phased build logCLAUDE.md— context for AI-assisted development