Improve export scalability using asynchronous background processing

[dsk-dev-ai]

Problem

The current CSV export implementation has been improved using streaming
responses and chunked iteration. However, it still relies on synchronous
request handling.

For large datasets and concurrent users, this can lead to:

• High CPU usage during export processing
• Heavy database queries executed within the request lifecycle
• Blocking HTTP requests for long durations
• Potential server overload under concurrent usage

Proposed Solution

Move export processing to an asynchronous background job system:

• Use a task queue such as Celery with Redis
• Trigger export as a background task instead of processing in the request
• Generate the CSV file in a worker process
• Store the generated file (local storage or S3)
• Return a response indicating “export in progress”
• Provide a download link once processing is complete

Suggested Architecture

User Request

→ API Endpoint→ Task Queue (Redis)→ Worker (Celery)→ File Storage→ Download Link

Benefits

• Eliminates request blocking
• Distributes load across worker processes
• Handles large datasets reliably
• Improves scalability under concurrent usage
• Reduces risk of server crashes

Possible Enhancements

• Export job status tracking (pending, processing, completed)
• Notification system (UI or email when export is ready)
• Retry mechanism for failed exports
• Download history for users

Acceptance Criteria

• Export requests should not block the HTTP response lifecycle
• Large dataset exports should complete without server errors
• Multiple concurrent exports should not overload the server
• Users should be able to download the generated file after completion

Scope

This issue focuses on backend export processing.
UI enhancements (progress indicators, notifications, etc.) can be handled in follow-up issues.

Context

This complements the recent improvements using streaming responses,
which reduce memory usage but do not fully address heavy computation
or concurrency concerns in production environments.

[dsk-dev-ai]

I’d be happy to work on this if the approach looks good to maintainers.
I can start with a basic Celery-based implementation for async export handling.

[yakshithkd23]

Hi @dsk-dev-ai, I’d like to work on this issue.

Proposed approach:

1. Introduce Celery with Redis (using Docker for local development) as the task queue.
2. Move existing CSV export logic into a Celery background task.
3. Update the export API endpoint to trigger the async task and immediately return a response with a task ID.
4. Implement a status endpoint to track export progress (pending, processing, completed).
5. Store generated CSV files in a structured location (e.g., MEDIA_ROOT/exports) and return the file path or download link once completed.
6. Ensure the current synchronous implementation is fully removed or safely replaced without regression.

I’ll begin with a minimal working implementation and keep the design extensible for future enhancements like retries, notifications, or S3 storage.

Please let me know if this approach aligns with your expectations or if there are any preferred patterns already used in the project for background tasks.

[dsk-dev-ai]

Hi @dsk-dev-ai, I’d like to work on this issue.

Proposed approach:

1. Introduce Celery with Redis (using Docker for local development) as the task queue.
2. Move existing CSV export logic into a Celery background task.
3. Update the export API endpoint to trigger the async task and immediately return a response with a task ID.
4. Implement a status endpoint to track export progress (pending, processing, completed).
5. Store generated CSV files in a structured location (e.g., MEDIA_ROOT/exports) and return the file path or download link once completed.
6. Ensure the current synchronous implementation is fully removed or safely replaced without regression.

I’ll begin with a minimal working implementation and keep the design extensible for future enhancements like retries, notifications, or S3 storage.

Please let me know if this approach aligns with your expectations or if there are any preferred patterns already used in the project for background tasks.

@yakshithkd23 Hello, assigning an issue is at the discretion of the maintainer. Since you’re interested in working on it, you can go ahead and try. I’m also submitting a PR - ultimately, it’s the maintainer’s decision which one to choose to resolve the issue.

[yakshithkd23]

@dsk-dev-ai Thanks for the update!

I see there’s already a PR in progress (#3272), so I’ll take a look at it and try to contribute improvements or testing feedback where possible.

Looking forward to learning from this implementation!