Supporting django.tasks

[matthewhegarty]

hi All

Now that there is basic support for django.tasks in django6 are there any plans to add support to django-import-export-extensions?

We are considering adding background task support for django-import-export but perhaps it would be better suited to this project.

• It would save us duplicating work and would build on the effort you have already put in.
• Users could opt-in to installing this package for background task support.

However I would prefer that django-import-export users would not have to modify their existing code or there would be an easy route for them being able to define an import or export which runs as a background task. In the existing implementation they have to subclass CeleryModelResource which might not be possible for some users, and it locks them into the Celery implementation.

There is the start of a celery backend for tasks here.

I would love to hear your thoughts on this. There are a couple of us who could potentially help out.

Some more background here.

[OttoAndrey]

Hi! Sorry for delayed answer. Yes, I think we can try add support for django.tasks.

So as I can see your plan in parallel thread in django-import-export is:

1. add property for Meta-class;
2. add global setting;
3. re-use import/export tables from die-extensions;
4. re-use templates for django-admin from die-extensions;

and is it right that when developer choose use django.tasks (instead our celery implementation) he will also use his backend that he mentioned in TASKS setting? I mean die-extensions won’t force use celery?

For current moment I think about one problem: statuses. Looks like they are different. And our current implementation depends on celery. I need deeper investigation of this.

[matthewhegarty]

Hi Otto

Thank you for the response. Yes the developer would have the choice of backends which would be defined in settings, meaning that die-extensions would not force the use of celery. It should be possible to change the backend in settings config without requiring any other code changes.

django.tasks is still in its early phases, and the only existing celery implementation is quite basic. However I think being able to opt-in to die-extensions to add background tasks to django-import-export would be a great addition.

Please let me know if I can help at all.

[OttoAndrey]

hi! sorry for long wait answer.
I checked the code and i see a lot of places depends on celery. Adding setting for django.tasks and run export/import doesn’t problem. But make admin and API workable...

So, i see several places where code depends on celery, some of them easy to fix, some not:

• Rename app name. If we will add support for different backends. In that case “Celery …” in app name looks odd.
  • link
• Rename admin classes. Now they contains “Celery…” in their names, but actually they don’t use celery directly.
  • link
• Split CeleryResourceMixin to CeleryResourceMixin and ResourceMixin
  • link
• Export and Import models contain celery calls. Maybe we should move code with celery from Job models to Resource class
  • link
  • also remove “Celery” mention in field verbose names

I tried implement django-rq instead celery as example to understand how much changes client needs to do to use another backend. That what i found:

• Implement two methods for resource. Depends on django.tasks backend they use:
  • example for django-rq: first method, second method
• Overwrite logic for current status method for ExportJob model:
  • example for django-rq for ExportJob

without those change at least progress bar in django-admin doesn’t work and export also crashes.

Places that i didn’t check yet:

• Statuses. Probably we used statuses from celery for ExportJob/ImportJob and for consumers of package they need add map statuses for celery and their backend;
• cancel_export/cancel_import. These functions also relate to celery;
• I didn’t try import process by django-rq. But i guess it also requires changes;

At the moment, I’m skeptical about adding django.tasks because the end user has to perform many steps to implement their backend:

• implement two methods for resource
• overwrite logic for export;
• overwrite logic for import;
• overwrite cancel method;
• to map statuses;

Even if we provide an interface for implementing all the changes, integrating django.tasks will still seem complicated. It seems that adding just a single metaparameter or global setting isn’t enough

[matthewhegarty]

Thanks for investigating it. Sounds like the effort outweighs the benefits. I'm not planning to pursue this any further with django-import-export at this time.