Defer task on Django startup

[eitchtee]

I'm using Procrastinate on my Django application and I have a task that is ran periodically. What I would like to do is also run it when my app starts, after everything is correctly initialized. Something along the lines of @app.on_startup()

I've tried AppConfig.ready() and PROCRASTINATE_ON_APP_READY and both options seems to work, but besides raising a RuntimeWarning for accessing the database during app initialization, both are ran multiple times during the application's life-cycle including every time a worker is started using manage.py, which leads to multiple defers at the same time.

P.S.: Not sure if this is even possible as Django itself doesn't provide a proper way to detect when initialization is done, but would be nice if it was.

[ewjoachim]

when my app starts

What exactly does "when my app starts" means? You may have many different servers running your app (and multiple workers executing your jobs), and you may have rolling releases, so, do you mean "each time a process launches"? If so, I'd say what you mentioned seems to be the way.

My main point is: you handle processes. All your processes together form "your app". Processes live and die, but I don't think there's a common understanding for what "when your app starts" means.

If you want to do something "at deploy time", with your own definition of what "deploy" means, the best way to do it is to tie that to your deploy process (do you want to do it after the first server has been upgraded? Or the last? Or after every server upgrade? What if a single server has multiple independant processes?) Note that you may also tie that that to your service (either systemd's ExecStartPost, kubernetes' lifecycle postStart, and there's probably an equivalent for most systems), or embed that in a docker image or something.

[eitchtee]

My application runs gunicorn and a single procrastinate worker inside a single docker container using supervisord. I didn't try defering from the start script, that could work.

What exactly does "when my app starts" means?

When Django is fully initialized, including database connections, but as you said, I'm not sure if that's even a thing within Django.

My idea was:

1. I have a task that periodically (every 12 hours) checks for new versions of my application against an API, stores the result in a cache key;
2. The cache key is them checked elsewhere to correctly notify the user;
3. Ideally I would like to defer a task to re-check once the application was restarted in case it was updated

What I ended up doing was clearing the cache key on AppConfig.ready() and let procrastinate do it's thing later on as scheduled.

---

You may close this issue if it's not doable. Thanks for taking the time to respond.

[ewjoachim]

It's not that it's not doable, it's that I didn't have enough information to understand the specifics :) I'm afraid the details of what you're asking are very specific to your specific use-case, but that doesn't mean the problem isn't worth solving.

Here are a few ideas:

• AppConfig.ready() can work with caveats. You can watch sys.argv to get an idea of whether this is the server or the worker, but better yet: use an environment variable for that (ROLE=server / ROLE=worker), if you're doing structured logging, your logs will benefit from having that too (but of course, it's not the only way of passing this info). If you use gunicorn max_requests param, then logically, this will re-trigger every once in a while
• I think in your case, ExecStartPost might even work better (make your management command, or use procrastinate's management command or CLI)
• You could also launch a docker run --rm ./manage.py ... from your deploy script out of docker.
• It all depends on the specifics, but your task seems lightweight, so running it every 10 minutes could also be a solution.