Multiprocess help

[npcole]

Thank you so much for this library, which is invaluable.

There is a small part of my code which needs to be executed using the multiprocess module and which will need database access within each process. Do you have any advice as to how to handle this? Is doing something like:

new_session = flask_scoped_session(app.session_factory, app)

at the top of the relevant function the right way to do it?

Best wishes,

Nicholas

[dtheodor]

Are you running multiprocessing within a flask request? Because if you aren't, you don't need to use flask_scoped_session at all. I'm not sure if it makes sense to use a scoped session at all, if I would use multiprocessing with a few processes, I think I would create a new engine per process without a connection pool and create a session out of that. If you use a lot of processes and you really need a pool, I'm not sure SQLAlchemy provides a scoped session that will work across processes

[npcole]

There's a bit of processing for one flask call that definitely needs a multiprocessing approach (unless cached data is available) so I'm just looking to make that work...

[dtheodor]

I don't think SQLAlchemy's scoped_session supports multiprocessing at all. If you don't need the same database transaction that you have in your flask process to be transfered to the multiprocessing processes, create a new engine + NullPool + regular non-scoped session in the beginning of the function your processes will execute

[mirhmousavi]

I think you are wrong, as describe http://docs.sqlalchemy.org/en/latest/orm/contextual.html#using-thread-local-scope-with-web-applications for web applications or in general multithread/multiprocess applications you must use scoped sesions
@dtheodor @npcole

[Chris2048]

@npcole How are you running flask? Are you using a WSGI container such as uWSGI or Gunicorn?