How can I create a SocketIO Flask App inside each background task in a redis-queue?

[caffeinatedMike]

I'm working on a Flask app that allows users to configure and schedule reports at a later time. I'm utilizing Flask-APScheduler to handle the scheduling, then inside the callback function I'm passing the actual task off to a redis-queue. What I want to do is before submitting the task to the redis queue I want to create a unique logfile name for that task, save it in my database, then send it off to the queue.

The reason I want to do this is so that I can allow users to click on a running task and view the logs in real-time.
My question really becomes: How can I create a SocketIO Flask App when that task starts, so that it can stream the logs to the browser if a user is viewing that particular task's logs?

I think where I'm getting caught up is how I would determine which SocketIO Flask App to connect to on the browser-side to stream the right logs. From my perspective (which could be incorrect), it looks like I'd need to create one SocketIO Flask App per task, even though it seems logical to utilize a single SocketIO Flask App and just declare one channel per redis task, but since redis task workers function independent of each other, how would that even be plausible?

A side note: My main app is a plain Flask App that is not utilizing SocketIO. I'm just looking to communicate info (stream lines of the logs) from the background rq worker processes to the browser/frontend for the end-user to see.

[miguelgrinberg]

I'm not sure you are looking at this problem in a way that is doable. Your Socket.IO app is your Flask app, and it needs to exist from the beginning and there is only one of them. The Flask app and the Socket.IO app are the same thing. What you are calling a "Socket.IO Flask app" is just a Socket.IO connection that the client makes to be able to see the logs of the background task.

[caffeinatedMike]

@miguelgrinberg Sorry for the confusion, I realize it's difficult to communicate what I'm trying to accomplish. For starters, here's a overview of how my background tasks work. I currently have a tasks.py file that houses the functions that are run in the rq workers. At the beginning of this file, I create an app instance using my app factory function so that each worker essentially has it's own isolated app in a sense while all still having access to the core components, such as the database. So, when I deploy my apps it's using two separate supervisor conf files, one to manage the main app, the other to manage the background rq workers. What I'm asking is could I, in my tasks.py wrap that app instance in the Flask-SocketIO?

For example:

from app import create_app

# needed because redis is not aware of the current running application in the other process
app = create_app()  # app factory function that passes back a configured flask app
app.app_context().push()

def background_task_one(a_param, b_param, c_param):
    pass

def background_task_one(a_param, b_param, c_param):
    pass

In the above file, could I use Flask-SocketIO to allow the app to emit lines of the log? Obviously the second aspect of this is allowing the end-users to receive the messages from the app. So, I figure each function could emit to a different channel (since a worker would only be executing a single function per job, is. rq worker 1 is executing background_task_one, rq worker 2 is executing background_task_two).

Would something like the following work?

from app import create_app
from flask_socketio import SocketIO, emit

# needed because redis is not aware of the current running application in the other process
app = create_app()  # app factory function that passes back a configured flask app
socketio = SocketIO(app)
app.app_context().push()

def background_task_one(a_param, b_param, c_param):
    socketio.emit('task_log',  {'data': 'task starting'}, namespace='/task/one')

def background_task_one(a_param, b_param, c_param):
    socketio.emit('task_log',  {'data': 'task starting'}, namespace='/task/two')

[miguelgrinberg]

As I said, I think you are overcomplicating this. Your workers cannot be servers, so they cannot host a Socket.IO server. You have one server (the one you call the Flask app). This should also be your Socket.IO server. So now your clients have the ability to send requests to your server, and also to make Socket.IO connections.

Your RQ workers can use the "emit from an external process" technique described in the Flask-SocketIO documentation to emit log lines to clients through the Socket.IO server.

Your use of namespaces in the last code example above is incorrect, because namespaces in Flask-SocketIO cannot be dynamically created. So drop that, and instead emit the logs to the appropriate client (by its sid) or to a room that interested clients can join when they want to see the logs.