Support for task logs in Loki

[lithorus]

Currently the render logs are stored locally on the render nodes and upload to the server when the task is done. For incomplete tasks, the server fetches the log from the render node and or streams it (listen).

This is not very scalable and the render server is often the bottleneck, but also if nodes crashes in the middle of a task the log might not be uploaded to the server.

Also, when jobs are removed the logs are also deleted from the server.

I propose that we have the option for streaming the logs from the renders nodes directly to a Loki server and then use the same server for fetching the logs from afwatch. It also makes searching/analyzing the log files MUCH easier.

[timurhai]

Hello!
We have a some busy,
I will talk

[timurhai]

Hell, Jimmy! Sorry for a delay.
May be this can be implemented by some plug-in.
May be just in Python?
I am not familiar with Loki.
But i agree that it can be useful to store all afrender logs.
I think it should be configurable for a special jobs, and disables by default, because it can be a big amount of data.

[lithorus]

Loki is a logging server which also supported object storage as a backend which makes it very scalable. It also compresses it in a very efficient way. Currently logs on the afserver are rather large compared to how well they can be compressed.

I think it's best to have in c++ since it will also be needed in afwatch. The point would also be to stream all logs directly to the logging server, so it's not necessary to fetch logs from the render nodes if they are not done.

It would of course be optional and controlled from the server. I wouldn't mind implementing it myself. I've already added it as an optional feature for OpenCue.

Another advantage is that it's very easy to do analytics on the logs since they are centralized. Loki also supports alert rules, which can notify on specific triggers.

[timurhai]

I am already changing logs workflow, but on server side, and this is not about tasks output.
#607
Now i am testing it a real work.

We can say that previously there was no logs workflow on server. Each node has own logs. Other events server just prints in stdout. There was no centralized logs. Also, no server logs were stored on restart.
Now server logs have some structure and stored in SQL database, the same as statistic.

Later it can be some option to send server logs to Loki too.

But if user does not want to store any logs, it should work "out-of-the-box" like now.
(Most users are small companies or even freelancers with several nodes, they do not manage any SQL database, they definitely does not want to deal with Loki or any other analytics).

[lithorus]

Again, it will only be optional.

Also, it's not about the server logs. That's totally fine it's on a separate server. What I'm referring to are the render task logs which usually are much larger then the server logs.

But I still think that it will be more scalable if the render nodes ingest it directly to Loki instead of having to go through the server first. It will also make it possible to use Loki for streaming the logs in afwatch instead of going through the server.