Configure Prefect logger to capture default logs

[marvin-robot]

Opened from the Prefect Public Slack Community

akumar: Hi Team, How can make the prefect to log the messages from the default python logger to log to cloud, so for example If I get the logger via:

logger = logging.getLogger(__name__)

I have this in code everywhere and I don't want replace it with from prefect.utilities.logging import get_logger everywhere, is there a way to send the default logger messages to cloud?

josh: Hi <@U017DLT7H5L> you can access the logger that your task is using during the run like this:

@task
def my_task():
    from prefect import context
    logger = context.get("logger")
    <http://logger.info|logger.info>("log this")

Alternatively you could log stdout and that will be sent to the cloud logger:

@task(log_stdout=True)
def my_task():
    print("log this")

akumar: Hi Josh thanks for your reply, I have the default logger already everywhere in the code base, is there no way to use that without changing it to prefect logger?

akumar: As we would have to replace all of them with prefect logger and that code doesn't only runs on prefect, it is ran on other system as well.

josh: Oh I see, you can attach extra loggers to the prefect logger if you know the name of it like this https://docs.prefect.io/core/concepts/logging.html#extra-loggers

akumar: Yeah, I saw that but the name is not constant in this case logger: logging.getLogger(__name__) I guess, we need to use a common name then

josh: We’re looking to make some improvements to the logger and if you would like to open a ticket for this as a suggestion on the repo that would be great 🙂

akumar: Sure, will add it thanks. 🙂

akumar: Can you trigger the automated conversion of this conversation as a Github issue, then I can add more details if required?

josh: <@ULVA73B9P> open “Configure Prefect logger to capture default logs”

Original thread can be found here.

[aktech]

The motivation for this is to easily integrate prefect into existing systems from logging point of view, as in being able to use the default logging instead of the need to replace all logging initialisations in modules with prefect logger.

[saulbein]

Another possibility to avoid maintaining a list of libraries that log stuff is to attach the root logger ('' - empty string) to the extra loggers. This works, but causes duplicate logs for the pre-installed prefect loggers.

Some potential better solutions:

• Expose the entire logging configuration similarly to Dask (https://docs.dask.org/en/latest/debugging.html#logs)
• Allow users to not add the default prefect loggers

Personally, I would prefer the former rather than the latter, as this means I could have the entire logging setup in one place and would have full control on which libraries log at which levels (some libraries are really spammy on info level).

[namedgraph]

@cicdw so what is the solution here? Using Prefect 3.

[cicdw]

@namedgraph using the prefect.loggers.extra_loggers setting should do what you want: https://docs.prefect.io/v3/develop/logging#include-logs-from-other-libraries

[namedgraph]

@cicdw it does not work :) Logs for our libraries are not showing up in the UI.

[cicdw]

Hi @namedgraph - I'll need more info including an MRE in order to help; could you open a new issue with the needed information?

[namedgraph]

@cicdw attaching an MRE, to the best of my understanding :)

prefect-logging-mre.zip

[cicdw]

Hey @namedgraph thank you! Could you copy / paste the code and setup directly into GitHub? That way it's searchable and doesn't require anyone to download .zip files to their machines.

[namedgraph]

@cicdw here it is: https://github.com/namedgraph/prefect-logging-mre

[cicdw]

Great thank you @namedgraph - that was extremely helpful and I now understand what's going on. It's a little subtle but it is intended behavior (there was actually a discussion about it here for reference).

Prefect's extra_loggers configuration does one and only one thing: it adds new Prefect handlers to the extra loggers. It does not adjust any other configuration of those other loggers because oftentimes these extra loggers are emitted from package loggers that have their own configuration that needs to be respected.

What's happening in your case is that the logger you created with logging.getLogger(__name__) does not have a logging level set and so the handler is defaulting to NOTSET thereby not emitting INFO level logs. If you add logger.setLevel('INFO') it will begin emitting logs to the API as you expect. (One other side note: the name of the logger for me was 'logging_mre.main' so I had to adjust the Prefect setting accordingly).

[namedgraph]

@cicdw thanks for your response!

the name of the logger for me was logging_mre.main so I had to adjust the Prefect setting accordingly

I'm relatively new to Python logging, but shouldn't the loggers from submodules like logging_mre.main propagate to the top-level logger like logging_mre in this case? Our projects have extensive sub-module structures so adding all of them to config would not be feasible. Is such propagation possible with PREFECT_LOGGING_EXTRA_LOGGERS? Or maybe the modules are not setup correctly for this to work in this MRE?

[cicdw]

Ah you're right @namedgraph! Now that I've explicitly called setLevel I can set PREFECT_LOGGING_EXTRA_LOGGERS=logging_mre and it still works. Good catch.