Skip to content

Setting a basicConfig before importing distributed changes the scheduler logging level. #8805

Open
@Thomas-Z

Description

@Thomas-Z

Hello,

Some kind of follow-up to the #7527 issue.

Describe the issue:

Setting a basicConfig before the distributed import changes the logging level of the scheduler to INFO.
Doing the same with the distributed import before the logging configuration does work as intended.

Minimal Complete Verifiable Example:

import logging

DEFAULT_FORMAT = "%(asctime)s %(levelname)-7s %(message)s"
DEFAULT_TIME_FORMAT = "%Y-%m-%d %H:%M:%S"

logging.basicConfig(format=DEFAULT_FORMAT, datefmt=DEFAULT_TIME_FORMAT)


import distributed

loc_cluster = distributed.LocalCluster(
    processes=False, n_workers=1, silence_logs="ERROR"
)

This code output the following logs:

2024-07-27 09:52:13 INFO    To route to workers diagnostics web server please install jupyter-server-proxy: python -m pip install jupyter-server-proxy
2024-07-27 09:52:13 INFO    State start
2024-07-27 09:52:13 INFO      Scheduler at: inproc://10.40.206.22/2275623/1
2024-07-27 09:52:13 INFO      dashboard at:  http://10.40.206.22:8787/status
2024-07-27 09:52:13 INFO    Registering Worker plugin shuffle
2024-07-27 09:52:13 INFO          Start worker at: inproc://10.40.206.22/2275623/4
2024-07-27 09:52:13 INFO             Listening to:         inproc10.40.206.22
2024-07-27 09:52:13 INFO              Worker name:                          0
2024-07-27 09:52:13 INFO             dashboard at:         10.40.206.22:41269
2024-07-27 09:52:13 INFO    Waiting to connect to: inproc://10.40.206.22/2275623/1
2024-07-27 09:52:13 INFO    -------------------------------------------------
2024-07-27 09:52:13 INFO                  Threads:                         96
2024-07-27 09:52:13 INFO                   Memory:                 629.52 GiB
2024-07-27 09:52:13 INFO          Local Directory: /tmp/dask-scratch-space-10238/worker-frdknfmy
2024-07-27 09:52:13 INFO    -------------------------------------------------
2024-07-27 09:52:13 INFO    Register worker <WorkerState 'inproc://10.40.206.22/2275623/4', name: 0, status: init, memory: 0, processing: 0>
2024-07-27 09:52:13 INFO    Starting worker compute stream, inproc://10.40.206.22/2275623/4
2024-07-27 09:52:13 INFO    Starting established connection to inproc://10.40.206.22/2275623/5
2024-07-27 09:52:13 INFO    Starting Worker plugin shuffle
2024-07-27 09:52:13 INFO            Registered to: inproc://10.40.206.22/2275623/1
2024-07-27 09:52:13 INFO    -------------------------------------------------
2024-07-27 09:52:13 INFO    Starting established connection to inproc://10.40.206.22/2275623/1

Environment:

  • Dask version: 2024.7.1
  • Python version: 3.12.4
  • Operating System: Ubuntu 22.04
  • Install method (conda, pip, source): conda

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions