Can diagnose=True prevent process exit by keeping traceback references alive?

[akozlu]

Description

Looking for confirmation if I get the following behavior right:

When using diagnose=True, the loguru exception formatting keeps references to the traceback frames and keeps the variables alive.
Is it then possible that this prevents garbage collection of objects referenced in those frames and hangs the process indefinitely, IFF the referenced objects manage background threads (SSHTunnel, connection pools) and they do not have atexit defined in their cleanup?

Minimal Reproducible Example

  import atexit # would be the workaround
  import threading                                                                                                                                                                                               
  from loguru import logger                                                                                                                                                                                      
                                                                                                                                                                                                                 
  class ResourceWithThread:                                                                                                                                                                                      
      """Simulates a resource that starts a non-daemon thread (e.g., SSH tunnel)."""                                                                                                                             
      def __init__(self):                                                                                                                                                                                        
          self._stop = threading.Event()                                                                                                                                                                         
          self._thread = threading.Thread(target=self._run, daemon=False)                                                                                                                                        
          self._thread.start()                                                                                                                                                                                   
                                                                                                                                                                                                                 
      def _run(self):                                                                                                                                                                                            
          self._stop.wait()  # Blocks until stop() is called                                                                                                                                                     
                                                                                                                                                                                                                 
      def stop(self):                                                                                                                                                                                            
          self._stop.set()                                                                                                                                                                                       
          self._thread.join(timeout=1)                                                                                                                                                                           
                                                                                                                                                                                                                 
  def main():                                                                                                                                                                                                    
      logger.remove()                                                                                                                                                                                            
      logger.add(                                                                                                                                                                                                
          sink=lambda msg: print(msg, end=""),                                                                                                                                                                   
          diagnose=True,  # Setting to False allows clean exit                                                                                                                                                   
      )                                                                                                                                                                                                          
                                                                                                                                                                                                                 
      resource = ResourceWithThread()                                                                                                                                                                            
                                                                                                                                                                                                                 
      try:                                                                                                                                                                                                       
          raise RuntimeError("Something went wrong")                                                                                                                                                             
      except Exception:                                                                                                                                                                                          
          logger.exception("Error occurred")                                                                                                                                                                     
                                                                                                                                                                                                                 
      # Even though we're done, the process hangs because:                                                                                                                                                       
      # 1. diagnose=True keeps the traceback alive for variable inspection                                                                                                                                       
      # 2. The traceback references the frame where `resource` is a local                                                                                                                                        
      # 3. ResourceWithThread.__del__ is never called                                                                                                                                                            
      # 4. The non-daemon thread keeps the process alive                                                                                                                                                         
                                                                                                                                                                                                                 
  if __name__ == "__main__":                                                                                                                                                                                     
      main()                                                                                                                                                                                                     
      print("This prints, but process never exits")                                                                                                                                                              

The reference chain that prevents the exit looks something like:

  loguru exception handler → traceback → stack frames → local variables → ResourceWithThread → thread  

Is there a recommended pattern or API I'm missing that would allow me to keep diagnose=True while ensuring traceback references are released after the exception is logged?

Thank you for the wonderful library.

[Delgan]

When using diagnose=True, the loguru exception formatting keeps references to the traceback frames and keeps the variables alive.

Not really. When diagnose=True, Loguru will iterate the stack trace and temporarily inspect the variables to build the annotated error message as a string. The variables live in the traceback frames returned by sys.exc_info(), which is an utility function commonly used by exception formatters. Once the formatted stack trace has been constructed (which happens immediately when the logging function is called), Loguru does not keep any references to the variables.

However, the record produced by the logger and sent to the sinks does keep a reference to the sys.exc_info() output: this is the error the sinks can access through record["exception"]. Usually, the log message is processed immediately (e.g. printed to sys.stderr) and afterward the record is discard, thus causing the record and frames to be eventually garbage collected. However, if you use a custom sink that somehow do not disregard the record objects, then the frame variables likely wont be deleted.

Here is a simplified example that shows that no reference to resource is kept by Loguru:

import gc
import weakref
from loguru import logger


class ResourceWithThread:
    pass


def main():
    logger.remove()
    logger.add(sink=lambda msg: print(msg, end=""), diagnose=True)

    resource = ResourceWithThread()
    ref = weakref.ref(resource)

    try:
        raise RuntimeError("Something went wrong")
    except Exception:
        logger.exception("Error occurred")

    return ref


if __name__ == "__main__":
    ref = main()
    gc.collect()  # Just in case.
    assert ref() is None

Regarding the snippet you shared, I don't seem to be able to reproduce it. More precisely, the program always blocks regardless of the value of diagnose. Actually, the problem still occurs while removing Loguru entirely:

import threading

class ResourceWithThread:
    """Simulates a resource that starts a non-daemon thread (e.g., SSH tunnel)."""

    def __init__(self):
        self._stop = threading.Event()
        self._thread = threading.Thread(target=self._run, daemon=False)
        self._thread.start()

    def _run(self):
        self._stop.wait()  # Blocks until stop() is called

    def stop(self):
        self._stop.set()
        self._thread.join(timeout=1)


def main():
    resource = ResourceWithThread()


if __name__ == "__main__":
    main()
    print("This prints, but process never exits")

This is expected: since stop() is never called, there is an obvious deadlock due to the non-daemon thread never being able to finish.

Maybe you intended to call self.stop() in the __del__ method of ResourceWithThread, but that would not help: there is a circular reference between the instance of ResourceWithThread and the started thread. Indeed, the thread keeps a reference to ResourceWithThread through self._run, and ResourceWithThread keeps a reference to the thread through self._thread. Therefore: the ResourceWithThread won't be deleted until the thread is deleted, but the the thread will never be deleted since would only be stopped when ResourceWithThread is deleted.

[akozlu]

Thank you for the thorough explanation 🙏
You're correct, upon a secondary look, the root cause was indeed a circular reference pattern; the diagnose=True correlation was likely coincidental or a timing artifact during shutdown.

Appreciate the clarification on how loguru handles traceback references 💪